Object-Oriented Design and the Fetish of Reusability

 

1371806059-0

Sculpture by Finn Stone

Software Development Best Practices in 2016.

One of the touchstone differentiators of Axial Engineering is to constantly recognize that “Engineer is a subclass of HumanBeing”. As such, we are prone to act with implicit biases which might affect our effectiveness. Since another one of our principles is to maximize the impact of our work, we strive to constantly challenge our assumptions, especially about how our course of action relates to our highest goals as a company, as professionals and as Human Beings. One area that is ripe for re-assessment is the set of accepted ‘Best Practices’ in Software Development, and how they relate to our goals. In that spirit, we’ll be touching over this series of posts on several of the dogmas of our craft, and thinking through their applicability to our situation and point in time.

Part I: Object-Oriented Design and the Fetish of Reusability

Over the last couple of decades, Object-Oriented Design achieved the status of dogma in terms of what constitutes good quality in software development. That’s how “Object-Oriented” and “good” became almost synonymous in our craft. There is also, on the surface, plenty of agreement on what constitutes “Object Orientation” in software design. Almost every interviewee is able these days to recite the expected mantra: “Encapsulation, Inheritance, Polymorphism and Function Overloading”. …”You know, like Java, or Ruby or C++”.

We should all be concerned about unexamined assumptions, and there are a couple of huge ones in the common views of OOD, which in my opinion affect our whole industry considerably. The first one is on the definition of what constitutes good Object-Oriented Design and its relationship with these “Four Noble Truths” mentioned above, which we’ll touch upon in a future post. The second one is more implicit, hidden and perhaps more pernicious: It pertains to the reason on why we want Object Orientation in our software. Let’s start there, for there would be no profitable reason to invest time reasoning about the essence of a technique, if we can’t identify the benefits of using it.

It might sound strange to even ask what’s the benefit of Object-Oriented Design. Ask almost anyone and the same answer will be forthcoming: Reusability, of course! … Finally moving software construction into the Industrial Age, where we can buy or download standard parts, just like microchips and build good quality software in record time!… The best I understand it, the common perception of Reusability is to create software components as generic and flexible as possible, so that they can be applicable to the greatest number of unforeseen future uses while remaining unchanged. As the theory goes, if we build our software this way, we’ll be able to never touch these components again, and use them in many thereto unforeseen scenarios.

So far so good, I suppose. Alas, in order to build such extreme level of ‘genericity’ in our software, the complexity and cost of building it goes up almost exponentially. Let’s dwell on that for a moment: Suppose you work for a company that sells some gadgets called ‘Things’. Let’s say you want to build some software, like your “Inventory Manager of Things’where ‘Thing’ is a well-defined concept that everyone in your company understands. But let’s, as a thought experiment, make this software as reusable as possible. To achieve this, typically, we’d start with classes, then we’d make them into class hierarchies using Inheritance, and then we’d abstract the interfaces into protocols, contracts or interfaces, depending on your language of choice. But… wait! Who knows if we’ll need to apply this to some future ‘OtherTypeOfThing’? So, let’s make some ‘AbstractThing and ‘SpecialThing’ and their corresponding ‘IAbstractThing’ and ‘ISpecialThing’ abstract interfaces, while considering every combination of concepts or ideas in which such a hierarchy could be remotely applicable. Done? Not so fast: At that point we might want to throw in our ‘AbstractThingFactory’ and several ‘ConcreteThingFactories’, (after all, we want to throw in some Design Patterns), and while we are at it, we might as well make ‘Thing’ generic, with all the ‘AbstractThing<T>’, ‘Thing<T>’ and even ‘ThingHelper<T>’ paraphernalia that it will ever require. And, -bonus!-, as a software developer, it is likely that you love dealing with abstraction, so most likely you’ll have a blast thinking through this. Life as a reusable software developer feels at this point like it it can’t get any better. Throw in some Inversion-of-Control and Configuration-by-Convention, while controlling all of these options with a nice, juicy (but computationally unaccountable) set of XML or JSON configuration files, and you’ll be seemingly well on your way to the Holy Grail of Reusability.

Dizzy yet? Let’s go back to Earth. Shall we?

The return on every investment is the relationship between the cost or effort put into it and the real benefits it gives you over time. First, on the effort: The more parts something has, the more it is, by definition, complex. And complexity is always costly: It costs more to build it and it costs more to maintain it. In most cases, it also creates barriers of to diagnosing problems efficiently. For a very simple example, think about the added difficulty of debugging software that goes through abstract interfaces and injected components. Finally, the added complexity creates many different points of touch when a change is needed. Think of how you cursed your luck when you had to change an abstract interface that had many different concrete implementations. And I doubt your QA engineer liked you any more after that…You get the picture: Add up the hours of added effort required to keep such software operating well over its (shorter than you think) lifespan, and you’ll get a good idea of the cost incurred.

Let’s think, on the other hand, about the return side. Don’t get me wrong: Abstract Interfaces, Inversion of Control and Generic Containers or Algorithms all have use cases in which they provide many measurable benefits. We’ll even be discussing some interesting ones in a future post.   But more often than not, the kinds of software for which Reusability is at the top of the priority stack are associated with frameworks that are intended from the beginning for an industry as a whole, and created at a great cost in order to save time in implementing concepts that are by definition abstract. They are also used by hundreds of unrelated teams, in unrelated domains. Think STL, or NumPy or the Java Collections framework. However, these are projects that operate in domains that are orthogonal to the problems most developers face day-to-day. This article from 1998 gives us a very interesting rule of thumb: “…You can’t call something reusable unless it’s been reused at least three times on three separate projects by three separate teams.”.

On narrower domains, if we examine the track record of our previous efforts, we’ll come to confront a disquieting reality: Most “reusable” software is actually never reused. And even when we account for the time saved for the cases when we do reuse it, for most domains we’ll come to see that the return-on-investment of building all software components as highly reusable is, by and large, negative. And yet we persist, as an industry, in perpetuating this myth, while ignoring other potential benefits that we can target in our development process.

And so, we arrive to the point where, if Reusability is the main benefit of Object-Oriented Design, from the cost/benefit point-of-view we might as well dispense with it, unless we are creating the next framework-level collections library. Otherwise, we are not likely to enjoy the benefits of the increased complexity. However, it is our contention that Object-Oriented Design does provide some other real, measurable benefits to the quality of software design, and that these benefits can be achieved without the exponential growth in complexity discussed above. But in order to discuss these benefits lucidly, we need to reexamine our notions of what is essential vs accidental in Object-Oriented Design Practices. That will be the subject of our next blog. For more on that, watch this space.


References

  1. SOA and the Reality of Reuse
  2. The Elusive Search for Business Frameworks
  3. Software Reusability: Myth Or Reality?
  4. The Reality of Object Reuse
  5. A Comparison of Software Reuse Support in Object-Oriented Methodologies
  6. Software Reuse:  Principles, Patterns, Prospects

 

Ionic Creator – From idea, to prototype, to real life app

Edit 02/26: access to the repo for this code here

The Ionic team has been hard at work trying to lower the barrier of entry in the mobile development world.

The Ionic Creator is a simple prototyping tool that helps developers, designers, and project managers to quickly build mobile apps and websites without having to code.

This allows for a quick feedback loop among a team which helps speed up the development process drastically. A project manager might prototype a design and generate real, clean Ionic code to give to a developer. Or, a developer might use Creator to quickly generate UI snippets and rapidly bootstrap new app projects.

Unfortunately, as of now, dynamic data prototyping is not directly supported in the tool and this tutorial aims at highlighting how this can be done.

What is Creator ?

As Matt Kremer puts it in the very first Ionic Creator tutorial video:
“Ionic Creator is a simple drag and drop prototyping tool to create real app with the touch of your mouse.”

Indeed, Ionic Creator is provides a full set of Ionic components that you can simply drag and drop into your project and rapidly prototype a fully working app.

ionic.creator.components

Who is Ionic Creator for?

  • Novice Developers trying to get their hand in hybrid mobile development
  • Designers tweaking around options for product development
  • Experienced Ionic developers looking to bootstrap their projects
  • Freelance developers gathering clients feedback via sharing features

Collaborate, Share and Export your App

Ionic Creator makes it simple to collaborate and share your app in many ways.

You can send a link to the app running in the browser via URL, Email or SMS so a user can run the app from a browser.

Using the Creator App (available on Android & iOS), you can share the app and have it run directly on the device in similar conditions as if it was a stand alone app.

Finally, you can package your app for iOS and/or Android directly through the Ionic Package service.

Introducing Axial Events!

The goal of v1 of the app is to show a list of events and allow the member to indicate which other attendees they want to meet up with so we can send reminders during the event.

We will need:

  • List of Events: title, content, image
  • Event Detail page: list of attendees with a way to indicate interest

Step 1: Project Creation

First we will pick a blank project as we do not need a side menu or a tab layout.

ionic.creator.project.creation

Step 2: Create the list of Events

Then, let’s rename the new page as Events, drag in some List Item w/ Thumbnail and give each of them some details.

ionic.creator.events.list

Step 3: Create Event details page

For each event, we will need to create a detail page which we will name according to the event and add a list of List Item w/ Thumbnail for the attendees:

ionic.creator.event.details

Step 4: link everything together!

Finally, for each item in our events list, let’s adjust the link to their respective
target page:

ionic.creator.linking

Step 5: let’s take it out in the wild!

At this point, we have an app that showcases how the flow will go from screen to screen.

Let’s take it live and plug it to our API. It’s time to export our app.

ionic.creator.export

Once you open the repository in your favorite text editor, everything has been wired up for you.

With a little bit of Angular knowledge, and a dash of CSS, the sky is the limit!

Let’s clean up the code a bit and plug in our API.

Step 6: Cleanup – Views – Events

First let’s clean up the Events List view and make use of repeating element:

<br /><img src="img/6YS77136Q5yo9V7yjI6g_Atlanta-Summit-hero-image.jpg" alt="" />
<h2>Atlanta Summit 2016</h2>
February 9th, 2016

<img src="img/oQ9mqXNQzeoPtXXwzXzZ_san-francisco-summit-hero-image.jpg" alt="" />
<h2>San Francisco Summit 2016</h2>
March 16, 2016

<img src="img/WTUsCmmCQjWl43vRjGOx_dallas-summit-homepage-image.jpg" alt="" />
<h2>Dallas Summit 2016</h2>
April 13, 2016

then becomes:

<br /><img alt="" />
<h2>{{::event.title}}</h2>
{{::event.date}}

Step 6: Cleanup – Views – Event Info

When duplicating the Event Details page, we created three identical page which should really be one template instead. Therefore, atlantaSummit2016.html, dallasSummit2016.html and sanFranciscoSummit2016.html are replaced by one event.html file which ressembles:

<div style="text-align:center;"><img alt="" width="100%" height="auto" /></div>
<h3>Attendees</h3>
<img alt="" />
<h2>{{::attendee.name}}</h2>
{{::attendee.title}}

Step 6: Cleanup – Routes

Since we have removed the duplicated views, we need to clean up the routes.js file a little from this:

$stateProvider
.state('events', {
url: '/events',
templateUrl: 'templates/events.html',
controller: 'eventsCtrl'
})
.state('atlantaSummit2016', {
url: '/events/atlanta',
templateUrl: 'templates/atlantaSummit2016.html',
controller: 'atlantaSummit2016Ctrl'
})
.state('sanFranciscoSummit2016', {
url: '/events/sanfrancisco',
templateUrl: 'templates/sanFranciscoSummit2016.html',
controller: 'sanFranciscoSummit2016Ctrl'
})
.state('dallasSummit2016', {
url: '/events/dallas',
templateUrl: 'templates/dallasSummit2016.html',
controller: 'dallasSummit2016Ctrl'
});

to this:

$stateProvider
.state('events', {
url: '/events',
templateUrl: 'templates/events.html',
controller: 'eventsCtrl'
})
.state('event', {
url: '/events/:id',
templateUrl: 'templates/event.html',
controller: 'eventCtrl'
});

Step 6: Adjust Controllers

Instead of one controller per event, we will need one eventCtrl controller:

angular
.module('app.controllers', [])
.controller('eventsCtrl', function($scope) {
})
.controller('atlantaSummit2016Ctrl', function($scope) {
})
.controller('sanFranciscoSummit2016Ctrl', function($scope) {
})
.controller('dallasSummit2016Ctrl', function($scope) {
})

then becomes:

angular
.module('app.controllers', [])
.controller('eventsCtrl', function($scope, EventsService) {
$scope.events = [];
EventsService.getEvents().then(function(res) {
$scope.events = res;
});
})
.controller('eventCtrl', function($scope, $stateParams, EventsService) {
$scope.event = [];
EventsService.getEventDetails($stateParams.id).then(function(res) {
$scope.event = res;
EventsService.getEventAttendees($stateParams.id).then(function(res) {
$scope.event.attendees = res;
});
});
})

Step 6: Implement Services

First of all, we need to put together a quick API which will provide data manipulation layer for our app.

For the purpose of this demo, I put together a quick Express API running with nodeJS available here.

Given the API is now running at http://localhost:3412/, we have the following endpoints:

  • GET /events
  • GET /events/:id
  • GET /events/:id/attendees

Let’s plug all those in our EventsService:

angular.module('app.services', [])

.service('EventsService', ['$http',
function($http) {
return {
getEvents: function() {
var promise = $http.get('http://localhost:3412/events').then(function (response) {
return response.data;
}, function (response) {
console.log(response);
});
return promise;
},
getEventDetails: function(id) {
var promise = $http.get('http://localhost:3412/events/'+id).then(function (response) {
return response.data;
}, function (response) {
console.log(response);
});
return promise;
},
getEventAttendees: function(id) {
var promise = $http.get('http://localhost:3412/events/'+id+'/attendees').then(function (response) {
return response.data;
}, function (response) {
console.log(response);
});
return promise;
}
}
}
]);

Step 7: Serve!

At this point, we have our app connected to a working API and we are ready to publish!

Display the app in a browser with android and iOS version side by side

$ ionic serve --lab

Build and run the app in iOS simulator

$ ionic build ios && ionic run ios

Build and run the app in android simulator

$ ionic build android && ionic run android

Package your app for store publication via Ionic package

$ ionic package build ios --profile dev

Resources

Ionic Framework – http://ionicframework.com/
Ionic Services – http://ionic.io/
Ionic Creator – http://usecreator.com
Ionic Creator Tutorial Videos on Youtube
Ionic Package – http://blog.ionic.io/build-apps-in-minutes-with-ionic-package/
Find help on Slack – http://ionicworldwide.herokuapp.com/

What’s Blocking My Migration?

At Axial, as at many other companies, we have software written in the olden days still in active use. It does what it’s supposed to do, so replacing it is not the highest priority. However, sometimes these older scripts keep database transactions open longer than are necessary, and those transactions can hold locks.

This poses a problem when we’re running Alembic migrations; modifying a table requires a lock not just on that table, but on other tables, depending on the operation. (For example, adding a foreign key requires a lock on the destination table to make sure the foreign key constraint is enforced until the transaction is committed.) Sometimes we developers run a migration and it seems to be taking a long time; usually this is because it’s waiting for a lock. Fortunately, it’s really easy to check on the status of queries and locks in PostgreSQL.

There are three system views we care about: pg_class contains data about tables and table-like constructs, pg_locks contains the locking information (with foreign keys to pg_class), and pg_stat_activity contains information on every connection and what it’s doing. By combining data from these three views (technically pg_class is a system catalog, not a view, but whatever) we can determine which connections are holding locks on which tables, and on which tables a given connection is awaiting locks. We have a script called whats_blocking_my_migration that queries these views.

Rather than explain the columns on these views (which you can see very easily in the docs anyway), I’ll show you the queries we execute.

The first thing we do is get information on Alembic’s connection to the DB. In env.py, we set the “application_name” with the connect_args argument to create_engine; this way we can identify the connection Alembic is using. Once we do that, we can use application_name the check the status of the Alembic connection. If there are no rows, or if none of the connection rows are actually waiting, the script prints that info and exits.

SELECT pid, state, waiting FROM pg_stat_activity WHERE application_name = 'alembic';

However, if an Alembic connection is “waiting”, that means something’s holding a lock that it needs. To find out what, we use the pid (which is the pid of the Postgres process handling the connection) to query pg_locks. Our script actually does a join against pg_class to obtain the table name (and make sure the relation is kind ‘r’, meaning an actual table), but the table name can also be obtained directly from the relation by casting it from oid to regclass.

SELECT relation, mode FROM pg_locks WHERE lock type = 'relation' AND granted IS FALSE and pid IN alembic_pids;

This gets us the list of tables that the Alembic connection needs to lock, but there’s one more step we can take. pg_class will also tell us the pids of the connections that are holding locks on these tables, and we can join that pid back against pg_stat_activity to obtain the address and port number of the TCP connection holding the lock. (Unfortunately, most of our connections don’t yet populate application_name, so we have to provide this info, which the user can cross-reference with lsof -i to figure out which service is holding the lock.) The query looks a bit like this:

SELECT client_addr, client_port FROM pg_stat_activity JOIN pg_locks ON pg_stat_activity.pid = pg_locks.pid WHERE pg_locks.granted IS TRUE AND pg_locks.relation IN locked_relations;

There’s no real secret sauce here; it’s just some basic queries against the Postgres system views, but this makes the lives of our developers a little bit easier.

Anonymizing data with SQLAlchemy and Ansible

One of the key aspects of a good configuration management tool is that it provides the aspect of idempotence to your system configuration.

Idempotence (/ˌaɪdɨmˈpoʊtəns/eye-dəm-poh-təns) is the property of certain operations in mathematics and computer science, that can be applied multiple times without changing the result beyond the initial application.”

– http://en.wikipedia.org/wiki/Idempotence

Given a system, a desired target state and the sequence of steps required to take an unconfigured host to that state, an idempotent CM (configuration management) tool will make only the  subset of changes required to bring the system to the desired target state.  On a system that is already in the desired state, repeated runs will have no effect as the tool will makes zero changes, since none are required.

Ansible is a configuration management written in Python that performs its actions by way of modules.  Ansible configurations are defined as YAML-formatted files called playbooks, which by way of modules, compose a description of what the state of the system should be.

---
- hosts: localhost

  tasks:
  - name: Ensure sample.txt is correctly permissioned
    file:
      path:   /data/sample.txt
      owner:  bobby
      mode:   0600

In the above example, a playbook ensures that the file “sample.txt” located in the “/data” directory on the system is owned by the user “bobby”, and that only bobby can read or write to the file (as specified by mode 0600).

If this playbook is run in Ansible, the file module will examine the file at that specified location path, and make changes to its ownership and permissions if they do not match what we’ve defined in the playbook.

An Ansible module can be written in any language, but the default set of modules provided with Ansible are written in Python.

One could technically write a set of carefully written shell scripts to do tasks like these, for example, but a CM unit such as an Ansible module has a well-specified interface, that when you write against, will automatically give you access to all the tooling and features that the CM framework provides. The most obvious benefit is that your operation now becomes a unit that you can verily easily use in combination with other units to define non-trivial workflow.

In this instance, we wanted to use Ansible to ensure that certain types of data within a given database were anonymized (where necessary). This might be for ensuring that sensitive user information is removed before a dataset is used for testing or development. Since Ansible does not have this capability out-of-the-box, we added the functionality as a module by way of the Python SQLAlchemy library.

SQLAlchemy

SQLAlchemy is quite a capable database toolkit and ORM (object relationship mapper) for Python, and it’s been covered to some extent here on the Corps blog.

SQLAlchemy gives us the tooling we need to interact with the variety of databases in a (mostly) vendor agnostic way using plain Python, and Ansible gives us a framework within which we can define system configuration and combine complex actions to take a system to a desired state, or define and execute a specification for a particular process.

Our basic aim is to examine a database’s schema, recognize columns that look like they contain the type of data we want to anonymize, and make changes to the data only where necessary.  The type of data that should be anonymized will vary, but in this case we wanted to apply changes to user contact information.

An example of anonymization happens when we encounter a column that contains email information. The anonymization function that is applied returns a query that transforms all email values in this column into an anonymized email address format that looks like ‘user+8b7578454fdb311af0ce727f8944bd0a@example.com‘. The hash (8b75…d0a) is generated from the original email address, so you get the same output for the same input every time. This is assures integrity across the dataset for all email columns that are modified.

Finding these columns happens when SQLAlchemy is examining the database. This is done by way of SQLAlchemy’s reflection functionality.  The identification and marking of columns is performed during the reflection phase, as this is when SQLAlchemy is building its understanding of what the database’s structure looks like.

During reflection, we want to flag columns of interest (candidates for anonymization). There’s more than a few ways of doing this, but SQLAlchemy offers a very interesting feature which came in handy for this.

SQLAlchemy Events

The SQLAlchemy event API, introduced here, allows you to register callback functions or methods that fire whenever certain operations take place within SQLAlchemy, effectively letting you hook any arbitrary logic you desire into its internal operation.

There are events for a large portion of the operations that SQLAlchemy performs, from DDL operations to the moment when a transaction is rolled back, for example.  As a result of this, you can use events for quite a number of purposes, such as:

We use the ‘column_reflect’ event to step in and “decorate” the Column objects that SQLAlchemy creates while it performs database reflection.

from sqlalchemy import ..., event

class DBAnonymizer(object):
    def __init__(self, params):
        ...
        # subscribe our column decorator function to column reflection events
        event.listens_for(Table, 'column_reflect')(
            self.column_decorator_callback)

    ...
    def column_decorator_callback(self, inspector, table, column_info):
        ...

(Above, we’re calling the event.listens_for event subscription decorator directly as a function to register our own instance method that we want to use for column reflection callbacks.)

By “decorate”, we basically mean that we mark the column to be anonymized with the query generation function that is pertinent to the type of contact information it contains.

By default, the module attempts to guess which columns should be decorated by looking at the column names.  But we can make additional decisions as specified by the parameters passed in to our module from the user’s playbook, such as ignoring all columns except those from a particular table, or decorating an explicitly specified list of columns.

When reflection is complete, the database structure is stored in SQLAlchemy’s MetaData object, which now also contains columns that have been “marked” by us.

Idempotence in Database Operations

Following the idempotence model, any actions we would execute should only happen to the column values that aren’t in the state we want.

Since we’re working in a database, that’s easily accomplished using a WHERE clause.  We simply define the conditions that don’t match what we want within the WHERE clause being used for our UPDATE.  Any rows that are already in the desired state don’t match, and won’t be updated.

Additionally, Ansible defines a special mode called Check mode, intended for running a playbook against a system with the intent of only checking what changes would be made, and not making any actual changes.  It’s very useful when you want to get an idea of how far a system is from the state defined in the playbook, without modifying anything on it.

We implement check mode by simply changing our anonymizing queries from UPDATEs into SELECTs, which then shows us how many rows do not match our desired state.

Building as many of our configuration operations into well-defined units such as Ansible modules affords us the ability to keep the processes we use in a consistent and composable format that we can then easily build upon.

If you’re interested in the anonymization module, you can grab it here.

Testing Front-End Components using Karma, Jasmine and Spokes

jasmine+karma+spokeAs your applications grow more complex and involved on the client-side, it becomes incredibly important to find ways to create reusable and encapsulated components that you can share between services. At Axial, we use spokes to solve that problem. With spokes, we can generate small executable Javascript files that encapsulate all of the Javascript, CSS and HTML necessary for our components and include them wherever we see fit.

There’s one more thing that we can do to gain even more confidence in the execution of our front-end components. Unit Testing. By breaking our application into small testable parts, or units, we can be sure that it is working exactly as we intend, protect against regressions in the future and most importantly refactor with confidence.

Using spokes is the first step in creating those small testable units within our application. The next step is actually writing the tests! In this article we are going to use Jasmine to write the tests themselves, and Karma to aid in creating our testing environment and utilizing spokes to their fullest extent.

To start, you’ll want to follow these directions to get Karma up and running on your machine. Once that’s all set, let’s create a new project and set up our Karma Configuration.

mkdir bootstrap-spoke && cd bootstrap-spoke

Then create a file called karma conf.js with these basic settings.

module.exports = function(config) {
    config.set({
        frameworks: ['jasmine'],
        files: [
            'spec/*.js'
        ],
        browsers: ['Chrome']
    });
};

Here, we’ve told Karma to look for all Javascript files in the spec directory. Let’s create that along with our first test file.

mkdir spec && touch spec/main_spec.js

We’ll write a basic test just to confirm everything is up and running.

in spec/main_spec.js

describe("Our First Test", function() {
    it('should pass', function() {
        expect(true).toBe(true);
    })
})

and run:

karma start

1

Awesome! Now that we know Karma is finding our test files and we’re seeing the output we expect, we can start adding spokes to our project. We are going to make a fairly basic spoke that will include Backbone and its dependency Underscore in our project.

mkdir spokes && touch spokes/backbone.cfg

We’ll give our spoke the following config:

[backbone]
js = ["underscore.min.js", "backbone.min.js"]

Grab those files from http://backbonejs.org/ and http://underscorejs.org/ respectively, and place them under spokes/js.

Now, we just want to make sure we’re able to generate our spokes from the command line before we get fancy and have Karma do the work for us.

spokec -c spokes/backbone.cfg -p spokes backbone > backbone.spoke.js

You should now see a new file named backbone.spoke.js in your root directory with the combined contents of both the Underscore and Backbone libraries. Let’s have Karma include any generated spokes in its build.

files: [
    '*.spoke.js',
    'spec/*.js'
],

And write a simple test to make sure Backbone and Underscore have been loaded correctly:

describe('Backbone', function() {
    it('should be loaded', function() {
        expect(Backbone).toBeDefined();
    })
})

describe('Underscore', function() {
    it('should be loaded', function() {
        expect(_).toBeDefined();
    })
})

All tests pass!

2

Now, we could simply run this command to generate our spoke anytime we run our specs but that would get old quick if we are modifying these files with any sort of frequency. We want Karma to automatically create the spokes for us before it runs its specs. Plus, I don’t like having these new generated files hanging around my test directory.

Enter Karma Preprocessors

A preprocessor will ingest files that you provide it, and perform a particular action on each. When it’s done, it will return the file and in most cases include the results of its work in the current Karma build. For example, the popular Karma Coffee Preprocessor will simply take all .coffee files and compile them to .js files at build time.

We’re going to make our own preprocessor, and have it compile our spokes and automatically include its results in our test build.

First thing’s first, delete backbone.spoke.js and its entry in karma.conf.js so we know we aren’t accidentally including that instead of our newly generated spokes.

Creating Karma-Spoke-Preprocessor

In order to create and utilize our own preprocessor, we need to create a new project (outside of our current directory) and include it in our backbone-spoke project as a Karma plugin. In a separate terminal window:

mkdir karma-spoke-preprocessor && cd karma-spoke-preprocessor

Karma plugins are loaded via NPM, so there is a little bit of boilerplate we need in order to get things up and running. First, I’ll create a package.json to describe our new project:

{
    "name": "karma-spoke-preprocessor",
    "version": "",
    "description": "A Spoke Preprocessor for Karma",
    "main": "index.js",
    "scripts": {
        "test": "echo "Error: no test specified" && exit 1"
    },
    "author": "Nader Hendawi",
    "license": "ISC"
}

and a new file index.js with the following (which I will explain shortly):

var createSpokePreprocessor = function(args, config, logger, helper, basePath) {
    return function(content, file, done) {
        done(content);
    };
};

createSpokePreprocessor.$inject = ['args', 'config.spokePreprocessor', 'logger', 'helper', 'config.basePath'];

// PUBLISH DI MODULE
module.exports = {
    'preprocessor:spoke': ['factory', createSpokePreprocessor]
};

This may look intimidating at first, but it’s really rather straightforward once you understand what’s going on. The first thing we want to do is create a function that will act on our preprocessed files. We are calling this createSpokePreprocessor here and it simply returns another function.

When all is said and done, Karma will be calling this function for each file that needs to be preprocessed. So, in our return function we are provided arguments for the contents of the file, some metadata about the file and a done function.

The contents of the file will be just that…a string representation of the file in question. The file argument will provide us with an object resembling the following:

{ path: '{{PATH}}/file.ext',
    originalPath: '{{PATH}}/file.ext',
    contentPath: '{{PATH}}/file.ext',
    mtime: Wed Apr 30 2014 16:39:32 GMT-0400 (EDT),
    isUrl: false,
    sha: 'a573bab0110b98ceaba61506246c18c4cdb179f9' }

Lastly, we have a done argument. This is a function that we must call when our preprocessor has finished its work so that Karma knows to proceed onto the next one. This function will take an argument that represents the output of our preprocessing. For now, we are simply passing the content back into the done function so that the file doesn’t undergo any processing.

Let’s throw some logging in there just to see that our app is actually using our preprocessor:

return function(content, file, done) {
    console.log(file);
    done(content);
};

Using Our Preprocessor

Adding a preprocessor to our project is very easy. Back in our karma.conf.js add the following:

preprocessors: {
    'spokes/*.cfg': 'spoke'
},

This will tell Karma to preprocess all .cfg files in the spokes directory using the spoke preprocessor.

If we were to run our tests now, everything would run just fine…but we’re not seeing our logging from the preprocessor. That’s because, according to Karma, there are no .cfg files in spokes, so it has nothing to pass to the spoke preprocessor. We have to tell karma to include this directory in its build.

files: [
    'spec/*.js',
    'spokes/*.cfg'
],

Now, if we run karma start again we should be good to go….but we aren’t.

3

Oh yea, we never told our project about our new preprocessor. Normally we’d do this by simply installing via npm npm install karma-spoke-preprocessor but our plugin only lives on our machine for now.

One way to do this is to provide the install command with the path to our module like so:

npm install ../karma-spoke-processor

This would work, but we would need to rebuild our spoke preprocessor and reinstall it in our project directory everytime we made a change.

NPM LINK

The npm link command is extremely helpful for local package development. The docs have a much more detailed explanation but essentially, we want to symlink our spoke preprocessor’s directory in our local npm installation:

In our karma-spoke-preprocessor directory:

npm link

We should see output similar to the following:

/usr/local/lib/node_modules/karma-spoke-preprocessor -> /Users/nhendawi/Projects/blog_posts/karma-spoke-preprocessor

This will tell npm to provide our karma-spoke-preprocessor directly from its directory rather than a remote repository.

Then, we need to tell our backbone-spoke project to use this symlink’ed version of our plugin. In backbone-spoke:

npm link karma-spoke-preprocessor

We should see the following output which confirms that we have successfully installed a linked version of our preprocessor:

/Users/nhendawi/node_modules/karma-spoke-preprocessor -> /usr/local/lib/node_modules/karma-spoke-preprocessor -> /Users/nhendawi/Projects/blog_posts/karma-spoke-preprocessor

Now, we can make changes to our preprocessor and have them be immediately available to our app!

Running karma start now, we should see the output from our log:

{ path: '/Users/nhendawi/Projects/blog_posts/fake-app/spokes/backbone.cfg',
    originalPath: '/Users/nhendawi/Projects/blog_posts/fake-app/spokes/backbone.cfg',
    contentPath: '/Users/nhendawi/Projects/blog_posts/fake-app/spokes/backbone.cfg',
    mtime: Wed Apr 30 2014 16:39:32 GMT-0400 (EDT),
    isUrl: false,
    sha: 'a573bab0110b98ceaba61506246c18c4cdb179f9' }

4

Great! Our project directory is using our preprocessor for the files that we have configured.

However, our tests are failing since we are no longer including Backbone and Underscore in our build. We’ll solve that once we actually generate our spokes and include them in our project.

Generating the spokes

Configuration Options

Now that we know our files are being delivered safely to our preprocessor, we have to find a way to use them to generate their corresponding spokes. To do that, we are going to pass in some configuration options from our Karma project and use them in our preprocessor.

In karma.conf.js:

spokePreprocessor: {
    options: {
        path: "spokes",
    }
},

By passing in an object named after our preprocessor, we can provide context-specific configuration options right to our preprocessor. In our case, we want to give our spokec command line client some information on what it should use for config and file paths when generating spokes.

To use these in our preprocessor is just as easy:

var createSpokePreprocessor = function(args, config, logger, helper, basePath) {
    config = config || {}

    var defaultOptions = {
        path: "/"
    };

    var options = helper.merge(defaultOptions, args.options || {}, config.options || {});

    return function(content, file, done) {
        console.log(file);
        done(content);
    };
};

With this, we can take the config that has been passed in from karma and access it directly in our preprocessor code. I’ve added some defaultOptions just for good measure and used the Karma helper class to merge it with our passed in options. We could just as easily set this to be something like:

{
    path: "spokes",
}

This would allow us to avoid having to define these in our karma.conf.js at all, but we’ll leave it as is for the purposes of this tutorial and to show how easily we can configure our plugins.

Now that we have our configuration options all ready, we just need to find a way to run our spokec command and let it work its magic.

Enter Exec

In order to do that, we are going to use the exec npm package. Exec will allow us to “Call a child process with the ease of exec and safety of spawn”. In our case, we will be using Exec to run our spokec -c etc etc command directly from within our preprocessor’s return function.

First, add it to our package.json.

"dependencies": {
    "exec": "0.1.1"
},

and run npm install to add it to our preprocessor project.

Then, require it at the beginning of our index.js.

var exec = require('exec');

For each file that is passed into our preprocessor, we want to run the following command:

spokec -c CONFIG_PATH -p FILE_PATH FILENAME | cat

Now, CONFIG_PATH and FILE_PATH are both available in our config options and filename can be parsed easily from our file argument:

var filename = file.originalPath.replace(basePath + '/' + options.path + '/', '').split('.')[0];

It’s important to note that we are piping the output into the cat command. This means that the output from spokec will be provided directly to our exec callback method as the out argument. This will then be provided to the done function and finally evaluated within our project.

So, our return function will now look like this:

return function(content, file, done) {
    var filename = file.originalPath.replace(basePath + '/' + options.path + '/', '').split('.')[0];

    exec('spokec -c ' + file.originalPath + ' -p ' + options.path + ' ' + filename + " | cat", function(err, out, code) {
        if (err instanceof Error)
            throw err;

        process.stderr.write(err);
        done(out);
    });
};

Now head back to our project directory, and run karma start:

5

And all of our tests are back to passing!

UserName Spoke

Let’s take this one step further and really see how we can test a spoke complete with HTML, JS and CSS. We’re going to follow the steps in the Spoke Project on GitHub and create the username spoke within our project.

When all is said and done, our project directory structure should look like the following:

6

There are a couple of changes that we will make for the purposes of this tutorial. First off, since we are already including the backbone spoke as part of our build, we are going to remove the following line from username.cfg

spokes = ['backbone']

If we run karma start now, we’ll see the following error:

7

Oops, we never included jQuery in either of our spokes. To get things moving, we’ll grab it from jQuery add it to our Karma build:

files: [
    'spokes/js/jquery.min.js',
    'spec/*.js',
    'spokes/*.cfg'
],

Testing our Backbone Model

At this point, our tests are still passing but we haven’t added any new ones for our Username Spoke. Let’s change that. First, let’s do some simple tests for our UsernameModel to ensure it behaves correctly. In a new file at spec/username_spec.js:

describe("UsernameModel", function() {
    it('should exist', function() {
        expect(UsernameModel).toBeDefined();
    })

    it('should have the correct defaults', function() {
        var default_model = new UsernameModel;

        expect(default_model.get('first_name')).toBe('');
        expect(default_model.get('last_name')).toBe('');
        expect(default_model.get('is_internal')).toBe(false);
        expect(default_model.get('is_masq')).toBe(false);
    })

    it('should honor passed in attributes', function() {
        var new_model = new UsernameModel({is_internal: true, is_masq: true});

        expect(new_model.get('is_internal')).toBe(true);
        expect(new_model.get('is_masq')).toBe(true);
    })
})

I normally don’t like bundling up so many assertions in each it block, and some of these tests would be considered redundant, but I’m making an exception in this case for both the purposes of this tutorial and the simplicity of our spoke. In each test, we instantiate a new model with the attributes we’d like to test, and simply check to make sure it has the expected values. If we had a more complicated model, this is also where we could mock out any HTTP Requests we would make to the server, or any events that may occur when our model changes.

Testing our Backbone View

Testing the UsernameView takes a little bit more work than its Model counterpart. For one, we need to instantiate a model to pass into our view, and provide it with a DOM element in which to render itself. So, our it blocks will have a little more setup:

describe("UsernameView", function() {
    it('should exist', function() {
        expect(UsernameView).toBeDefined();
    })

    it('should set the model correctly', function() {
        var model = new UsernameModel({first_name: "Nader", last_name: "Hendawi"}),
        view = new UsernameView({model: model});

        view.render();

        expect(view.model.get('first_name')).toBe("Nader");
        expect(view.model.get('last_name')).toBe("Hendawi");

    })

    describe('template rendering', function() {
        it('should render the correct text when name is too long', function() {
            var model = new UsernameModel({first_name: "Nader", last_name: "Hendawi The Magnificent"}),
            view = new UsernameView({model: model});

            view.render();

            expect(view.$el.html().trim()).toBe("N.nnHendawi The Magnificent");
        })

        it('should render the correct text when the name is short', function() {
            var model = new UsernameModel({first_name: "Nader", last_name: "Hendawi"}),
            view = new UsernameView({model: model});

            view.render();

            expect(view.$el.html().trim()).toBe("NadernnHendawi");
        })

        it('should render the internal badge when necessary', function() {
            var model = new UsernameModel({first_name: "Nader", last_name: "Hendawi", is_internal: true}),
            view = new UsernameView({model: model});

            view.render();

            expect(view.$el.html().trim()).toBe("NadernnHendawinn(internal)");
        })

        it('should render the masq badge when necessary', function() {
            var model = new UsernameModel({first_name: "Nader", last_name: "Hendawi", is_masq: true}),
            view = new UsernameView({model: model});

            view.render();

            expect(view.$el.html().trim()).toBe("NadernnHendawinn(masq)");
        })
    })
})

Feel free to go through each test and see exactly what’s going on. Some of the setup logic can easily be moved into a beforeEach block such as:

beforeEach(function() {
    model = new UsernameModel({first_name: "Nader"})
});

where each subsequent test would simply update this one model object. But, for now I think it’s okay to repeat some of this code as each spec does test a distinct case.

If we run our tests now, we should see them all passing!

8

Next Steps

This is really the tip of the iceberg when it comes to testing our front-end components. From here we could easily add more spokes, create more complex view logic, integrate some communication with a RESTful backend service…anything we would want to do in a modern client-side application. But, we’re no longer at the mercy of the assumption that our code just works. By building up a sufficient test framework around our components, we have the liberty to add new features or refactor existing components with the confidence that we haven’t caused any regressions or bugs along the way. To change the output of our UsernameView’s template, we would simply update the tests with our new expected result, watch them fail, and rewrite the template until our tests pass.

But the testing doesn’t stop with Backbone. If you’re writing applications in Angular, Ember, React…whatever, the process is almost identical to the one we’ve followed here. Alot of the time, the hardest part will be getting our test environment in a state where we can easily test single units of our application. In our case, that’s where spokes, and ultimately our karma-spoke-preprocessor come in.

Leveraging Karma’s plugin interface, we were able to create a preprocessor that would ingest our spoke configuration and automatically embed its result in our test build. That’s awesome! And there’s so much more we can do. In future articles, I’ll be exploring ways to build our own test reporters for Karma, refactoring existing Javascript code to make it more easily tested, and other exciting ways to make our front-end architecture more robust and reliable.