Reaxial Update – On Stages And Actors

Since I last wrote about Reaxial we’ve come up with some new abstractions that make it easier to write reactive handlers, and have been busy transitioning our code to use the new architecture. I thought I’d take this opportunity to share our progress with you.

As we started transitioning to Reaxial, we realized that creating an entire service for each reactive component was a bit of overkill. Many features we have implemented with reactive components run sporadically and are not particularly time sensitive, and typically there are a number of features that depend on the same updates. Having a separate process and a separate connection to Kafka is wasteful and inefficient in these cases. However, other features have to react in a timely fashion, so for those we do want a dedicated process with its own Kafka connection.

To accommodate these different use cases, we came up with the concept of a “stage” service that can host one or more “actors”. An “actor” is our basic building block for reactive components. Each actor is a python class that derives from this abstract base class:

class Actor(object):
 def topics(self):
 """ Return a list of the topic(s) this actor cares about. """
 raise NotImplemented

def interval(self):
 """ Return the batching interval for this actor. This is the maximum
 interval. If another actor on the same stage has a shorter interval,
 then the batching interval will match that interval.
 """
 return 30

def process(self, topic, messages):
 """ Called periodically for this actor to process messages that have been
 received since the last batching interval. If messages for multiple
 different topics have been received, then this method will be called
 once for each different topic. The messages will be passed as an array
 of tuples (offset, message).
 """
 raise NotImplemented

@property
 def log(self):
 return getLogger(self.__module__)

All that is required for an actor class to override is topics() and process(). The topics() method simply returns a list of Kafka topics that the actor wants to handle, and the process() method is then called periodically by the stage service with a set of messages from one of these topics. The stage service works by collecting a batch of messages (1000 by default) across all the topics that all the actors within that stage care about, and then invoking each actor’s process() method with the messages in the topics that that actor cares about. If the batching interval expires while the stage is collecting messages, then the messages that have already been collected are processed immediately.

Once an actor is defined, it has to be configured to run within a specific stage. We are using a simple INI-style config file using betterconfig to define the various stages. Each stage is a section in the config file and the actors are specified by adding the python dotted path to the actor class to a list inside the section. In addition, the batch size for the stage can be changed here too.

We are still in the middle of the process of converting the functionality in our legacy platform to Reaxial, but we have already defined 30 actors running on 7 different stages. Having the infrastructure to easily decompose a feature into reactive components like actors improves the modularity and reliability of our system, and also improves testability. We can very easily write unit tests that pass specific messages to an actor and by mocking out the methods that the actor calls, we can test arbitrary scenarios without having to set up anything in the database. Plus, because actors only implement one feature, or one piece of a feature, they are straightforward unit testing targets.

One obvious area for improvement is to enhance the stage service so that it dynamically decides which actors to run on which stages by observing their behavior. This has always been in our plans, but because it is a complicated optimization problem and carries significant risks if not implemented properly, we decided to stick with the manual stage configuration for now, coupled with monitoring of the stages to ensure that time-sensitive messages are being handled within the expected time. So far this is working well, and as we improve this system we’ll keep you updated on our progress.

Reaxial – A reactive architecture for Axial

Software engineering is hard. Even a small software project involves making countless trade-offs between the ideal solution and “good enough” code. Software engineering at a startup is even harder because the requirements are often vague and in constant flux, and economic realities force us to release less-than-perfect code all the time.

Over time these decisions pile up and the technical debt becomes overwhelming. Axial has hit this point a few times. Until about two years ago, Axial’s platform was a single monolithic Django application that was becoming increasingly bloated, slow and unmaintainable. At that time, the decision was made to “go SOA” and we started to decompose this Django application into smaller services mostly based on Flask, and, more recently, Pyramid.

Some of the services that we’ve broken out since then are small and focused and make logical sense as independent services, but others turned out to be awkward and inefficient and resulted in brittle and tightly-coupled code. Further, it has become clear that our current architecture does not align well with the features and functionality in our roadmap, such as real-time updates to our members. We realized that we needed a new architecture. Thus was born Reaxial.

The design keywords for Reaxial are reactive, modular and scalable. Reactive means that the system responds immediately to new information, all the way from the backend to the frontend. Modular means that parts of the system are decoupled, making it easier and faster to develop and test (and discard) new features without disturbing existing code. Scalable means that as we add members, we can simply add hardware to accommodate the additional load without slowing down the site.

To achieve these goals, we knew that we needed to use some sort of messaging middleware. After researching the various options out there, including commercial solutions like RTI Connext and WebSphere, and open-source packages like RabbitMQ, nanomsg and NATS, we settled on Apache Kafka. Kafka was developed by LinkedIn and offers a very attractive combination of high throughput, low latency and guaranteed delivery. LinkedIn has over 300 million users, which is an order of magnitude more than we ever expect to have to support, so we are confident that Kafka will scale well as we grow. Further, because Kafka retains messages for a long time (days or weeks), it is possible to replay messages if necessary, improving testability and modularity. With Kafka as the underlying message bus, the rest of the architecture took shape:

Reaxial Architecture

Probably the most important new service is the entity service. The entity service handles CRUD for all top-level entities, including classes like Company, Member and Contact, among many others. Whenever an entity is created or updated, a copy of it is published on the message bus, where it can be consumed in real-time by other services. Simple CRUD does not handle the case where multiple entities need to be created or updated in a transaction, so to handle that the entity service also offers a special API call, create_entity_graph, that can create and update a set of related entities atomically. In the Reaxial architecture, most features will be implemented as a service that subscribes to one or more entity classes, and then reacts to changes as they occur by either make further updates to those entities or by creating or updating some other entity.

Recall that our design goals were to enable real-time updates all the way to the member. To accomplish this, we created a subscription service that uses SockJS to support a persistent bidirectional socket connection to the browser. This service, written in NodeJS, subscribes to changes in all the entity classes and allows the browser to subscribe to whatever specific entities on which it wants updates, and for which the user session is permissioned to see, of course.

We have deployed these components to production and are just starting to use them for some new features. As we gain confidence in this new infrastructure, we will slowly transition our existing code base to the Reaxial model. We decided to deploy Reaxial gradually so that we can start to reap the benefits of it right away, and to give us an opportunity to detect and resolve any deployment risks before we are fully dependent on the new architecture. We have a lot of work ahead of us but we are all quite excited about our modular, scalable and reactive future.

Validating JSON-RPC Messages

Here at Axial we have standardized on JSON-RPC 2.0 for our inter-service communications. We recently decided to start using JSON Schemas to validate our JSON-RPC 2.0 messages. JSON Schema (http://tools.ietf.org/html/draft-zyp-json-schema-04) is a flexible declarative mechanism for validating the structure of JSON data. In addition to allowing for definitions of simple and complex data types, JSON Schema also has primitives ‘allOf’, ‘anyOf’ and ‘oneOf’ for combining definitions together. As we will see, all three of these primitives are useful for validating even just the envelope of JSON-RPC 2.0 messages.

The simplest JSON Schema is just an empty object:

{ }

This JSON Schema validates against any valid JSON document. This schema can be made more restrictive by specifying a “type”. For example, this JSON Schema:

{ “type”: “object” }

would validate against any JSON document which is an object. The other basic schema types are: array, string, number, integer, boolean and null. Most of the basic schema type comes with additional keywords that further constrain the set of valid documents. For example, this JSON Schema could be used to validate an object that must contain a ‘name’ property which is a string:

{
    “type”: “object”,
    “properties”: {
        “name”: { “type”: “string” }
    },
    “required”: [ “name” ]
}

If “name” was not in the “required” list (or if “required” was not specified) then objects would not have to actually have a “name” property to be considered valid (however, if the object happened to have a “name” property whose value was not a string, then the object would not validate). Normally, objects are allowed to contain additional properties beyond those specified in the “properties” object; to change this behavior, there’s an optional “additionalProperties” keyword that can be specified with ‘false’ as its value.

The combining primitives ‘allOf’, ‘anyOf’ and ‘oneOf’ can be used to express more complicated validations. They work pretty much as you might expect; ‘allOf’ allows you to specify a list of JSON Schema types which all must be valid, ‘anyOf’ means that any one of the list of types must be valid, and ‘oneOf’ means that EXACTLY one of the list of types must be valid. For example:

{
    “anyOf”: [
        { “type”: “string” },
        { “type”: “boolean” }
    ]
}

would validate against either a string or a boolean.

It turns out that all three of these combining primitives are useful for accurately validating JSON-RPC objects. JSON-RPC objects come in two flavors; the request and the response. Requests have the following properties:

  • jsonrpc: A string specifying the version of the JSON-RPC protocol. Must be exactly “2.0”.
  • method: A string containing the name of the method to invoke. Must *not* start with “rpc.”.
  • id: An identifier for this request established by the client. May be a string, a number or null. May also be missing, in which case the request is a “Notification” and will not be responded to by the server.
  • params: Either an array or an object, contains either positional or keyword parameters for the method. Optional.

Responses must have these properties:

  • jsonrpc: As for the request, a string which must be exactly “2.0”.
  • id: This must match the ‘id’ of the request, but could be null if the request ‘id’ was null or if there was an error parsing the request ‘id’.

In addition, if the request was successful, the response must have a ‘result’ property, which can be of any type. Conversely, if the request was not successful, the response must NOT have a ‘result’ property but must instead have an ‘error’ property whose value is an object with these properties:

  • code: An integer indicating the error type that occurred.
  • message: A string providing a description of the error.
  • data: A basic JSON Schema type providing more details about the error. Optional.

Here’s a schema for validating JSON-RPC Request objects:

{
    "title": "JSON-RPC 2.0 Request Schema”,
    "description": "A JSON-RPC 2.0 request message",
    "type": "object",
    "properties": {
        "jsonrpc": { "type": "string", "enum": ["2.0"] },
        "id": { "oneOf": [
            { "type": "string" },
            { "type": "integer" },
            { "type": "null" }
        ] },
        "method": { "type": "string", "pattern": "^[^r]|^r[^p]|^rp[^c]|^rpc[^.]|^rpc$" },
        "params": { "oneOf": [
            { "type": "object" },
            { "type": "array" }
        ] }
    },
    "required": [ "jsonrpc", "method" ]
}

Note the use of “oneOf” in the definition of the ‘id’ and ‘params’ properties. In addition, the ‘pattern’ in the definition of the ‘method’ property could use some explanation. Recall above that methods starting “rpc.” are not allowed in JSON-RPC. JSON Schema provides for the use of regular expressions to constraint string types, and the pattern here matches all strings except those starting with “rpc.”. If you are familiar with Perl-compatible regular expressions, you probably are thinking that this pattern would be more elegantly written as “^(?!rpc\.)” which is certainly true, but for the widest compatibility, it is recommended to not use “advanced” features like look-ahead in JSON Schema patterns.

The JSON Schema for validating JSON-RPC Response objects is a bit more complicated:

{
    "title": "JSON-RPC 2.0 Response Schema",
    "description": "A JSON-RPC 2.0 response message",
    "allOf": [
        {
            "type": "object",
            "properties": {
                "jsonrpc": { "type": "string", "enum": ["2.0"] },
                "id": { "oneOf": [
                    { "type": "string" },
                    { "type": "integer" },
                    { "type": "null" }
                ] }
            },
            "required": [ "jsonrpc", "id" ]
        },
        {
            "oneOf": [
            {
                "type": "object",
                "properties": {
                    "result": { },
                    "jsonrpc": { },
                    "id": { }
                },
                "required": [ "result" ],
                "additionalProperties": false
            },
            {
                "type": "object",
                "properties": {
                    "error": {
                        "type": "object",
                        "properties": {
                            "code": { "type": "integer" },
                            "message": { "type": "string" },
                            "data": {
                                "anyOf": [
                                    { "type": "string" },
                                    { "type": "number" },
                                    { "type": "boolean" },
                                    { "type": "null" },
                                    { "type": "object" },
                                    { "type": "array" }
                                ]
                            }
                        },
                        "required": [ "code", "message" ]
                    },
                    "jsonrpc": { },
                    "id": { }
                },
                "required": [ "error" ],
                "additionalProperties": false
            } ]
        }
    ]
}

In this example, “allOf” is used to combine an object that defines the properties shared between error and success responses with “oneOf” two different objects that define the specific properties in either an error or success response. The “additionalProperties” property setting is needed because it is an error to supply both ‘result’ and ‘error’ properties. Because of the use of “additionalProperties” it was also necessary to specify the “jsonrpc” and “id” properties in both the error and success responses, but no further type information is required there because their types are fully specified in the first “allOf” definition. Although the “anyOf” used in the definition of the “data” property of the error object could be replaced with just an empty schema, it is convenient here as it clearly documents what types are allowed.

As you can see, JSON-RPC envelopes are actually rather tricky to validate accurately, but JSON Schemas are flexible and generic enough to do the job. We also use JSON Schemas to validate the parameters to and return data from our JSON-RPC; look for the details of how that works and how it integrates with our RPC service definitions in a future blog post.

GLitter – A Simple WebGL Presentation Framework

Example GLitter Page

Example GLitter Page

A couple of weeks ago, I gave an Axial Lyceum talk on WebGL, a technology that allows hardware-accelerated 3D graphics to be implemented in the browser using pure javascript code. Before I came to Axial, I was doing a lot of 3D graphics work, which is how I came to learn WebGL. I had a moment of panic a few days before my talk when I realized it had been almost a year since I’d done any WebGL programming and I was feeling a little rusty. I was about to fire up powerpoint and start creating what probably would have been a boring presentation when I had a flash of inspiration– I could implement the slides for my WebGL talk directly in WebGL!

This both forced me to get back into the swing of WebGL and resulted in a much more engaging and interactive presentation. I used the fantastic Three.js library to make a simple framework for my presentation. After my talk, a few of the attendees asked if they could get a copy of the code that I used for the presentation, so I spent a little time making the code a bit more modular and it is now available at http://github.com/axialmarket/GLitter/. Before I explain how the presentation framework works, a little background on three.js is necessary.

WebGL is a very powerful technology, but it is also very complicated and has a steep learning curve. Three.js (http://threejs.org) makes creating 3D graphics with WebGL drastically simpler. While you still need some understanding of matrix mathematics and 3D geometry, Three.js abstracts away some of the highly technical details of WebGL programming like writing fragment shaders in GLSL or manipulating OpenGL buffers. In Three.js, you create a scene, a camera and a renderer object, and then use the renderer object to render each frame. To actually render something, you add 3D objects to the scene. These 3D objects have a geometry and materials, a transformation matrix that specifies translation, rotation and scaling relative to the object’s parent, and the ability to contain other 3D objects.

With GLitter, you define a “Page” object for each step in the presentation that creates the 3D objects and behaviors needed to implement that step. The GLitter “Scene” object manages the Three.js scene, camera and renderer and implements transition logic for switching between steps, and provides some common keypress handling functionality. One of the neat things about WebGL is that it renders inside of the HTML5 canvas so it is easy to composite the WebGL scene with HTML content overlayed on top. In GLitter, there are a few different types of HTML content you can overlay on top of the WebGL canvas. First, each Page provides a title and optionally some subtitle content. Second, GLitter uses the dat.GUI control framework to allow Page objects to easily add controls for properties of javascript objects. Lastly, GLitter provides an “info” interface that can be used to show dynamic content.

To see how this works in practice, let’s create a presentation with two steps. The first will show a spinning cube and provide controls to change the cube’s size, and the second will show a sphere with controls to change the camera’s field-of-view, aspect ratio and near and far clipping planes. On the first step, we will show the cube’s transformation matrix in the “info” overlay and in the second, we will show the camera’s projection matrix.

We first create GLitter Page objects CubePage and SpherePage:

var CubePage = new GLitter.Page({
    title: "Cube",
    subtitle: "Spinning!",
    initializor: function (scene) {
        var context = {};
        var cubeMaterial = new THREE.MeshLambertMaterial({ color: 0xee4444});
        var cubeGeometry = new THREE.BoxGeometry(1, 1, 1);
        context.cube = new THREE.Mesh(cubeGeometry, cubeMaterial);
        context.cube.position.z = -2.5;
        scene.add(context.cube);

        var spin = function() {
            new TWEEN.Tween(context.cube.rotation)
                     .to({y: context.cube.rotation.y - 2*Math.PI}, 2000)
                     .start()
                     .onComplete(spin);
        }
        spin();
        scene.add(new THREE.PointLight(0x999999));
        scene.camera.position.z = 2.5;
        return context;
    },
    finalizor: function() {
        GLitter.hideInfo();
    },
    updator: function (context) {
        return function (scene) {
            GLitter.showInfo(GLitter.matrix2html(context.cube.matrix));
            return ! context.STOP;
        }
    },
    gui: function (scene, context) {
        scene.gui.add(context.cube.scale, 'x', 0.1, 5);
        scene.gui.add(context.cube.scale, 'y', 0.1, 5);
        scene.gui.add(context.cube.scale, 'z', 0.1, 5);
    }
});
var SpherePage = new GLitter.Page({
    title: "Sphere",
    initializor: function (scene) {
        var context = {};
        var sphereMaterial = new THREE.MeshLambertMaterial({ ambient: 0xee4444 });
        var sphereGeometry = new THREE.SphereGeometry(1, 20, 20);
        context.sphere = new THREE.Mesh(sphereGeometry, sphereMaterial);
        context.sphere.position.z = -5;
        scene.add(context.sphere);

        scene.add(new THREE.AmbientLight(0x999999));
        return context;
    },
    finalizor: function() {
        GLitter.hideInfo();
    },
    updator: function (context) {
        return function (scene) {
            GLitter.showInfo(GLitter.matrix2html(scene.camera.projectionMatrix));
            return ! context.STOP;
        }
    },
    gui: function (scene, context) {
        var upm = function(){scene.camera.updateProjectionMatrix()};
        scene.gui.add(scene.camera, 'fov', 1, 179).onChange(upm);
        scene.gui.add(scene.camera, 'aspect', 0.1, 10).onChange(upm);
        scene.gui.add(scene.camera, 'near', 0.1, 10).onChange(upm);
        scene.gui.add(scene.camera, 'far', 0.1, 10).onChange(upm);
    }
});

The “title” and “subtitle” values are pretty self-explanatory. The “initializor” function is called to initialize the page when GLitter transitions to it. It adds the desired 3D objects to the GLitter Scene object, including lights, and returns a context object holding any objects that need to be referenced later. The “finalizor” function is called just before GLitter transitions away from this page. The “updator” function is called just before every frame is rendered. The “gui” function is used to update controls in scene.gui, which is a dat.GUI object.

Note that in CubePage, the spinning of the cube is handled by the TWEEN javascript library. TWEEN requires an update call to be made in every frame, but this is handled automatically by GLitter.

Also note that the “updator” functions return “! context.STOP”. The idea here is that if the “updator” function returns a false value, rendering is paused. The GLitter Scene object intercepts ‘keydown’ events, and will set context.STOP to true if Enter or Space is pressed. In addition, if “n” or “p” is pressed, GLitter transitions to the next or previous step, respectively. Page objects can add handling for other keypresses by defining an ‘onKeydown’ function. If this function returns a true value, then the standard GLitter keypress handling is skipped.

Now that these Page objects are defined, we can create an HTML page that sets up the basic structure GLitter needs and loads all of the required files. Currently GLitter is not a javascript module, so we load all of the files explicitly:

<!DOCTYPE html>
<html>
  <head>
    <title>GLitter Blog Example</title>
    <meta charset="utf-8"> 
    <link rel="stylesheet" type="text/css" href="example/example.css">
    <script src="//rawgithub.com/mrdoob/three.js/master/build/three.js"></script>
    <script src="lib/tween.js"></script>
    <script src="lib/dat.gui.min.js"></script>
    <script src="lib/OrbitControls.js"></script>
    <script src="lib/EffectComposer.js"></script>
    <script src="lib/MaskPass.js"></script>
    <script src="lib/RenderPass.js"></script>
    <script src="lib/ShaderPass.js"></script>
    <script src="lib/CopyShader.js"></script>
    <script src="lib/HorizontalBlurShader.js"></script>
    <script src="lib/VerticalBlurShader.js"></script>
    <script src="GLitter.js"></script>
    <script src="Page.js"></script>
    <script src="Scene.js"></script>
    <script src="example/CubePage.js"></script>
    <script src="example/SpherePage.js"></script>
    <script src="example/example.js"></script>
  </head>
  <body>
    <div id="content" class="content">
        <div id="title" class="title"></div>
        <div id="subtitle" class="subtitle"></div>
    </div>
    <div id="info" class="info">
    </div>
  </body>
</html>

The files in lib/ are third-party libraries, and GLitter itself is comprised of GLitter.js, Page.js and Scene.js. The contents of CubePage.js and SpherePage.js are shown above, so all that’s left is example.js:

window.addEventListener('load',
    function () {
        CubePage.nextPage = SpherePage;
        SpherePage.prevPage = CubePage;
        var scene = new GLitter.Scene({});
        scene.initialize();
        document.getElementById('content').appendChild(scene.domElement);
        scene.loadPage(CubePage);
    }
);

You can see the completed example at: http://axialmarket.github.io/GLitter/example.html

There’s still quite a bit of work to do to make GLitter more powerful and easier to use,  but as you can see, GLitter is already pretty easy to use, and doesn’t look too shabby either!

SPOKES: Towards Reusable Front-End Components

Hub and Bespoke

For some time here at Axial we’ve been migrating a large monolithic app to a set of small and simple services. One challenge that has come up in this process is how to share front-end components without unnecessarily coupling services together, and without imposing too many restrictions on how these front-end components can be implemented. The solution we’re evolving involves an abstraction we call a “spoke“.

What is a spoke?

A spoke is an executable javascript file that can contain javascript, CSS and HTML templates. In addition, a spoke can have dependencies on other spokes, which allows us to partition front-end components into small discrete chunks and at build time create a single loadable javascript file for a page that includes just the functionality we need. Now, you may be wondering how we embed CSS and HTML templates into an executable javascript file. We’re using a somewhat unsophisticated approach; we URI-encode the CSS or HTML content into a string and embed it in some simple javascript that decodes the content and dynamically adds it to the DOM.

A simple example

Let’s create a spoke for rendering a user’s name. This perhaps sounds like it’s too simple a task, but there could be some complexity to the logic required:

  • To save space, if the user’s full name would be more than 20 characters, we will render just their first initial followed by their last name.
  • If the user is an internal user, we want to annotate their name with (internal).
  • If the user is an internal user masquerading as a regular user, we want to annotate their name with (masq).

For this example, we will use a Backbone model and view, and an Underscore template, but these are implementation choices and not imposed on us just because we are creating a spoke.

Here is the Backbone model we will use:

var UsernameModel = Backbone.Model.extend({
    defaults: { first_name: "",
                last_name: "",
                is_internal: false,
                is_masq: false }
});

The view is pretty straightforward:

var UsernameView = Backbone.View.extend({
    className: 'username',
    render: function() {
        this.$el.html(this.template(this.model.attributes));
        return this;
    },
    template: _.template($('#username-template').html())
 });

We will store the Underscore template in a <script> tag with type “text/template”:

<script id="username-template" type="text/template">
    <% if (first_name.length + last_name.length >= 20) { %>
        <%= first_name.substr(0,1) %>.
    <% } else { %>
        <%= first_name %>
    <% } %>
    <%= last_name %>
    <% if (is_internal) { %>(internal)<% } %>
    <% else if (is_masq) { %>(masq)<% } %>
</script>

In addition, we have a CSS file to control the styling of the username:

.username {
    font-size: 18px;
    color: #333;
    white-space: nowrap;
}

To turn this into a spoke, all we have to do is store these source files in the spoke source tree:

js/models/Username.js
js/views/Username.js
html/username.html.tpl
css/username.css

Then we add a definition for this spoke (which we will call, surprise, surprise, “username”) to a spoke config in /etc/spoke/, for use by the “spoke compiler”, which is a python script spokec:

    # /etc/spoke/username.cfg
    [username]
    js     = [ 'models/Username.js', 'views/Username.js' ]
    html   = 'username.html.tpl'
    css    = 'username.css'
    spokes = 'backbone'

Spokes do not need to have all of these types of files; a spoke might contain only CSS or only javascript content. Note, also, that we have made the “username” spoke dependent on the “backbone” spoke. The definition of the “backbone” spoke in turn references the “underscore” spoke. When we use spokec to generate a spoke, these dependencies are followed and included in the output. As you probably anticipate, if a spoke is referenced multiple times, it only gets included in the output once.

Now that we’ve defined this spoke, here’s how we would call spokec to generate it:

spokec username [additional spokes] path/to/output.js

Each invocation of spokec generates a single executable javascript file containing all of the specified spokes and their dependencies. So typically a service will create a single spoke file for all of its pages, or sometimes a few different spoke files if the pages that service provides are significantly different. Currently we apply minification and fingerprinting to the spokes after generating them, but we will probably add this functionality directly to spokec soon.

Now, because we specified that “backbone” is a requirement for the “username” spoke, the resulting output is somewhat too large to paste here, but spokec has a feature that allows you to exclude specific dependencies from the generated spoke file by specifying them on the command-line prefixed with a ‘-‘. So, for example,

spokec username -backbone path/to/output.js

would create a spoke file with *only* the “username” spoke in it, which looks like this:

$("<style type='text/css'>").appendTo("head").text(decodeURIComponent(".username%20%7B%0A%20%20%20%20font-size%3A%2018px%3B%0A%20%20%20%20color%3A%20%23333%3B%0A%20%20%20%20white-space%3A%20nowrap%3B%0A%7D%0A"));
$("<div style='display: none'>").appendTo("body").html(decodeURIComponent("%3Cscript%20id%3D%22username-template%22%20type%3D%22text/template%22%3E%0A%3C%25%20if%20%28first_name.length%20%2B%20last_name.length%20%3E%3D%2020%29%20%7B%20%25%3E%0A%3C%25%3D%20first_name.substr%280%2C1%29%20%25%3E.%0A%3C%25%20%7D%20else%20%7B%20%25%3E%0A%3C%25%3D%20first_name%20%25%3E%0A%3C%25%20%7D%20%3E%0A%3C%25%3D%20last_name%20%25%3E%0A%3C%25%20if%20%28is_internal%29%20%7B%20%25%3E%28internal%29%3C%25%20%7D%20%25%3E%0A%3C%25%20else%20if%20%28is_masq%29%20%7B%20%25%3E%28masq%29%3C%25%20%7D%20%25%3E%20%0A%3C/script%3E%0A"));
var UsernameModel = Backbone.Model.extend({
   defaults: {
       first_name: "",
       last_name: "",
       is_internal: false,
       is_masq: false }
});

;
var UsernameView = Backbone.View.extend({
   className: 'username',
   render: function() {
       this.$el.html(this.template(this.model.attributes));
       return this;
   },
   template: _.template($('#username-template').html())
});
;

As you can see, the implementation of spokec assumes that jQuery is already included on the page, which for us is a practical assumption but would be easy to change if we wanted to. It should also be clear that the spoke abstraction makes very few other assumptions about how a specific spoke is implemented, as long as they can be represented as a series of javascript, CSS and HTML files. This allows us the flexibility to change the tools and libraries we are using but maintain a consistent and logical structure to our reusable components.

Getting Started

To start using spoke today, install it using pip:

sudo pip install spoke

The Future of Spokes

One type of content that we do not yet support in spokes is images; thus far we have just been using data URLs when we’ve needed to include images, but particularly for larger images this may become somewhat impractical. Another future enhancement we’ve been considering would make it much safer and easier to create reusable components by providing a way to automatically prefix CSS/HTML classes (and perhaps IDs) with a spoke identifier so that we can create very generic CSS class names without fear of a conflict with CSS classes in a different spoke. Doing this for CSS and HTML content is relatively straightforward, but to do so in javascript is a little trickier, although we have some ideas. Look for a future blog post once we’ve got solutions to this problem that we’re happy with!

Minimalist Backbone.js

Backbone.js is a popular tool for organizing front-end javascript code. It is used by well-known sites like Airbnb, Hulu and USA Today. It is also complex and full of features. Here at Axial, we have started building new services using a subset of Backbone to reduce the complexity but retain some of the key benefits.

The fundamental abstraction underlying Backbone is that data is stored in Models and HTML rendering and interactivity is handled by Views that may be backed by Models. This pattern not only helps organize front-end code, but its consistent use eliminates many common errors, especially errors where different representations of the same data are inconsistent.

For example, lets say we are creating a widget that has a bunch of checkboxes and also displays a count of the total number of selected checkboxes. Doing this the “old fashioned” way, we might add a change handler to the checkbox elements and then update the count whenever any checkbox changes:

HTML

<div class="checkboxWidget">
Count: <span>1</span><br>
<input type="checkbox" name="checkbox1" /><br>
<input type="checkbox" name="checkbox2" checked="checked" /><br>
...
</div>

JavaScript

$('.checkboxWidget input[type=checkbox]')
 .on('change', function() {
   $('.count')
     .html($('input[type=checkbox]:checked').length);
 });

This is pretty straightforward, but if some checkboxes are checked when the widget is initially loaded, then we need a separate piece of code to set the count on load:

$(
  function(){
    $('.count')
      .html(
        $('.checkboxWidget input[type=checkbox]:checked')
          .length
      );
   }
);

Now this is clearly still not very complicated, but once we have updates in more than one place, we have created an opportunity for them to get out-of-sync. Also, the HTML is probably being generated by the backend, which means the code for this widget is split into multiple files in multiple languages, with some of it executing on the server and some of it executing in the browser. For an example as simple as this, the complexity is not hard to manage, but it can quickly grow out of control.  Here’s how we might approach this in Backbone:

var model = Backbone.Model.extend({
  defaults: {
    'checkboxes': [],
    'selected': {}
  }
});

var view = Backbone.View.extend({
  initialize: function() {
    this.listenTo(this.model, 'change', this.render);
  },
  render: function() {
    // Backbone automatically creates a DOM element for
    // our view in this.el; this.$el is just this element
    // wrapped with jQuery.
    var $el        = this.$el,
        checkboxes = this.model.get('checkboxes'),
        selected   = this.model.get('selected');

    $el.html('Count: ' + _.size(selected) + '<br>');
    _.map(checkboxes,
      function (name) {
        var $checkbox = $("<input type='checkbox' name='" + name + "'>");
        if (selected[name]) {
          $checkbox.attr('checked', 'checked');
        }
        $el.append($checkbox).append($('<br>'));
      }
    );

    return this;
  },
  events: {
    'change input[type=checkbox]': 'checkboxChanged'
  },
  checkboxChanged: function (evt) {
    var selected = _.clone(this.model.get('selected')),
        $cb      = $(evt.target),
        name     = $cb.attr('name');

    if ($cb.is(':checked')) {
      selected[name] = true;
    } else {
      delete selected[name];
    }
    this.model.set('selected', selected);
  }
});

Ok, this might seem complicated, but it’s really not too bad. We have one model defined that contains a list of checkbox names and a ‘selected’ object whose keys are the checkbox names that are currently selected. The view takes this model and generates the corresponding HTML in the render() method. In this example, the HTML is created directly in render() using jQuery. Alternatively, any javascript templating system (e.g., Mustache, Handlebars or even just Underscore templates) could be used instead. The view listens for changes in the model, and automatically re-renders whenever there’s a change (that’s the reason we access the model attributes with .get() and .set(); it allows Backbone to keep track of changes). Because the count and the checkboxes are both being rendered by the same piece of code, with the same underlying data model, there’s no possibility that they could be out-of-sync.

If you are familiar with Backbone, you might be thinking that we could use Backbone.Collection for managing the set of selected checkboxes, since we could directly listen to changes in the collection instead of clone()ing the selected object every time it changes. That is certainly true, and probably not unreasonable, but I think in this case it would have added additional complexity that we don’t need.

Now, the class definitions above won’t do anything by themselves; we need to actually instantiate a view object and attach the rendered view to the page in order to see anything:

var checkboxModel = new model(
  {checkboxes: ['Checkbox 1', 'Checkbox 2'],
   selected: { 'Checkbox 2': true }}
);
$('body').append(
  new view(
    {model: checkboxModel }
  ).render().$el
);

We could have instead attached view.$el to the body inside the render() method, but delegating this responsibility to code outside of the views allows all the DOM to be generated and then rendered by the browser once. In this case, it doesn’t make much difference, but if we were creating lots of content, rendering only once would be an important optimization.

To really start to see the power of using a Model/View front-end pattern, let’s consider what would happen if we needed to dynamically add some checkboxes. Using the old-fashioned methodology, we might have a server template that returns a bunch of checkboxes that we request via AJAX and then add in to the ‘checkboxWidget’ class:

HTML

<input type='checkbox' name='checkbox3' checked='checked' />
<input type='checkbox' name='checkbox4' />
...

 

JavaScript

$.get('/more/checkboxes/html', function (data) {
  $('.checkboxWidget').append(data);
  // and we have to update the count...
  $('.count').html(
    $('.checkboxWidget input[type=checkbox]:checked').length
  );
});

With Backbone, this becomes simpler; we can have the server just feed us some JSON for the new checkboxes, and then we simply have to update the ‘checkboxes’ array and ‘selected’ object in the model:

JSON

{ 'checkboxes': ['Checkbox 3', 'Checkbox 4'],
  'selected':   {'Checkbox 3': true} }

JavaScript

$.get('/more/checkboxes/json', function (data) {
  checkboxModel.set('checkboxes',
    checkboxModel.get('checkboxes').concat(data.checkboxes));

  checkboxModel.set('selected',
    $.extend({},checkboxModel.get('selected')),data.selected));
  // that's it...we changed the model so the view will automatically
  // re-render with the new checkboxes and count!
});

Although we still have roughly the same number of lines of code, the conceptual overhead is reduced by using Backbone. When we want to change the data, all we have to worry about is changing the data; the view will then take care of correctly rendering the changed data. Because each view renders within a DOM element that it creates, we do not need to add extra markup like the ‘.checkboxWidget’ div, thus simplifying the markup. We don’t have to split knowledge about how the data is rendered between code running on the server and code running in the browser. And we can have the server just send JSON instead of HTML, which is simpler and more efficient.

Backbone includes a Routes facility for synchronizing models with data on the backend. We considered using this pattern at Axial, but we decided that it makes more sense for the models in the front-end to be separate from the models in the back-end. Instead of using a full REST interface, we simply defined a few JSON-RPC endpoints to deliver the data the front-end needs and wrote a simple javascript controller to make requests and marshall the data into our models.

The great thing about Backbone is it supports these usage patterns well: views don’t care how the models get their data, and models don’t care how the views are implemented. The great thing about Axial is developers are empowered to select the best tools for the job, and encouraged to find patterns that are efficient and pragmatic.