Asynchronous method queue chaining in JavaScript

Thursday, May 6th, 2010

Chaining. It’s an extremely popular pattern these days in JavaScript. It’s easily achieved by continually returning a reference to the same object between linked methods. However one technique you don’t often see is queueing up a chain of methods, asynchronously, by which functions can be linked together independent of a callback. This discussion, of course, came from a late work night building the @anywhere JavaScript API with two other mad scientists, Russ D’Sa (@dsa) and Dan Webb (@danwrong).

Anyway, let’s have a look at some historical conventions and compare them to newer ones.

Imagine an iterator class that operated on arrays. It could look like this:

// no chaining
var o = new Iter(['a', 'b', 'c', 'd', 'e']);
o.filter(function(letter) {
  if (letter != 'c') { return letter; }
});
o.each(function(letter) {
  append(letter);
});

// with chaining
new Iter(alphabet).filter(remove_letter_c).each(append);

This is a simple because we’re working on a known existing object in memory (the alphabet array). However an easy way to spoil our soup is to make our methods continue to operate without existing objects. Like say, for example, a result set you had to make an async request to the server to get. Thus, imagine making this work:

ajax('/server/results.json').filter(remove_duplicates).append('div#results');

In the grand scheme of things, the above example isn’t too far off from from currying (which it’s not). And to make another point, currying can often lead to bad coupling of code… which in its defense, is often the point as well. Some libraries even call this pattern binding… so… yeah.

Anyway, to the point, here is a basic Queue implementation that can be used as a tool to build your own asynchronous method chains.

function Queue() {
  // store your callbacks
  this._methods = [];
  // keep a reference to your response
  this._response = null;
  // all queues start off unflushed
  this._flushed = false;
}

Queue.prototype = {
  // adds callbacks to your queue
  add: function(fn) {
    // if the queue had been flushed, return immediately
    if (this._flushed) {
      fn(this._response);

    // otherwise push it on the queue
    } else {
      this._methods.push(fn);
    }
  },

  flush: function(resp) {
    // note: flush only ever happens once
    if (this._flushed) {
      return;
    }
    // store your response for subsequent calls after flush()
    this._response = resp;
    // mark that it's been flushed
    this._flushed = true;
    // shift 'em out and call 'em back
    while (this._methods[0]) {
      this._methods.shift()(resp);
    }
  }
};

With this code, you can put it straight to work for something useful, like say, a jQuery plugin that fetches content remotely and then appends the results to your selector input. For you plugin developers our there, it would look like this…

<script src="jquery.js"></script>
<script src="async-queue.js"></script>
<script>
(function($) {

  $.fn.fetch = function(url) {
    var queue = new Queue;
    this.each(function() {
      var el = this;
      queue.add(function(resp) {
        $(el).html(resp);
      });
    });

    $.ajax({
      url: url,
      dataType: 'html',
      success: function(html) {
        queue.flush(html);
      }
    });
    return this;
  };

})(jQuery);
</script>

Then voila! You can make your DOM queries, fetch remote content, and continue your chain, asynchronously.

$("<div/>")
  .fetch('/server/navigation.html')
  .addClass('column')
  .appendTo('#side');

Here’s a brief example of showing off the example above. Point being, one can only imagine the possibilities you could do. Say for example, having multiple items in the queue waiting to operate on a response. Thus imagine this…

fetchTweet(url).linkify().filterBadWords().appendTo('#status');

Your internals would look like this with the Queue.

function fetchTweet(url) {
  this.queue = new Queue;
  this.tweet = "";
  var self = this;
  ajax(url, function(resp) {
    self.tweet = resp;
    self.queue.flush(this);
  });
}
fetchTweet.prototype = {

  linkify: function() {
    this.queue.add(function(self) {
      self.tweet = self.tweet.replace(/\b@(\w{1,20}\b/g, '<a href="...">$1</a>');
    });
  return this;
  },

  filterBadWords: function() {
    this.queue.add(function(self) {
      self.tweet = self.tweet.replace(/\b(fuck|shit|piss)\b/g, "");
    });
  return this;
  },

  appendTo: function(selector) {
    this.queue.add(function(self) {
      $(self.tweet).appendTo(selector);
    });
  return this;
  }

};

And with that, you can call it a night. Cheers.

17 Responses to “Asynchronous method queue chaining in JavaScript”

Rhys
May 6th, 2010 at 1:11 am

Nicely done. This should be really useful. I’m working on a site using the google maps api at the moment making ajax calls some of the time, but sometimes getting locally stored data. At the moment I’ve got different bits of code to deal with each scenario, but smoething like this should help cut atht down a bit.

Cheers
Michael Greenberg
May 6th, 2010 at 1:24 am

This is a basic form of FRP, or functional reactive programming. Your Queue structure will work great until you need to merge two events or create a cycle, when it will start to be “glitchy”.

Related work (which I worked on): Flapjax, an FRP library for JavaScript.
Dustin Diaz
May 6th, 2010 at 1:44 am

Rhys,
Glad it can be useful.

Michael,
I’m a little unclear on your point. Can you provide an example where this Queue class, in its simplest form, will break down? If you need to merge two events (or dispatchable actions), you could simply create a master queue, and you sub-queues will respond appropriately.

Nonetheless, I’m interested in understanding what you’re actually talking about.

An aside, nice work on Flapjax!
Vishal
May 6th, 2010 at 2:52 am

Hi Dustin,

Before I start, let me tell you, I am a huge fan of your Photography and have looked at your flickr photos(all of it – 11,361) twice :) and follow you too :)

My name is Vishal, PHP Developer! I am working on one of the sports website where we need to update live scores to the user’s browser. I did some research and found Polling, Long Polling and Streaming methods, but I couldn’t find any good examples or any resource on how to start working on it and how to implement it with PHP/Apache.

Would you mind sharing your thoughts on it? Any website or resource you know about those methods?

Many Thanks
Vishal
Peter van der Zee
May 6th, 2010 at 3:54 am

I think I prefer a callback parameter for oncomplete, onfail, onabort, onetc. The chained calls make it seem like they are executed synchronously rather than being pushed on a method stack to fire at some later point.

O.prototype.ease = function(target:object, oncomplete:function, onabort:function, options:object)

Then again, if you’re working in a library for which this is the de facto standard, it might not be a big issue.
Jim
May 6th, 2010 at 6:19 am

I like what you came up with, and have something like it of my own. In what I wrote I had need for different kinds of buffers at different times, so my queue involved a separate buffer system that could be instantiated as fifo, filo, etc and allowed me to do some other controlling of what was in the buffer. I also saw Crockford talking this sort of thing in one of his JS talks.

One thing though: Your first example of $.fn.fetch, where the queue only holds one call back which just appends the returned content to the matched elements, is exactly what $.fn.load does, correct? Differences are that yours could allow for full use of $.ajax options as well as multiple queued callbacks, but as you showed it it’s really a mirror of $.fn.load. Is that a fair comparison?

Jim
arnabc
May 6th, 2010 at 8:37 am

On a side note, how about using something like Promise API which is being implemented in CommonJS?

http://www.sitepen.com/blog/2010/01/19/commonjsjsgi-the-emerging-javascript-application-server-platform/

Thanks for the great post!

Cheers
John-David Dalton
May 6th, 2010 at 9:00 am

I am not sure the `fetch` example is the best use of Queue. Here is the same example without Queue with less code. http://dl.dropbox.com/u/513327/jquery-puts.html

Also you need the `new` operator to call `fetchTweet` as a constructor.
Michael Greenberg
May 6th, 2010 at 1:30 pm

Dustin,

Thanks! I only had a small hand in it; the real heroes are Greg Cooper, Leo Meyerovich, and Arjun Guha. Oh, and Shriram. :)

Glitches can occur if you have a diamond-shaped dependency graph: the output will update twice for a single update of the input. This is why Flapjax is very careful to propagate new events topologically.

But I spoke too soon. Your Queue only does one-shot sequencing of function calls; setting up a diamond-shaped graph with two queues feeding into a “master”, you have no choice but to wait until both sub-queues flush before calling any of the master queue’s methods. This is a topologically sorted evaluation of the graph, so there you go. When you want to allow repeated firings, glitches can be a problem.
Ajaxian » JavaScript asynchronous method queue chaining
May 6th, 2010 at 2:42 pm

[...] has written up a pattern that he used in Twitter @anywhere, the asynchronous method queue chaining pattern: One technique you don’t often see is queueing up a chain of methods, asynchronously, by which [...]
l.m.orchard
May 6th, 2010 at 5:21 pm

Seems like this could benefit a bit from something like Python’s decorators, to wrap all the queue methods in the queuing boilerplate.

Lacking that, some utility method to iterate through methods in a prototype to “queue-ify” them in function wrappers might be handy.

For what it’s worth, I’ve been playing with a different style of callback-queuing someone might find useful:

http://github.com/lmorchard/weave-webos/blob/master/src/javascripts/Decafbad/Chain.js

http://github.com/lmorchard/weave-webos/blob/master/src/tests/Weave/Service_Tests.js#L137
JavaScript asynchronous method queue chaining | DesignerLinks | Home to Web design news, jQuery Tutorials, CSS tutorials, Web Designing tutorials, JavaScript tutorials and more!
May 6th, 2010 at 8:53 pm

[...] has written up a pattern that he used in Twitter @anywhere, the asynchronous method queue chaining pattern: One technique you don’t often see is queueing up a chain of methods, asynchronously, by which [...]
Webstandard-Blog
May 6th, 2010 at 11:50 pm

Very nice solution Dustin and thx for sharing it!
Steve
May 7th, 2010 at 2:48 am

Great article! It’s good to see you blogging about JS again.
Matthew Podwysocki
May 7th, 2010 at 2:09 pm

We have a similar approach using the Reactive Extensions for JavaScript (http://msdn.microsoft.com/en-us/devlabs/ee794896.aspx) which treats events and asynchronous operations as collections you can subscribe to. This approach, following the Subject/Observer pattern, gives us the ability to richly handle when the next value is rendered through chaining as well as cancellation and error handling as well.

I was able to take the existing approach using the chaining methods and queues and instead use RxJS. This allows me to also have custom compensation logic should the operation fail:
http://gist.github.com/393968

You can find more examples of using RxJS here on my blog:
http://codebetter.com/blogs/matthew.podwysocki/
Introduction to the Reactive Extensions for JavaScript – Async Method Chaining - Matthew Podwysocki - CodeBetter.Com - Stuff you need to Code Better!
May 10th, 2010 at 8:22 pm

[...] there was a blog post by Dustin Diaz about method chaining over asynchronous operations. This topic, of course is near and dear to my heart as it strikes the exact chord of what the [...]
Ray
May 22nd, 2010 at 12:53 pm

Thats a sweet piece of code, I can instantly think of a few places to implement that or something similar.

Nice to see you blogging about scripting again, welcome back! It would be even nicer to see some of your screen-capture-as-you-code tutorials again with your unique blend of inane banter, learn while you laugh!

i am dustin diaz

Asynchronous method queue chaining in JavaScript

17 Responses to “Asynchronous method queue chaining in JavaScript”

find it

recent

me on twitter

this is who i am