javascript - Node.js: Best way to perform multiple async operations, then do something else?

Question

In the following code I am trying to make multiple (around 10) HTTP requests and RSS parses in one go.

I am using the standard forEach construct on an array of URIs I need to access and parse the result of.

Code:

var articles;

feedsToFetch.forEach(function (feedUri)
{   
        feed(feedUri, function(err, feedArticles) 
        {
            if (err)
            {
                throw err;
            }
            else
            {
                articles = articles.concat(feedArticles);
            }
        });
 });

 // Code I want to run once all feedUris have been visited

I understand that when calling a function once I should be using a callback. However, the only way I can think of using a callback in this example would be to call a function which counts how many times it has been called and only continues when it has been called the same amount of times as feedsToFetch.length which seems hacky.

So my question is, what is the best way to handle this type of situation in node.js.

Preferably without any form of blocking! (I still want that blazing fast speed). Is it promises or something else?

Thanks, Danny

Answer 1

No hacks necessary

I would recommend using the async module as it makes these kinds of things a lot easier.

async provides async.eachSeries as an async replacement for arr.forEach and allows you to pass a done callback function when it's complete. It will process each items in a series, just as forEach does. Also, it will conveniently bubble errors to your callback so that you don't have to have handler logic inside the loop. If you want/require parallel processing, you can use async.each.

There will be no blocking between the async.eachSeries call and the callback.

async.eachSeries(feedsToFetch, function(feedUri, done) {

  // call your async function
  feed(feedUri, function(err, feedArticles) {

    // if there's an error, "bubble" it to the callback
    if (err) return done(err);

    // your operation here;
    articles = articles.concat(feedArticles);

    // this task is done
    done();
  });
}, function(err) {

  // errors generated in the loop above will be accessible here
  if (err) throw err;

  // we're all done!
  console.log("all done!");
});

Alternatively, you could build an array of async operations and pass them to async.series. Series will process your results in a series (not parallel) and call the callback when each function is complete. The only reason to use this over async.eachSeries would be if you preferred the familiar arr.forEach syntax.

// create an array of async tasks
var tasks = [];

feedsToFetch.forEach(function (feedUri) {

  // add each task to the task array
  tasks.push(function() {

    // your operations
    feed(feedUri, function(err, feedArticles) {
      if (err) throw err;
      articles = articles.concat(feedArticles);
    });
  });
});

// call async.series with the task array and callback
async.series(tasks, function() {
 console.log("done !");
});

---

Or you can Roll Your Own?

Perhaps you're feeling extra ambitious or maybe you don't want to rely upon the async dependency. Maybe you're just bored like I was. Anyway, I purposely copied the API of async.eachSeries to make it easy to understand how this works.

Once we remove the comments here, we have just 9 lines of code that can be reused for any array we want to process asynchronously! It will not modify the original array, errors can be sent to "short circuit" the iteration, and a separate callback can be used. It will also work on empty arrays. Quite a bit of functionality for just 9 lines :)

// void asyncForEach(Array arr, Function iterator, Function callback)
//   * iterator(item, done) - done can be called with an err to shortcut to callback
//   * callback(done)       - done recieves error if an iterator sent one
function asyncForEach(arr, iterator, callback) {

  // create a cloned queue of arr
  var queue = arr.slice(0);

  // create a recursive iterator
  function next(err) {

    // if there's an error, bubble to callback
    if (err) return callback(err);

    // if the queue is empty, call the callback with no error
    if (queue.length === 0) return callback(null);

    // call the callback with our task
    // we pass `next` here so the task can let us know when to move on to the next task
    iterator(queue.shift(), next);
  }

  // start the loop;
  next();
}

Now let's create a sample async function to use with it. We'll fake the delay with a setTimeout of 500 ms here.

// void sampleAsync(String uri, Function done)
//   * done receives message string after 500 ms
function sampleAsync(uri, done) {

  // fake delay of 500 ms
  setTimeout(function() {

    // our operation
    // <= "foo"
    // => "async foo !"
    var message = ["async", uri, "!"].join(" ");

    // call done with our result
    done(message);
  }, 500);
}

Ok, let's see how they work !

tasks = ["cat", "hat", "wat"];

asyncForEach(tasks, function(uri, done) {
  sampleAsync(uri, function(message) {
    console.log(message);
    done();
  });
}, function() {
  console.log("done");
});

Output (500 ms delay before each output)

async cat !
async hat !
async wat !
done