Node Kue and Child Process - get error from spawned process

Question

Node Kue and Child Process - get error from spawned process

I try to spawn a child process performing cpu intensive calculations through a job queue with Kue. My code at the moment looks like this:

consumer.js

var kue = require('kue');
var util  = require('util');
var spawn = require('child_process').spawn;

var jobs = kue.createQueue();

jobs.process('calc', 2, function(job, done){
  var work = spawn('Rscript', ['opti2.R', job.data.file]);

  work.stderr.on('data', function (data) {
    job.log('stderr: ' + data);
  });

  work.stdout.on('data', function (data) {
    job.log('stdout: ' + data);
  });

  work.on('exit', function (code, signal) {
    console.log('child process exited with code ' + code + ' with singal ' + signal);
    if(code != 0){
      done(****How to get the stderr of the child process as an error here***);
    } else {
      done(Error());
    }
  });
});

The code somewhat do what i would like it to do, but is there a better way to report the job as failed (to Kue) and get the stderr from the spawned process?

node.js
child-process
kue

Answer 1

You can use job.log method to send data directly to Kue.

I would also recommend you to switch from .spawn to .exec, because it returns stdout and stderr as strings in its final callback along with a good error, which suits your needs well:

var exec = require('child_process').exec;

jobs.process('calc', 2, function(job, done){
  exec('Rscript opti2.R ' + job.data.file, function (error, stdout, stderr) {
    if (stdout.length > 0) job.log('stdout: ' + stdout);
    if (stderr.length > 0) job.log('stderr: ' + stderr);
    done(error);
  });
});

Though solution should work with .spawn as well: simply replace each console.log call in your code with job.log.

Though, you may want to bufferize your stderr in order to send it to Kue in one chunk:

jobs.process('calc', 2, function(job, done){
  var work = spawn('Rscript', ['opti2.R', job.data.file]);
  var stderr = '';

  work.stderr.on('data', function (data) {
    stderr += data;
  });

  work.stdout.on('data', function (data) {
    job.log(data); // sending arriving `stdout` chunks as normal log events
  });

  work.on('close', function (code, signal) {
    console.log('child process exited with code ' + code + ' with singal ' + signal);
    if(code != 0){
      done(stderr); // sending all collected stderr as an explanation
    } else {
      done();
    }
  });
});

I would also recommend using close event instead of exit, because it waits for child's stdio streams.

For more information see Event: 'exit' docs:

This event is emitted after the child process ends.

Note that the child process stdio streams might still be open.

and Event: 'close' docs:

This event is emitted when the stdio streams of a child process have all terminated.