writestream and express for json object?

Question

writestream and express for json object?

I might be out of depth but I really need something to work. I think a write/read stream will solve both my issues but I dont quite understand the syntax or whats required for it to work. I read the stream handbook and thought i understood some of the basics but when I try to apply it to my situation, it seems to break down.

Currently I have this as the crux of my information.

function readDataTop (x) {
    console.log("Read "+x[6]+" and Sent Cached Top Half");
    jf.readFile( "loadedreports/top"+x[6], 'utf8', function (err, data) {
        resT = data
    });
};

Im using Jsonfile plugin for node which basically shortens the fs.write and makes it easier to write instead of constantly writing catch and try blocks for the fs.write and read.

Anyways, I want to implement a stream here but I am unsure of what would happen to my express end and how the object will be received.

I assume since its a stream express wont do anything to the object until it receives it? Or would I have to write a callback to also make sure when my function is called, the stream is complete before express sends the object off to fullfill the ajax request?

app.get('/:report/top', function(req, res) {
    readDataTop(global[req.params.report]);
    res.header("Content-Type", "application/json; charset=utf-8");
    res.header("Cache-Control", "max-age=3600");
    res.json(resT);
    resT = 0;
});

I am hoping if I change the read part to a stream it will allievate two problems. The issue of sometimes receiving impartial json files when the browser makes the ajax call due to the read speed of larger json objects. (This might be the callback issue i need to solve but a stream should make it more consistent).

Then secondly when I load this node app, it needs to run 30+ write files while it gets the data from my DB. The goal was to disconnect the browser from the db side so node acts as the db by reading and writing. This due to an old SQL server that is being bombarded by a lot of requests already (stale data isnt an issue).

Any help on the syntax here?

Is there a tutorial I can see in code of someone piping an response into a write stream? (the mssql node I use puts the SQL response into an object and I need in JSON format).

function getDataTop (x) {
    var connection = new sql.Connection(config, function(err) {
        var request = new sql.Request(connection);
        request.query(x[0], function(err, topres) {
            jf.writeFile( "loadedreports/top"+x[6], topres, function(err) {
                if(err) {
                    console.log(err);
                } else {
                    console.log(x[6]+" top half was saved!");
                }
            });
        });
    }); 
};

javascript
sql-server
json
node.js
express

Answer 1

Your problem is that you're not waiting for the file to load before sending the response. Use a callback:

function readDataTop(x, cb) {
  console.log('Read ' + x[6] + ' and Sent Cached Top Half');
  jf.readFile('loadedreports/top' + x[6], 'utf8', cb);
};

// ...

app.get('/:report/top', function(req, res) {
  // you should really avoid using globals like this ...
  readDataTop(global[req.params.report], function(err, obj) {
    // setting the content-type is automatically done by `res.json()`

    // cache the data here in-memory if you need to and check for its existence
    // before `readDataTop`

    res.header('Cache-Control', 'max-age=3600');
    res.json(obj);
  });
});