I am new to Node would appreciate some advice on writing to an array. I have high frequency real-time data coming in, and each tick will be written to a single array.
Would it be beneficial to do this in an asynchronous way? If so, how can this be done?
Here is a trimmed down snippet of my code. The "filter" function is called when a new tick is received. The "storeWindowData" function then saves to the array. I presume this could be blocking under high load?
Appreciate any comments.
Regards, Ben.
var window_data = [];
module.exports = {
filter: function (data) {
this.storeWindowData(timestamp, ticker, content);
}
},
storeWindowData: function(timestamp, ticker, content){
// Check if we have seen this ticker before for this minute
if(window_data[timestamp] !== undefined && window_data[timestamp][0] !== undefined && window_data[timestamp][0] === ticker){
window_data[timestamp][1] = window_data[timestamp][1] + ',' + content;
} else {
window_data[timestamp] = [ticker, content];
}
}
};
An array is an in memory primitive (well, built in type, not particularly primitive).
How exactly could you, at a JavaScript level, async write to it?
Where would it be stored temporarily while waiting to be written? A second array? (See the problem? Writing to that array would cause two array writes, which would be twice as slow.)
You're going to have to identify a different bottleneck, or if this really is the problem (which I'm very, very sure that it's not), you have either too weak of a machine handling this, or you have an insanely high amount of data coming in.