I am using log4js in my code to log the results and errors. The program runs for about 2,5 hours before the final console output is made and afterwards needs several hours to complete writing the logfile. The log is writing for 6 hours now (since the algorithm itself finished) and the filesize is 100mb. The log will be about 1,5 million lines (when done).
Is it normal for the log to be written as slow as this? Are there "standard" mistakes to make when using log4js that I could check?
In case you want to know: The program is running on an Intel i5 with 8gb RAM and an SSD drive, so the hardware shouldn't be the problem I guess.
I am not sure what other information I can give you, just ask ahead if you need to know something.
Dropbox sounds like a good candidate. Any anti virus software could also interfere.
Firstly I would confirm what your system is capable of by creating a mini log4js benchmark for the various configurations available on your PC, then compare that to your application performance.
var Benchmark = require('benchmark');
var log4js = require('log4js');
log4js.clearAppenders();
log4js.loadAppender('file');
log4js.addAppender(log4js.appenders.file('NUL'), 'nulnulnul');
var lognul = log4js.getLogger('nulnulnul');
log4js.addAppender(log4js.appenders.file('c:/your_dropbox/test.log'), 'normallog');
var lognorm = log4js.getLogger('normallog');
log4js.addAppender(log4js.appenders.file('c:/tmp/test.log'), 'nodropbox');
var lognodr = log4js.getLogger('nodropbox');
log4js.addAppender(log4js.appenders.file('c:/virus-exception/test.log'), 'nodropvir');
var lognodv = log4js.getLogger('nodropvir');
var suite = new Benchmark.Suite;
// add tests
suite.add('Log#Nul', function() {
lognul.info("Some lengthy nulnulnul info messages");
})
.add('Log#normal', function() {
lognorm.info("Some lengthy normallog info messages");
})
.add('Log#NoDropbox', function() {
lognodr.info("Some lengthy nodropbox info messages");
})
.add('Log#NoVirusOrDropbox', function() {
lognodv.info("Some lengthy nodropvir info messages");
})
// add listeners
.on('cycle', function(event) {
console.log(String(event.target));
})
.on('complete', function() {
console.log('Fastest is ' + this.filter('fastest').pluck('name'));
})
// run async
.run({ 'async': false });
If Dropbox or Virus software doesn't turn out to be the problem there are two Windows Sysinternal tools that will help you see what is going on your system when your process is running.
Process Explorer - Overall task manager/performance viewer
Gives you an overall view of your system so you can see which processes are doing what. You can also drill down into specific processes as well (right click/properties)
Process Monitor - Event profiler for processes.
Process Monitor is like a log file of all the system calls any process makes. You can filter down to specific processes or calls so in your case you would be able to monitor Dropbox and your Node.js process and see if their access to the file in question is interleaved while Dropbox does it's work.