web crawling performance in node.js vs C#

I'm writing an application that crawls over a long list of links , downloads pages, searching for html elements using xpath queries and stores some of the retrieved info in mysql database. I use multi-threaded solution to get the most of my servers and eliminate the effect of latency.

I write most of my applications using csharp and java, and my web applications I write with asp.net/c#.

What I want to ask is whether or not Node.js is something worth considering from performance view point? given the throughput is the most important factor. Node.js being more portable and cross platform is another reason, but performance is more important for me.

I think the bottleneck in your application is on the network (HTTP or MySQL), rather than on the code.

Up or down, Node.js is single-threaded and based on message queue, when all "messages" are completed, it quit. Every I/O is waited on thread pool (unless you use sync methods, which is not encouraged).

Few things on your case:

  • You don't have simple and full controls on processors/threads, but you can always limit number of simultaneous job yourself
  • You don't need to worry about concurrency because there is no concurrency at all
  • All APIs in Node.js are designed with streamlined I/O in mind, you may only write 20-50% of code (in terms of lines) to write a web crawler, than in C#
  • Node.js has no XML library built-in, you can find a few on NPM
  • JavaScript is an interpreted language, code is slower than C#. But since your are more on I/O, I think it shouldn't slow you down too much
  • Debugging JavaScript on Node.js is painful, but web crawling should not be very complicated either

I have wrote a few simple web crawlers on Node.js and would definitely recommend you to give it a try.