I have a (stupid) node.js server that is here for the sole purpose of calling a function on the data it's passed with each request and answer the result of that function. Here is the code I use:
var pageDown = require('./PageDown/Markdown.Sanitizer').getSanitizingConverter(),
http = require('http');
http.createServer(function (req, res) {
var data = "";
res.writeHead(200, {'Content-Type': 'text/plain'});
req.on('data', function (chunk) {
data += chunk;
});
req.on('end', function() {
res.end(pageDown.makeHtml(data));
});
}).listen(1337, '127.0.0.1');
console.log('HServer running at http://127.0.0.1:1337/');
I use this server from python with the following code (atm I'm just benchmarking so it's just a stress test):
#!/usr/bin/env python
# -*- coding: utf-8 -*-
import requests
for i in range(1, 100000):
r = requests.post('http://localhost:1337/', data="lol")
print i
print "I'm done :')"
My problem is that this way of getting things done is slow. I have a large database that needs to be processed with this javascript function and I'm looking for ways to make the above process quicker. So suggestions are welcome!
You are making no use of asynchronous design of node.js. In fact what You are doing IMHO is anti-pattern.
If You have to do calculations in node.js than server code is correct. But to leverage asynchronous architecture You should put many instances of node.js servers behind the load balancer. You can even run this instances on one machine to make use of multiple cores.
And to enable client to make profit out of that many instances You should make your post calls asynchronously too. To do it instead of requests use some async http client like http://www.tornadoweb.org/documentation/httpclient.html
It allows you to do computations in node.js instances in parallel.
If all your are doing is calling the node process to cleanup your HTML, why not just do that in python? I assume the markdown converter you are pointing to is this one: http://code.google.com/p/pagedown/source/browse/Markdown.Sanitizer.js. If that is the case, why not convert it to python (or find similar python code), and work with it in process. It seems nuts to make a network call to simply run this process since it will be much much slower than just doing it all in python. Here are some python markdown tools I was able to find: http://packages.python.org/Markdown/, https://github.com/trentm/python-markdown2.