I have a problem with a small project I've been doing. It is essentially a service allowing queries on a WFS server in a more restful manner. My problem is that the response I receive is complete gibberish, but only when I deploy my project on Red Hat OpenShift or Heroku. On Cloud9 everything works perfectly. Be aware that I've been using this project as a way to teach myself express and nodejs and that the coding might not be optimal.
My main problem is that the WFS server return the data in ISO-8859-1 with accented characters in french and I lose them when working on the response, they either become gibberish or the � character.
I tried two different ways to solve this, both return gibberish when deployed. For the first I extended Request with a collect
method to replace the error
, data
and end
event. Basically I query the WFS and if the format
parameter is passed I transform the response in json with the xml2js
node module.
req.pipe(request.get(app.get('baseUrl') + '&REQUEST=getcapabilities')).collect(function(err, body) {
if(err)
return next(err);
if(req.query.format && req.query.format.toLowerCase() == 'json'){
parseString(body,{mergeAttrs: true}, function(err, result){
if(err)
return next(err);
res.json(200, result);
});
}else if(!req.query.format){
res.type('application/xml; charset=utf-8');
res.send(200, body);
}else{
next(new Error('Le paramètre \'format\' n\'est pas du valide. La valeur acceptée est JSON.'));
}
});
and here is collect
, I got the idea from the iconv-lite
module.
request.Request.prototype.collect = function(cb){
var res = [];
this.on('error', cb);
this.on('data', function(chunk){ res.push(chunk); });
this.on('end', function(){
cb(null, Buffer.concat(res).toString('binary'));
});
return this;
};
While on Cloud9 it returns the data with no problem, on OpenShift I get this response:
¥UMoâ0½ó+¢\zj[ÔE@è¶BâK]íÍ2É4xÛ©=.ð±hKÕîºÍ)¶çͼyãw®62ÁX¡U÷¬5ÏPNʺáp>=¿¼üúí¼W½FÇ&+¼rN¸[ðºá
±hDZäCþ¢LØÈIAêvà SHeÛÒ~¥³ä[¯×.@ ÓYød¹±éËõE¤Mi6[ñ¯ñh^åòdïcÛÎdþ&:«,!
o´×pÏ]ÝðÁñ\ÜHãæa3"u{
t,´Á@½õõ·¯ËH'+5j¿mOñEÔZñ[( ñAMº¢Ð
ÇûÉæÃÅw6ë/îãédرI1Nèg¾¯ÌáÃmA iÛÿÄ-Èò%Öº¥E®LàÖhWtCâÖf7ÀÑ8L´,rØþd¬W;ÀZ!¥Ù«t`C˲A©Ê>t`Íà
báÁQ§@ïY'òI{ZmXkSú=ìÍÕÅàv¯j$ÎÐ=lÒo«V¿b¢%K[
PÄ÷]Ôl¥Q{Ø_-¥;gX
,ç¬ðt2ÓÅ,Ô°"wÖ=LéZ+ü@»L9RÝývF¤jíí;¤`¸ó#
#.«.(Ó´'XAQðÑÍtûDØßºvVpIÇÃÅPsðÃòÌ«ÚYæ<AÍÌ.+ä)µ(ÛÅ£»OâÇNªL~ °Ö'ÐC3õhz<1×`#
ÏjÏ·
W,°iH
á×DöX{ÆßuâÓAÜÓ¼úñë©Ü+§/½,ú±ì5þ¯
Êì
For the second way, after reading around I tried with the iconv-lite module to decode the response in UTF-8.
req.pipe(request.get({encoding: null, url:app.get('baseUrl') + '&REQUEST=DescribeFeatureType' + ((req.params.qTypeName) ? '&TYPENAME='+req.params.qTypeName : '' )}, function(err, response, body) {
if(err)
return next(err);
body = iconv.decode(new Buffer(body), "ISO-8859-1");
if(req.query.format && req.query.format.toLowerCase() == 'json'){
parseString(body,{mergeAttrs: true}, function(err, result){
if(err)
return next(err);
res.json(200, result);
});
}else if(!req.query.format){
res.type('application/xml; charset=utf-8');
res.send(200, body);
}else{
next(new Error('Le paramètre \'format\' n\'est pas du valide. La valeur acceptée est JSON.'));
}
})
);
I still get the same result as the first way I tried. While I normally try to solve my problems myself I think I hit a wall here. Like I said I am still learning about nodejs and express so it might be a really simple thing I did not account for. After reading around I suspect it has something to do with the encoding but what bug me is that it works fine on Cloud9 and it breaks only when deployed.
Sorry for the long wall of code and text but I wanted to be as clear as possible, English is not my main language so if you need clarifications just ask away.
Thanks!
Ok, it seems that after taking the time to write that big question I was able to figure out the problem. It was a remnant of old code I had forgotten, the req.pipe(...)
. I used to pipe the stream directly into res
before I decided to add the option to return json.
This way works well. Though it doesn't answer why it worked well in the Cloud9 development environment. I'll wait for a while before marking this as answer so I can maybe get insights from other people.
request.get({encoding: null, url:app.get('baseUrl') + '&REQUEST=DescribeFeatureType' + ((req.params.qTypeName) ? '&TYPENAME='+req.params.qTypeName : '' )}, function(err, response, body) {
if(err)
return next(err);
body = iconv.decode(new Buffer(body), "ISO-8859-1");
if(req.query.format && req.query.format.toLowerCase() == 'json'){
parseString(body,{mergeAttrs: true}, function(err, result){
if(err)
return next(err);
res.json(200, result);
});
}else if(!req.query.format){
res.type('application/xml; charset=utf-8');
res.send(200, body);
}else{
next(new Error('Le paramètre \'format\' n\'est pas du valide. La valeur acceptée est JSON.'));
}
});