I am streaming an xml file from S3. I need to build a new xml file with a different structure for sphinx search engine. I am already streaming the file from S3, piping it into my SAX parser but now I need to figure out how I can make modifications to the stream (after the SAX parser) and upload to S3.
parser.on('startElement', function(name, attrs) {
// Do something
});
I found what seems to be a great S3 library that streams called knox, so I am currently using that library. I'm not stuck on this library, just what I found that seems to be decent. The code that they have to stream data to S3 in the example, is only from an HTTP request. I am relatively new to streams, since I have a PHP background.
Knox Example Stream:
http.get('http://google.com/doodle.png', function(res){
var headers = {
'Content-Length': res.headers['content-length']
, 'Content-Type': res.headers['content-type']
};
client.putStream(res, '/doodle.png', headers, function(err, res){
// Logic
});
});
I am thinking I would need to do something on the lines of this.
parser.on('startElement', function(name, attrs) {
var headers = {
'Content-Length': res.headers['content-length']
, 'Content-Type': res.headers['content-type']
};
client.putStream(res, '/doodle.png', headers, function(err, res){
// Logic
});
});
Any help is greatly appreciated. Thanks.
This discussion of Node's new streams speaks explicitly about transforming streams from S3.
NB: This is in reference to streams as implemented in Node 0.10.x
https://www.npmjs.org/package/through
This module will do what you need.