I want to request or better pull data from a 3rd party website and parse the information afterwords. Parsing is not the topic here. I am interessted in the automatical request. What are my options and how to handle the security thing (authentication, authorization).
I have a simple node server just using connect. I also use the npm module "request" to process my request to the 3rd party website. I have also a url and the user and password. But I have not much information than the 3rd party website is a JSP (XSP) Webapplication running on tomcat which places a JSESSIONID after login. I tryed to add the header "Authorization": "Basic" with user:password as byte64 which did not work. Any other options, ideas?
Thanks for any comments and tips..
First find the URL the sign-in form is POSTing to, along with the names of the username / password fields. With this information, you can simulate the process (using request):
var cookies = request.jar();
request({
url: 'http://whatever.com/ajax/signin',
method: 'POST',
jar: cookies,
body: {
username: 'user',
password: 'pass'
}
}, function(error, response, body) {
// now if you use {jar: cookies} on later requests, the session
// established by the simulated sign-in will be used.
});
In complicated cases, you might have to parse the page containing the sign in form and read out any CSRF tokens from the form to include them in the body of your simulated post.