I am working on a rails application where users upload some videos. Apache with passenger is my server. The videos can be as big as 500MB or 1 GB. After the video is uploaded it is converted to mp4 and played.
I am using https://github.com/valums/file-uploader to upload the files. With this i can upload as large as a 500MB video file easily. I want to display the mp4 preview of the video as soon as the upload is complete. I am using ffmpeg to process convert the video to mp4. This conversion may take as long as 10+ minutes some times and my request times out after 5 mins due to the default apache timeout.
So, the problem here is the ajax upload keeps the request alive only till 5 minutes after the upload to complete. I need the request to stay alive for about 10-20 minutes after the file upload is complete. It would be great to know in case any other javascipt based solutions available to keep this request alive till the video processing completes and the preview can be shown to the user.
Keeping a blocking request until the video is transcoded can burden your servers. Most servers can handle 40-80 concurrent requests concurrently, depending on the memory footprint of your Rails stack, gems installed, and Apache server setting. If you allow 10 minutes requests, this means that 40 users who are uploading simultaneously could take down your service.
My suggestion is using Ajax to poll your Rails server every few seconds if the job is done transcoding. If it does, you can refresh the page or add some more javascript to initalize the video playback.
window.setInterval( function() {
$.get('video_path').done(function(data, code, xhr) {
location.reload(); // or some code for playing back the video
});
}, 2000);
Another similar solution could use a third party service like Pusher or PubNub which allow browsers to maintain persistent connection, and then allow your Rails server to notify the clients once an event is fired. Push services like these usually use Node.Js for better asynchronous handling and maintaing many open connections concurrently.
Third option would be to switch Apache to Puma and use Rails new streaming APIs. Puma is better suited to handling concurrency.
This sounds like an Apache config directive (KeepAliveTimeout and TimeOut).
KeepAliveTimeout 1200 # 60sec * 20 min
More info & syntax from Apache here: http://httpd.apache.org/docs/2.2/mod/core.html#keepalivetimeout
An alternative: can you modify your JS/AJAX to re-request every 2 mins., and check if a preview exists yet or not?
Two tomcat parameters (ATTENTION! Only available after Tomcat 7.0) allow you to keep the session alive while running your request. Also, only after the request is finished, the session timeout will start counting again:
org.apache.catalina.session.StandardSession.ACTIVITY_CHECK=true
org.apache.catalina.session.StandardSession.LAST_ACCESS_AT_START=false
You can set both in server.xml or pass them through JAVA_OPTS:
-XX:MaxPermSize=252m -Dorg.apache.catalina.session.StandardSession.ACTIVITY_CHECK=true -Dorg.apache.catalina.session.StandardSession.LAST_ACCESS_AT_START=false