client sends delayed FIN ACK (~500ms) to server

I have a node.js client (10.177.62.7) requesting some data from http rest service from server (10.177.0.1). Client is simply using nodejs http.request() method (agent=false). Client is on Ubuntu 11.10 box.

Why client sends FIN ACK after 475ms? Why so slow? He should send FIN ACK immediately. I have many situations like this. About 1% of whole traffic is request with delayed FIN ACK.

Cpu idle on the client is about 99%, so nothing is draining CPU.

How to debug this? What could it be? Is there any sysctl option I need to tune?

On screenshot 2nd column is the elapsed time between packets.

Link to bigger picture.

enter image description here

This behaviour is the Delayed ACK feature of RFC1122 TCP stack.

Normally you should add the TCP_QUICKACK option to your Linux TCP socket to disable delayed ACK but I think it is not obvious with JavaScript Node.js API (I only saw socket.setNoDelay for TCP_NODELAY option).

So your idea to apply a system-wide change on TCP stack seems good but I found no sysctl matching this socket option behaviour. Here is another full list with explanation.