My node.js app is generating a lot of text logs (100 per user per day). I have 10k active users daily, about 100k logs each day for my app to store. Format something like this:
time | log_type_id | action_id | message | user_ip
Cannot archive. Most of this logging data must be available for display to it's user.
Question is what's the fastest, most efficient disk space and performance wise way to store logs?
Bonus questions what's the best way to sort, analyze all of this data?
Question is what's the fastest, most efficient disk space and performance wise way to store logs?
Fastest, most efficient way to store logs? well, in sequentially written files of course...
Bonus questions what's the best way to sort, analyze all of this data?
Best to use a relational database for this.
This highlights the tradeoff. If all you are doing is saving the logs for a human to review, you will use less space, and write faster, to log files. On the other hand, while log files are great for writing information they are not so great for digesting and analyzing it. In general relational databases will win out even though they are not as efficient storage-wise or performance-wise on writing.