I have task to do where there is about 15000 items. I do it with backbone and node.js socket.io.js.
I load those items only initially, calling fetch. The script freezes for few seconds, but the everything so far is working fast.
The updated of this collection are handled using socket, so I dont need to fetch all 15K items when I want updated data.
The reason I choose this way - to get fast user interaction, so user would just search for item and it quickly is dislplayed. Not like search and wait for ajax response.
But we discussed today and my boss said the data set might grow by 10 times. I saw other discussions where its not recommended to have 100K arrays.
But when I am not going to fetch this whole data set constanly, but instead small peaces with node, how do you think will I have problems?
If so maybe I should plan to do part of data loaded, like 10K items loaded. Those items can be grouped so lets say part of groups loaded, part of groups not loaded.
So when user searches, he either gets lucky and see results instanly, or he sees ajax spinner - and data loads from server.
It felt its awesome when I dont need ajax and everything works so fast, but am bit worried when he told it might be 10x bigger data set :)
Well your page load takes a few seconds right now - with a 10x data set increase, that will likely increase to many, many seconds, as well as increasing load on your server and DB. Every application and data set has it's specific use, and based on that it depends on whether or not to and how you should paginate your data. It's also risky to expose long executing methods as it makes the system more brittle and susceptible to DOS attacks. The only client-side solution I can see for this is to fetch the data in parts or in whole and store in it local storage, but then you have to deal with that security risk. It would likely be more difficult to implement than a standard AJAX search as well, so AJAX is likely your best bet in this situation.