I am trying to find the best solution of servicing a user of a HTTP service, that requires a huge amount of data at once. The idea is that we have an extremely big table which can get to around 80 million records - it might happen that some clients request the whole data (and yes, they need it all at once); the data gets updated only once a day, so I was thinking whether it's not better to generate the response offline and when the request comes in, the user gets the data asap. On the other hand, will a memcache setup for this particular request suit my needs ?
I currently have a huge amount of data that needs to be served asap from an 8 GB RAM database server and I am not sure what would be the best solution