In a server-side application running on Tomcat, I am generating full HTML pages (with header) based on random user-requested sites pulled down from the Internet. The client-side application uses asynchronous callbacks for requesting processing of a particular web page. Since processing can take a while, I want to inform the user about progress via polling, hence the callbacks.
On server-side, after the web page is retrieved, it is processed and an "enhanced" version is created. Then this version has to go back to the user. Displaying the page as part of the page of the client-side application is not an option.
Currently, the server generates a temporary file and sends back a link to it. This is clearly suboptimal.
The next best solution I can come up with inolves creating a caching-DB that stores the HTML content together with its md5-sums or sha1-ids and then sends back a link to a servlet, with the hash-ID as an argument. The servlet then requests the site from the caching-DB.
Is there any better solution? If not, which DB-backend would you propose? I'm thinking of SQLite. Part of the problem to be solved is: how do I push a page <html> to </html> back to client side?