proxy for scale, performance (to load external content)?

Question

I am sure answer for this question will be very subjective, I simply want to know what the options are out there (for building a proxy to load external contents).

Typically I used cURL in php and pass a variable like proxy.url to fetch content. Then make an AJAX call with Javascript to populate the contents.

EDIT: YQL (Yahoo Query language) seems a very promising solution to me, however, it has a daily usage limit which essentially prevents me from using it for large scale projects.

What other options do I have? I am open to any language, any platform, key criteria are: performance and scalability.

Please share your ideas, thoughts and experience on this topic.

Thanks,

Answer 1

Hello,

you dont need a proxy server or something else. Just create a cronjob to fetch the contents every 5 minutes (or whenever you want).

You just need to create a script that grabs the content from the web and saves it (to a file, a database, ...), which will be started by the cronjob.

If somebody requests your page, you just need to send the cached content out and do with it whatever you want to do.

I think scalability and performance will be no problem.

Answer 2

Depending on what you need to do with the content, you might consider Erlang. It's lightening fast, ridiculously reliable, and great for scaling.