ansaurus

Question

Answer 1

+3 A:

Probably the correct approach is to request permission from the owners of the site you are scraping.

Even if you use a proxy, there is still a big chance that requests coming through the proxy will end up blocked as well.

Daniel Vassallo 2010-01-12 16:05:01

Ask Permission?...well...I'd rather not ( it's a torrent listing site ). Besides, I don't think they would be able to un-block just my Google App without un-blocking ALL Google Apps. Maybe they block Google Apps because other people (not me) write obnoxious Google App bots that hit their servers too much. I would still like to try a proxy -- I just don't know how to go about it. ( Maybe I could run my own proxy at home for this single purpose? hmm...)

Nick Perkins 2010-01-12 21:16:15

how does it being a torrent site affect your ability to ask permission? Is there a policy on the site that you are breaking?

Peter Recore 2010-01-12 22:25:27

Answer 2

+1 A:

Have you considered changing the user-agent?

result = urlfetch.fetch(u,headers = {'User-Agent': "Mozilla/5.0"},allow_truncated=True)

The API will always append "AppEngine-Google;" to the user-agent, but this might work if the restriction is not based on a IP address range.

jbochi 2010-01-12 16:07:14

Thanks for the idea, but this did not work ( in this case ).

Nick Perkins 2010-01-12 21:04:30

Answer 3

A:

Hi,

I'm currently having the same problem and i was thinking about this solution (not yet tried) :

-> develop an app that fetch what you want -> run it locally -> fetch your local server from your initial

so the proxy is your computer which you know as not blocked

Let me know if it's works !

MrGoodFriend 2010-02-07 21:35:11

Yes, that would work...but sort of defeats the purpose of using App Engine ( not having to run your own server ).In the end, I just switched to another website ( pirate bay ) that does respond to requests from App Engine.( the result is http://nicksmovietorrents.appspot.com )

Nick Perkins 2010-02-08 02:05:00

Answer 4

A:

I'm currently having this same problem. Has anyone gotten this working?

MrGoodFriend, do your potential solution work? I am not exactly sure what you mean by " fetch your local server from your initial".

Andrew 2010-09-16 23:35:49

Answer 5

A:

Well to be fair, if they don't want you doing that then you probably shouldn't. It's not nice to be mean.

But if you really want to do it, the best approach would be creating a simple proxy script and running it on a VPS or some computer with a decent enough connection.

Basically you expose a REST API from your server to your GAE, then the server just makes all the same requests it gets to the target site and returns the output.

Swizec Teller 2010-09-16 23:45:04

Answer 6

A:

What you are talking about is a valid bug in app engine sdk. Have a look at http://code.google.com/p/googleappengine/issues/detail?id=544 for bug updates, and workarounds for java and python.

pranny 2010-10-27 06:59:47

ansaurus

tags:

views:

answers:

AppEngine fetch through a free proxy

related questions