How to add random delays between the queries sent to Google to avoid getting blocked in python

Question

Hi, I have written a program which sends more than 15 queries to Google in each iteration, total iterations is about 50. For testing I have to run this program several times. However, by doing that, after several times, Google blocks me. is there any ways so I can fool google maybe by adding delays between each iteration? Also I have heard that google can actually learn the timesteps. so I need these delays to be random so google cannot find a patter from it to learn my behavior. also it should be short so the whole process doesn't take so much. Does anyone knows something, or can provide me a piece of code in python? Thanks

Answer 1

First, Google probably are blocking you because they don't like it when you take too many of their resources. The best way to fix this is to slow it down, not delay randomly. Stick a 1 second wait after every request and you'll probably stop having problems.

That said:

from random import randint
from time import sleep

sleep(randint(10,100))

will sleep a random number of seconds (between 10 and 100).

Answer 2

Since you're not testing Google's speed, figure out some way to simulate it when doing your testing (as @bstpierre suggested in his comment). This should solve your problem and factor its variable response times out at the same time.

Answer 3

Also you can try to use few proxy servers for prevent ban by IP adress. urllib support proxies by special constructor parameter, httplib can use proxy too