How to download a webpage in every five minutes ?

Question

I want to download a list of web pages. I know wget can do this. However downloading every URL in every five minutes and save them to a folder seems beyond the capability of wget. Does anyone knows some tools either in java or python or Perl which accomplishes the task?

Thanks in advance.

Answer 1

Write a bash script that uses wget and put it in your crontab to run every 5 minutes. (*/5 * * * *)

If you need to keep a history of all these web pages, set a variable at the beginning of your script with the current unixtime and append it to the output filenames.

Answer 2

Sounds like you'd want to use cron with wget

---

But if you're set on using python:

import time
import os

wget_command_string = "wget ..."

while true:
    os.system(wget_command_string)
    time.sleep(5*60)