How can I get all HTML pages from a website subfolder with Perl?

Question

Can you point me on an idea of how to get all the HTML files in a subfolder and all the folders in it of a website? For example: www.K.com/goo

I want all the HTML files that are in: www.K.com/goo/1.html, ......n.html

Also, if there are subfolders so I want to get also them: www.K.com/goo/foo/1.html...n.html

Answer 1

Read perldoc File::Find, then use File::Find.

Answer 2

I would suggest using the wget program to download the website rather than perl, it's not that well suited to the problem.

Answer 3

Look at lwp-mirror and follow its lead.

Answer 4

Assuming you don't have access to the server's filesystem, then unless each directory has an index of the files it contains, you can't be guaranteed to achieve this.

The normal way would be to use a web crawler, and hope that all the files you want are linked to from pages you find.

Answer 5

There are also a number of useful modules on CPAN which will be named things like "Spider" or "Crawler". But ishnid is right. They will only find files which are linked from somewhere on the site. They won't find every file that's on the file system.

Answer 6

You can also use curl to get all the files from a website folder. Look at this man page and go to the section -o/--output which gives u a good idead about that. I have used this a couple of times.