how to find all the urls/ pages on mysite.com

Question

i have a website that i now support and need to list all live pages/ url's. is there a crawler i can use to point to my homepage and have it list all the pages/url's that it finds.

then i can delete any that dont make their way into this listing as they will be orphan pages/url's that have never been cleaned up?

I am using DNN and want to kill un-needed pages.

Answer 1

Since you're using a database-driven CMS, you should be able to do this either via the DNN admin interface or by looking directly in the database. Far more reliable than a crawler.

Answer 2

Back in the old days I used wget for this exact purpose, using its recursive retrieval functionality. It might not be the most efficient way, but it was definitely effective. YMMV, of course, since some sites will return a lot more content than others.