tags:

views:

79

answers:

2
+1  Q: 

How to spider a password protected site in python?

currently I have a spider written in Java that logs into a supplier website and spiders the website. (using htmlunit)

It keeps the session (cookie) and even lets me enable/disable javascript etc.

I also use htmlparser (java) to help parse the html and extract the relevant information.

Does python have something similar to do this?

+4  A: 

Python has urllib2 to crawl pages, which supports password authentication and cookies.

There is also a HTMLParser for extracting html, but some people prefer the more feature-full BeatifulSoup.

Stephen  2010-07-04 01:43:35
very cool, i'm really getting excited by all things python!
Blankman  2010-07-04 01:46:27
What's _really_ cool is that it'll be about one-millionth of the amount of Java code you had to write ;)
Stephen  2010-07-04 01:49:33
indeed, that is exactly what I meant.
Blankman  2010-07-04 02:11:29
+1 for mentioning `BeautifulSoup`
Aviral Dasgupta  2010-07-04 02:11:51
before using BeautifulSoup check out lxml, its a **much** better/faster general parser, BeautifulSoup is good for pocket cases and munged HTML, it can also be embedded in the lxml API as well. If you go the BS route :) get version 3.0, version 3.1 is absolute junk.
ebt  2010-07-04 09:19:44
A: 

Scrapy API uses urllib2 plus adds wires up some different parsers and helper routines.

ebt  2010-07-04 09:16:13

related questions

Programmatically talking to a Serial Port in OS X or Linux
Best ways to teach a beginner to program?
Calling a Function From a String With the Function's Name in Python
An executable Python app
Text Editor For Linux (Besides Vi)?
What Hosting Service is best for Django applications?
File size differences after copying a file to a server vía FTP
Python: what is the difference between (1,2,3) and [1,2,3], and when should I use each?
Python: What OS am I running on?
How do I make a menu in python that does not require the user to press (enter) to make a selection?
How do you express binary literals in Python?
What is the most efficient graph data structure in Python?
Adding a Method to an Existing Object
How to learn Python: Good Example Code?
How do I use Python's itertools.groupby()?
Python and MySQL
Class views in Django
Is there an IDE that provides code completion for Python
Using 'in' to match an attribute of Python objects in an array
cx_Oracle - what is the best way to iterate over a result set?
cx_Oracle - How do I access Oracle from Python?
Continuous Integration System for a Python Codebase
Get a preview jpeg of a pdf on windows?
How can I find the full path to a font from its display name on a Mac?
XML Processing in Python