Python ClientForm Error

Question

Hello, check this snippet:

import ClientForm
from urllib2 import urlopen

page = urlopen('http://garciainteractive.com/blog/topic_view/topics/content/')
form = ClientForm.ParseResponse(page, backwards_compat=False)
print form[0]

The problem is that ClientForm parses the first html form the following way:

<POST http://garciainteractive.com/blog/topic_view/topics/content/ application/x-www-form-urlencoded
  <HiddenControl(ACT=1) (readonly)>
  <HiddenControl(RET=http://garciainteractive.com/blog/topic_view/topics/content/) (readonly)>
  <HiddenControl(URI=/blog/topic_view/topics/content/) (readonly)>
  <HiddenControl(PRV=) (readonly)>
  <HiddenControl(XID=d840927d4eaf95cef7aeca789009fb3991f574da) (readonly)>
  <HiddenControl(entry_id=42) (readonly)>
  <HiddenControl(site_id=1) (readonly)>
  <CheckboxControl(save_info=[yes])>
  <CheckboxControl(notify_me=[yes])>
  <TextControl(captcha=)>
  <SubmitControl(submit=Submit) (readonly)>>

Thus, not finding name, email and url inputs. How can I fix it? TIA

Update: Actually, I'm not using ClientForm separately, but as a part of mechanize, thus would prefer a solution allowing to fix without rewriting mechanize code

Answer 1

The problem is likely that the HTML itself is invalid - for example it re-uses the id="comment_form" over and over again, while there is only supposed to be one id of a given name per document.

Your best solution would probably be to use BeautifulSoup to parse your urlopen page result first, then pretty-print it back into a string for ClientForm - this is likely to get rid of most of the rough edges and give ClientForm a better chance of doing its thing.

If this doesn't work, get a pretty-print of the result out and work out what kind of transform you'll have to do on the HTML to make the form very simple for ClientForm - by removing extraneous tags and cruft.

Answer 2

As Richard suggested use BeautifulSoup.

from BeautifulSoup import BeautifulSoup, SoupStrainer
from StringIO import StringIO
from urllib2 import urlopen
import ClientForm

url='http://garciainteractive.com/blog/topic_view/topics/content/'           

html=urlopen(url).read()
forms_filter=SoupStrainer('form',id="comment_form")
soup = BeautifulSoup(html,parseOnlyThese=forms_filter)
forms = ClientForm.ParseFile(StringIO(soup),"", backwards_compat=False)
forms[0]['name']='Kalmi'
forms[0]['email']='[email protected]'