How do I submit a form given only the HTML source?

Question

I would like to be able to submit a form in an HTML source (string). In other words I need at least the ability to generate POST parameters from a string containing HTML source of the form. This is needed in unit tests for a Django project. I would like a solution that possibly;

• Uses only standard Python library and Django.
• Allows parameter generation from a specific form if there is more than one form present.
• Allows me to change the values before submission.

A solution that returns a (Django) form instance from a given form class is best. Because it would allow me to use validation. Ideally it would consume the source (which is a string), a form class, and optionally a form name and return the instance as it was before rendering.

NOTE: I am aware this is not an easy task, and probably the gains would hardly justify the effort needed. But I am just curious about how this can be done, in a practical and reliable way. If possible.

Answer 1

Since the Django test framework does this, I'm not sure what you're asking.

Do you want to test a Django app that has a form?

• In which case, you need to do an initial GET
• followed by the resulting POST

Do you want to write (and test) a Django app that submits a form to another site?

Here's how we test Django apps with forms.

class Test_HTML_Change_User( django.test.TestCase ):
    fixtures = [ 'auth.json', 'someApp.json' ]
    def test_chg_user_1( self ):
        self.client.login( username='this', password='this' )
        response= self.client.get( "/support/html/user/2/change/" )
        self.assertEquals( 200, response.status_code )
        self.assertTemplateUsed( response, "someApp/user.html")

def test_chg_user( self ):
    self.client.login( username='this', password='this' )
    # The truly fussy would redo the test_chg_user_1 test here
    response= self.client.post(
        "/support/html/user/2/change/",
        {'web_services': 'P',
         'username':'olduser',
         'first_name':'asdf',
         'last_name':'asdf',
         'email':'[email protected]',
         'password1':'passw0rd',
         'password2':'passw0rd',} )
    self.assertRedirects(response, "/support/html/user/2/" )
    response= self.client.get( "/support/html/user/2/" )
    self.assertContains( response, "<h2>Users: Details for", status_code=200 )
    self.assertContains( response, "olduser" )
    self.assertTemplateUsed( response, "someApp/user_detail.html")

Note - we don't parse the HTML in detail. If it has the right template and has the right response string, it has to be right.

Answer 2

Check out mechanize or it's wrapper twill. I think it's ClientForm module will work for you.

Answer 3

It is simple... and hard at the same time.
Disclaimer: I don't know much about Python and nothing at all about Django... So I give general, language agnostic advices... If one of the above advices doesn't work for you, you might want to do it manually:

• Load the page with an HTML parser, list the forms.
• If the method attribute is POST (case insensitive), get the action attribute to get the URL of the request (can be relative).
• In the form, get all input and select tags. The name (or id if no name) attributes are the keys of the request parameters. The value attributes (empty if absent) are the corresponding values.
• For select, the value is the one of the selected option or the displayed text is no value attribute.

These names and values must be URL encoded in GET requests, but not in POST ones.

HTH.

Answer 4

You should re-read the documentation about Django's testing framework, specifically the part about testing views (and forms) with the test client.

The test client acts as a simple web browser, and lets you make GET and POST requests to your Django views. You can read the response HTML or get the same Context object the template received. Your Context object should contain the actual forms.Form instance you're looking for.

As an example, if your view at the URL /form/ passes the context {'myform': forms.Form()} to the template, you could get to it this way:

from django.test.client import Client
c = Client()

# request the web page:
response = c.get('/form/')

# get the Form object:
form = response.context['myform']

form_data = form.cleaned_data
my_form_data = {} # put your filled-out data in here...
form_data.update(my_form_data)

# submit the form back to the web page:
new_form = forms.Form(form_data)
if new_form.is_valid():
    c.post('/form/', new_form.cleaned_data)

Hopefully that accomplishes what you want, without having to mess with parsing HTML.

Edit: After I re-read the Django docs about Forms, it turns out that forms are immutable. That's okay, though, just create a new Form instance and submit that; I've changed my code example to match this.