How can I put all twill commands together into one piece of code in a .py file?

Question

Hello everybody!

I have just started exploring TWILL.

Twill is an amazing scripting language for Web browsing and it does all I want!!!

So far I've been using twill from a Python shell (IDLE (Python GUI) to be precise) and I do things there in the way of executing commands one by one (I mean, I type one command, run it, then type the next command): But I don't know how to put all these commands together in one .py file, so that they would all be executed one by one automatically.

It seems that there is such possibility in twill. This example from the twill documentation page (you can see it HERE) shows us one piece of code consisting of several commands: So, my question is: How can I put all commands together in twill?

---

Update 1:

(this update is my response to S.Mark)

Hello, S.Mark!!! I am sorry for the late response. First of all, some info about the location of my twill and python related folders:

The path where Python2.5 is installed on my computer: C:\Python25

The path to my twill-0.9 on my computer now: E:\tmp\twill-0.9

Let’s say I want the following commands to be carried out automatically:

go http://www.yahoo.com

save_html result.html

This code should look into yahoo page and then save its HTML code into result.html file. So, trying to follow Your instructions, I firstly created “test.txt” file containing this code consisting of only 2 lines and saved that file as “test.twill” in the twill-0.9 folder, which means that the full path to that file now was E:\tmp\twill-0.9\test.twill

Then I tried to pass the file name as parameter to twill-sh command in many different ways, but it never worked (I must’ve been doing something wrong): But you know what, I decided to experiment a bit and created a test.py file that also contained only those two commands. This file I also placed in the twill-0.9 folder (E:\tmp\twill-0.9\test.py) and then I decided to try running it from twill shell using twill’s runfile command, and, surprisingly, it worked! :

After running it, I looked up my C:\Python25 folder and found the newly-created result.html file there!

Well, what I've done here is simply running a file from the twill shell using a twill command. While at the moment it is exactly what I need, other supporters (as you can see below) suggest I should do all things from python shell, not from twill shell, and that is something that I still don’t know how to do.

My next step will be to try running a similar code on “Google App Engine”, but there, as far as I know, only Python is recognized, not twill, which means that if I only know how to do things in twill, but not in python, I won’t be able to have “Google App Engine” execute my commands.

---

Update 2:

(Friday 23, April, 2010, 3:48:15 a.m.(GMT+0.00))

(This update is my second response to S.Mark)

It seems that running it from command prompt isn't successful either:

Answer 1

I think that instead of using the twill shell, you should instead directly call the functions using the twill python api http://twill.idyll.org/python-api.html.

Answer 2

Put your twill commands into a file, for example test.twill

setlocal query "twill Python"

go http://google.com/

fv 1 q $query
submit btnI     # use the "I'm feeling lucky" button

show

And then just pass filename as parameter to twill-sh command, like

python twill-sh test.twill

And you might want to check .twill sample codes in tests folder of twill source

test-back.twill
test-basic.twill
test-dns.twill
test-equiv-refresh.twill
test-find.twill
test-form.twill
test-formfill.twill
test-global-form.twill
test-go-exit.twill
....

Answer 3

import string, re, sys, os
import twill.commands

class browser:
   def __init__(self, url="www.google.com", query="python code", log = None):
      self.a=twill.commands
      self.a.config("readonly_controls_writeable", 1)
      self.b = self.a.get_browser()
      self.b.set_agent_string("Mozilla/5.0 (Windows; U; Windows NT 5.1; en-GB; rv:1.8.1.14) Gecko/20080404 Firefox/2.0.0.14")
      self.log = log
      self.b.clear_cookies()
      self.url=url
      self.b.go(self.url)
      f = self.b.get_form("1")
#      self.log.debug("form is %s" % f)
      f["q"] = query
      self.b.submit()
      self.log.debug( "Links\n%s" % self.b.showlinks())
      self.log.debug( "Forms\n%s" % self.b.showforms())
      pageContent = self.b.get_html()
      self.log.debug("html is <<%s>>" % pageContent)

Answer 4

Here it is in action (changed a wee bit):

>>> import twill.commands
>>> import BeautifulSoup
>>> 
>>> class browser:
...    def __init__(self, url="http://www.google.com",log = None):
...       self.a=twill.commands
...       self.a.config("readonly_controls_writeable", 1)
...       self.b = self.a.get_browser()
...       self.b.set_agent_string("Mozilla/5.0 (Windows; U; Windows NT 5.1; en-GB; rv:1.8.1.14) Gecko/20080404 Firefox/2.0.0.14")
...       self.log = log
...       self.b.clear_cookies()
...       self.url=url
...    def googleQuery(self, query="python code"):
...       self.b.go(self.url)
...       #self.b.showforms()
...       f = self.b.get_form("f")
...       #print "form is %s" % f
...       f["q"] = query
...       self.b.clicked(f, "btnG")
...       self.b.submit()
...       pageContent = self.b.get_html()
...       soup=BeautifulSoup.BeautifulSoup(pageContent)
...       ths = soup.findAll(attrs={"class" : "l"})
...       for a in ths:
...          print a
... 
>>> t=browser()
>>> t.googleQuery("twill queries")
==> at http://www.google.ie/
Note: submit is using submit button: name="btnG", value="Google Search"

<a href="http://pyparsing.wikispaces.com/WhosUsingPyparsing" class="l" onmousedown="return clk(this.href,'','','res','1','','0CBMQFjAA')">pyparsing - WhosUsingPyparsing</a>
<a href="http://www.mail-archive.com/[email protected]/msg00048.html" class="l" onmousedown="return clk(this.href,'','','res','2','','0CBcQFjAB')">Re: [<em>twill</em>] <em>query</em>: docs, and web site.</a>
<a href="http://www.mail-archive.com/[email protected]/msg00050.html" class="l" onmousedown="return clk(this.href,'','','res','3','','0CBkQFjAC')">Re: [<em>twill</em>] <em>query</em>: docs, and web site.</a>
<a href="http://www.genealogytoday.com/surname/finder.mv?Surname=Twill" class="l" onmousedown="return clk(this.href,'','','res','4','','0CB4QFjAD')"><em>Twill</em> Genealogy and Family Tree Resources - Surname Finder</a>
<a href="http://a706cheap-apparel.hobby-site.com/ladies-cotton-faded-twill-le-chameau-breeks-42" class="l" onmousedown="return clk(this.href,'','','res','5','','0CCEQFjAE')">Ladies Cotton Faded <em>Twill</em> Le Chameau Breeks 42</a>
<a href="http://twill.idyll.org/examples.html" class="l" onmousedown="return clk(this.href,'','','res','6','','0CCMQFjAF')"><em>twill</em> Examples</a>
<a href="http://panjiva.com/Sri-Lankan-Manufacturers-Of/twill+capri" class="l" onmousedown="return clk(this.href,'','','res','7','','0CCcQFjAG')">Sri-Lankan <em>Twill</em> Capri Manufacturers | Sri-Lankan Suppliers of <b>...</b></a>
<a href="http://c586cheap-apparel.dyndns.ws/twill-beige-blazer" class="l" onmousedown="return clk(this.href,'','','res','8','','0CCoQFjAH')"><em>Twill</em> beige blazer</a>
<a href="http://stackoverflow.com/questions/2267537/how-do-you-use-relative-paths-for-twill-tests" class="l" onmousedown="return clk(this.href,'','','res','9','','0CCwQFjAI')">How do you use Relative Paths for <em>Twill</em> tests? - Stack Overflow</a>
<a href="http://mytextilenotes.blogspot.com/2010/01/introduction-to-twill-weave.html" class="l" onmousedown="return clk(this.href,'','','res','10','','0CC8QFjAJ')">My Textile Notes: Introduction to <em>Twill</em> Weave</a>
>>>  

I use ubuntu so I use the following to install BeautifulSoup and twill:

sudo apt-get install BeautifulSoup*  
sudo apt-get install python-twill*

How this helps

A