Using Mechanize with Ruby I get a certain file using agent.get('http://example.com/foo.torrent'), with FileUtils or otherwise, how do I save this file to my hard drive (for instance, in a directory wherefrom the script is running)? P.S. class => WWW::Mechanize::File ...
I'm trying to fetch some urls via urllib and mechanize through my proxy. With mechanize I try the following: from mechanize import Browser import re br = Browser() br.set_proxies({"http": "MYUSERNAME:*******@itmalsproxy.italy.local:8080"}) br.open("http://www.example.com/") I get the following error: httperror_seek_wrapper: HTTP Er...
I am aware of previous questions regarding mechanize + Google App Engine, What pure Python library should I use to scrape a website? and Mechanize and Google App Engine. Also there is some code here, which I cannot get to work on app engine, throwing File “D:\data\eclipse-php\testpy4\src\mechanize\_http.py”, line 43, in socket._fileo...
Hello, I understand a few people have asked about JS support in Mechanize. My question might be a little bit different, and/or posed differently. I am looking to ignore the Javascript check and log in to a page that requires Javascript. The web site itself does not really need JS, but the developers felt they needed to require it for th...
I want to screen-scrape a web-site that uses JavaScript. There is mechanize, the programmatic web browser for Python. However, it (understandably) doesn't interpret javascript. Is there any programmatic browser for Python which does? If not, is there any JavaScript implementation in Python that I could use to attempt to create one? ...
i am looking for a library that is similiar to mechanize for perl, ruby. basically i can do .click() and it will click on form buttons, or navigate to links. ...
Hello All Happy New Year! i have some question about python mechanize 's proxy support. im making some web client script, and i would like to insert proxy support function into my script. for example ,if i have such like following some script. how can i add proxy support into my mechanize script? i was look for some reference ,...
Here is a related question but I could not figure out how to apply the answer to mechanize/urllib2: http://stackoverflow.com/questions/1540749/how-to-force-python-httplib-library-to-use-only-a-requests Basically, given this simple code: #!/usr/bin/python import urllib2 print urllib2.urlopen('http://python.org/').read(100) This result...
I'm trying to use Webrat in a standalone script to automate some web browsing. How do I get the assert_contain method to work? require 'rubygems' require 'webrat' include Webrat::Methods include Webrat::Matchers Webrat.configure do |config| config.mode = :mechanize end visit 'http://gmail.com' assert_contain 'Welcome to Gmail' I ...
Have any of you guys/girls have used ruby's Mechanize library on a site that required SSL? The problem I'm experiencing at the minute is that when I try to access such a website the mechanize tries to use standard http protocol which results in endless redirections between http// and https:// ...
I am trying to parse information (html tables) from this site: http://www.511virginia.org/RoadConditions.aspx?j=All&r=1 Currently I am using BeautifulSoup and the code I have looks like this from mechanize import Browser from BeautifulSoup import BeautifulSoup mech = Browser() url = "http://www.511virginia.org/RoadConditions.aspx...
Hi My friend has a forum, which is full of posts containing information. Sometimes she wants to review the posts in her forum, and come to conclusions. At the moment she reviews posts by clicking through her forum, and generates a not necessarily accurate picture of the data (in her brain) from which she makes conclusions. My thought to...
Hi all I have just installed ruby+mechanize. It seems to me that it is posible in ruby nokogiri what I want to do but I do not know how to do it. What about this table? It is just part of html of vBulletin forum site. I tried to keep the html structure but deleted some text and tag attributes. I want to get some details per thread like...
Hello, I am trying to scrap some data from a website. The scripts I am trying to write, should get the content of the page: http://www.atpworldtour.com/Rankings/Singles.aspx Should simulate the user going trough every option for Additional Standings and the dates and simulate clicking on Go then after fetching the data should use the...
what is the best method using ruby/mechanize/nokogiri to go/click through all pages in case there is more than 1 page I need to access/click on? For example here Page 1 of 34 Should I click the page number or next? Or is out there any better solution? ...
I'm not very familiar using ruby with binary data. I'm using mechanize to download a large number of csv files to my local disk. I then need to search these files for specific strings. I use the save_as method in mechanize to save the file (which saves the file as binary). The content type of the file (according to mechanize) is: ap...
Looking for something similar to Mechanize for .NET C#. If you don't know what Mechanize is.. http://search.cpan.org/dist/WWW-Mechanize/ I will maintain a list of suggestions here. Anything for browsing/posting/screen scraping (Other than WebRequest and WebBrowser Control). Parsing HTMLAgilityPack - http://www.codeplex.com/htmlagil...
the html looks like below or you can find it here http://www.vbulletin.org/forum/index.php <!-- login form --> <form action="login.php?do=login" method="post" onsubmit="md5hash(vb_login_password, vb_login_md5password, vb_login_md5password_utf, 0)"> <script type="text/javascript" src="clientscript/vbulletin_md5.js?v=3612">...
Hi, I wrote a small program that uses Mechanize to traverse a site. I want to write tests for it, but don't want it to actually go log onto the site every time I run the tests. I would like to mock the internet, so that when it goes to some site, it simply returns the stored results. Here is a small example, pretend my code's purpose ...
In Python's mechanize.Browser module, when you submit a form the browser instance goes to that page. For this one request, I don't want that; I want it just to stay on the page it's currently on and give me the response in another object (for looping purposes). Anyone know a quick to do this? EDIT: Hmm, so I have this kind of working wi...