Download Content From External URL and save in db with Ruby

Question

This isn't webservices. I want to pass a url to a controller and then have it fetch the html from that page. Then store the information in a db.

What do you think? How can I accomplish this?

Answer 1

yes hints: http://en.wikibooks.org/wiki/Ruby_Programming/Standard_Library/OpenURI

and then use some ORM or use the mysql drivers directly.

Answer 2

When I read your post, the first thing I thought of was Watir @ http://watir.com/

Watir is a family of Ruby libraries but it supports your app no matter what technology it is developed in. They support Internet Explorer on Windows, Firefox on Windows, Mac and Linux, Safari on Mac, Chrome on Windows and Flash testing with Firefox.

Like other programming languages, Ruby gives you the power to connect to databases, read data files and spreadsheets, export XML, and structure your code as reusable libraries. Unlike other programming languages, Ruby is concise and often a joy to read.

You can easily grab HTML and then populate it to a database, excel, etc.

Answer 3

In your controller:

html = %x[curl #{params[:url]}]

That will execute the system curl command and save the result (this is, the content extracted from the url) in the variable html. Then you can make hot cakes with that string if you want to.