Using Java (HTMLUnit) to Scrape a Web Page then Write Results to a mySQL Database

Question

I've been able to use Java and HTMLUnit to scrape a web page, however I'm unsure how I can get this to write the resulting data to a MySQL database on a remote hosted server. I also need this to happen at regular intervals (perhaps once a day) if possible, without having to manually run the program.

Is this possible and can I have any guidance, thorough or not, on how to do this?

Thanks!

Answer 1

Get your page's HTML into some String variable

String scrapedHtml = "";

Create a table in mysql with a text field

create table html_store (
     id int primary key autoincrement,
     content text not null
);

Use JDBC code to connect to the database and inset the data. Set the connectionURL and other parameters correctly

Connection conn = DriverManager.getConnection(connectionURL, user, password);
PreparedStatement pstmt = 
    conn.prepareStatement("insert into html_store (content) values (?)");
pstmt.setString(1, scrapedHtml); 
pstmt.executeUpdate();
pstmt.close();
conn.close();