Parse large XML file w/ script or use BioPython API ?

Question

Hey guys this is my first question on here. I'm trying to make a local copy of the UniprotKB in SQL.

The UniprotKB is 2.1GB, and it comes in XML and a special text format used by SwissProt

Here are my options:

1) Use a SAX parser (XML) - I chose Ruby, and Nokogiri. I started writing the parser, but my initial reaction: how would I map the XML schema to the SAX parser?

2) BioPython - I already have BioSQL/Biopython installed, which literally created my SQL schema for me, and I was able to successfully insert one SwissProt/Uniprot txt file into the database.

I'm running it right now (crosses fingers) on the entire 2.1gb. Here is the code I'm running:

from Bio import SeqIO
from BioSQL import BioSeqDatabase
from Bio import SwissProt

server = BioSeqDatabase.open_database(driver = "MySQLdb", user = "root", passwd = "", host="localhost", db = "bioseqdb")
db = server["uniprot"]
iterator = SeqIO.parse(open("/path/to/uniprot_sprot.dat", "r"), "swiss")
db.load(iterator)
server.commit()

Edit: it's now crashing because the transactions are getting locked (since the tables are Innodb) Error Number: 1205 Lock wait timeout exceeded; try restarting transaction. I'm using MySQL version: 5.1.43

Should I switch my database to Postgrelsql ?

Answer 1

Switched to PostgrelSQL for convenience purposes. Some of the issues were resolved by downloading the NCBI taxonomy information (which I did not know was necessary, should have been more clear in the documentation), so I ended up using the Swiss parser from BioPython because it fits so nicely with BioSQL.