So I actually got my categories imported!
I ended up exporting each table that had anything to do with my categories from my old site through phpMyAdmin. Then I copied the INSERT statements from each SQL file individually and ran them on my new database.
There were a couple that required importing some other tables first, but if you get an error it will tell you which table it is calling on and then just import that first.
The only non-category table that I had to bring over was the product_entity table I believe. I forget the exact name, but when you try to import your other category tables it will let you know what it needs.
Not the easiest way to do it, but much faster than importing by hand, and relative simple to execute. Just takes a little time.
FYI, I imported my products through the admin panel after I got all of my categories in place.
Hope that helps someone else who is running into the same problem. Not sure how clear that answer was, but if you have any questions or need clarification just leave a comment and I will be happy to try and help you out.