Fastest way to get YouTube videos for over 100.000 songs

Question

What is the fastest way to get YouTube videos for over 100.000 songs? I'm currently using PHP and SimpleXML to parse YouTube's feed, but it seems to be pretty slow. Any other ideas?

Answer 1

Yeah, you're probably in for a bit of a wait my friend.

Answer 2

A possible different way to approach this problem - the user-edited MusicBrainz database has links from artist and record labels to their YouTube channels. Details on the wiki here.

Note that this doesn't support direct song to video links though, and is for official videos only.

Answer 3

10000 YouTube songs is going to take a while. Consider this:

Time to download: 0.1 seconds
Time to parse: 0.05 seconds
Time to hit your database: 0.05 seconds
Total: 0.2 seconds
0.2 * 10000 = 2000 seconds
2000 seconds = 33 minutes

Granted, this is a rough estimate. If it's even close, though, you can see why it would take a while.

Here are some suggestions:

• Run your operation in a CRON so that you're not hitting any arbitrary browser/application timeouts
• Use caching wherever possible
• Try to perform batch downloads if possible
• Do preliminary error testing on the feed (i.e.: check for empty feeds) using Regex before you send it off to SimpleXML. Running simple validation in SimpleXML is a lot slower than using preg_match.

Otherwise, we'll need to see some code to give you a more thorough answer.

Hope this helps!