Using http_referer
(sic) to validate anything is unsafe as some users turn this off. If your files are available on the web... well, they're available on the web. Lot's of people spend lots of time and money pondering the question of how restrict audio, to little effect. Without significantly more investment of time, I wouldn't worry to much about it.
Besides, here are the headers firefox sends when Flash requests an MP3:
GET /music.mp3 HTTP/1.1
Host: myhost:80
User-Agent: Mozilla/5.0 (Windows; U; Windows NT 6.0; en-GB; rv:1.9.2.2) Gecko/20100316 Firefox/3.6.2 GTB7.1 (.NET CLR 3.5.30729)
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: en-gb,en;q=0.5
Accept-Encoding: gzip,deflate
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
Keep-Alive: 115
Connection: keep-alive
Nothing identifying Flash as the source of the request. Out of luck.