That sounds like a good place to start for me. Max size: pick something sane, unless you want it to be valid to post a dissertation as an alt text - though it is probably possible. As for the regex to validate it's okay:
/^[^"<>&\\]{0,XXX}$/
where XXX is the maximum size you want. Or get rid of the {0,XXX}
altogether and replace it with *
to mean "zero or more". Syntax depends on language, of course.
Also found this, looked interesting:
http://www.cs.tut.fi/~jkorpela/html/alt.html
Update:
Yeah, you two make a good point. As long as the quotes used around the alt-text aren't themselves single-quotes, then they should be fine.
And as per other answers below, possibly also & and . Though you may need to be careful with how many slashes, whether they are before things that matter. And also, whether and such things are allowed in the text itself.