While searching on how to escape a single quote in String.Format, I found the answer at SO: Escaping single quote in String.Format()
It seems to be different for VB though. I tested it, and indeed C# needs
string s = DateTime.Now.ToString("MMM d \\'yy 'at' H:mmm");
while VB needs
Dim s As String = Now.ToString("MMM d \'yy 'at' H:mmm")
Why does C# need a double backslash, and VB a single backslash? This might be a bit of a silly question to C# users, but while I can read C#, I'm not used to writing it.