Can you propose a more elegant way to 'tokenize' c# code for html formatting?

Answer 1

http://refactormycode.com

Answer 2

I'd use an active pattern to encapsulate the Regex.IsMatch and Regex.Match pairs, like so:

let (|Matches|_|) re s =
  let m = Regex(re).Match(s)
  if m.Success then
    Some(Matches (m.Value))
  else
    None

Then your nexttoken function can look like:

let nexttoken (st:String) =         
  match st with        
  | Matches "^s+" s -> Whitespace(s)        
  | Matches "^//.*?\r?\n" s -> Comment(s)
  ...