Technically, it's supposed to be just this:
tell application "TextMate"
set theResult to make new document
end tell
But I get the same error in Script Debugger. Creating a new document manually and getting the document via a script works fine. I'm going to say you found a bug in TextMate's Applescript implementation. You could go the GUI scripting route here (shamelessly copied from the Mac OS Automation site):
return do_menu("TextMate", "File", "New")
--> result: true and a window appeared in TextMate
on do_menu(app_name, menu_name, menu_item)
try
-- bring the target application to the front
tell application app_name
activate
end tell
tell application "System Events"
tell process app_name
tell menu bar 1
tell menu bar item menu_name
tell menu menu_name
click menu item menu_item
end tell
end tell
end tell
end tell
end tell
return true
on error error_message
return false
end try
end do_menu