Deleting them off of disk should do the trick. Where they exist will depend a bit on where you installed them. If you installed them for all users they will be at
%ProgramFiles%\Microsoft Visual studio 9.0\Common7\Ide\ProjectTemplatesCache
For a single user they will be at
%USERPROFILE%\Visual Studio 2008\Templates\ProjectTemplates
EDIT
If InstallVSTemplates fails, it is likely one of the following. The one I can't help with is some program randomly dumping files inside of the ProjectTemplatesCache directory.
The other more likely cause is there is a bad template in the ProjectTemplates directory. InstallVSTemplates really just goes through this directory, and unzips all of the files into the ProjectTemplatesCache directory. You will have to search through this directory and find the .zip file which contains the junk files being added to your project Once you find the file, delete it, delete ProjectTemplatesCache, and then re-run InstallVSTemplates.