+1  A: 

Go to the source explorer and delete the files directly in there.

Julien Lebosquain
When I try delete the old files I get an error about pending changes, but if I try undo the changes then I get this error:TF14059: Unable to process the pending changes requested. The set of pending renames would cause a name collision for $/Bespoke/Religion/dhc.Scanning/dhc.Scanning/GUI/GoodsOutwards.vb.
themaninthesuitcase
This worked but I had to remove the new files first then undo the changes on the old files before I could remove them from source control.
themaninthesuitcase