First, against the arguments in favor of building your own:
"Wanting to 'eat our own dog food' in terms of some internally built web framework"
That of course raises the question why build your own web framework. Just like there are many worthy free bug trackers out there, there are many worthy frameworks too. I wonder whether your developers have their priorities straight? Whose doing the work that makes your company actual money?
Ok, if they must build a framework, let it evolve organically from the process of building the actual software your business uses to make money.
"Needing some highly specialised report, or the ability to tweak some feature in some allegedly unique way"
As others have said, grab one of the many fine open source trackers and tweak it.
"Believing that it isn't difficult to build a bug tracking system"
Well, I wrote the first version of my BugTracker.NET in just a couple weeks, starting with no prior C# knowledge. But now, 6 years and a couple thousand hours later, there's still a big list of undone feature requests, so it all depends on what you want a bug tracking system to do. How much email integration, source control integration, permissions, workflow, time tracking, schedule estimation, etc. A bug tracker can be a major, major application.
"What arguments might you use to support buying an existing bug tracking system?"
Don't need to buy. Too many good open source ones: Trac, Mantis, my own BugTracker.NET, to name a few.
"In particular, what features sound easy but turn out hard to implement, or are difficult and important but often overlooked?"
If you are creating it just for yourselves, than you can take a lot of shortcuts, because you can hard-wire things. If you are building it for lots of different users, in lots of different scenarios, that its the support for configurability that is hard. Configurable workflow, custom fields, permissions.
I think two features that a good bug tracker must have, that both FogBugz and BugTracker.NET have, are 1) integration of both incoming and outgoing email, so that the entire conversation about a bug lives with the bug and not in a separate email thread, and 2) a utility for turning a screenshot into a bug post with a just a couple clicks.