Even if you could query BorlandIDEServices for IOTAToolsFilter, that interface isn't going to help you do what you're asking. That interface was introduced as part of a mechanism for adding additional build tools (compilers, etc.) to the IDE (before the IDE used MSBuild). It allowed you to write a custom "filter" to handle output from a particular build tool, but it would not let you apply a filter to one of the built-in tools (like the delphi compiler).
The reason the Supports(BorlandIDEServices, IOTAToolsFilter, OTAToolsFilter)
call fails in Delphi2010 is that once MSBuild support was added to the IDE, the old way of adding build tools to the IDE was disabled, and the BorlandIDEServices interface no longer supported IOTAToolsFilter.
The declaration of IOTAToolsFilter should probably have been marked deprecated in ToolsAPI.pas (or least it should have been mentioned in the source code comment that it is no longer supported).
As far as your desire to filter a particular hint, I'm not aware of a way to do that via the ToolsAPI. It seems like a reasonable thing that can be added to IOTAMessageServices (the ability to enumerate, filter, and possibly change the messages in the IDE's Message View). I would enter a request in QualityCentral for that.
Also, please vote for QC #35774 (http://qc.embarcadero.com/wc/qcmain.aspx?d=35774), as if that were implemented, you would not need to use the ToolsAPI for this sort of thing.