I wouldn't dismiss the complexity of the OpenFileDialog. It's not so easy to build one that really works. When you do build your own, it's not the "normal" dialog and as a result it confuses users. This is true even if you do it well, which is difficult. So I'd suggest you stick to extending what is already there, rather than writing something new.
Check this article for an extension of OFD that might/could be tweaked to do exactly what you want. There's a callback that you write in C# that responds to path selection.
Related: FolderBrowserDialogEx is a similar extension on FolderBrowserDialog. Despite the name, you can configure it to search for files, as well as folders. There's a callback that gets invoked when something (a folder, a file) is selected, and within that callback you can do what you need to do. For example, peek inside the files within a folder and populate the list of files to display with only those files.
Another option you might consider is the dialog library from Ookii. This is an open source implementation of the OpenFileDialog, and it includes COM wrappers for all the new dialog stuff in Vista. Using that library you can pop a Vista OpenFileDialog and receive events from the IFileDialogEvents interface, in C# code. One such event is OnFolderChange(). Within the handler you could call IFolder.GetFolder() which will get you an IShellItem, which gives you the folder the user is changing to. The next step would be to itemize and potentially filter the set of files, which is an exercise I will leave to the reader...