There are many applications for Windows these days that don't use native windows controls, don't have standard window frames and generally look different. What are some recommended techniques for creating such interfaces?
From a design aspect, if your interface isn't going to follow documented standards (like the Windows UI guidelines), it has to be intuitive. I think the new generation of Windows applications will go through a growing phase in a manner similar to the early days of the Web. After a time, some standards or common themes will evolve.
There are good reasons not to. Like that you will most likely not do a better job than Windows does. (Maybe it will look better (in your opinion), but will it behave?). Or that it's not what most users expect. Or that it will look like s**** on Windows 2011.
That said, it's not hard. You simply handle the WM_NC* events like WM_NCPAINT or WM_NCHITTEST. NC stands for Non Client (window area). And of course, there is a trick on Vista/Win7 (you have to announce it to the DWM).
From an implementation aspect, you could employ WPF (Windows Presentation Foundation) assuming you code for .NET :) It has pretty bunch of skinnable controls, that may look like native and may not.
Can you give us some sample applications? Some apps that don't use native windows controls use cross-platform GUI libraries, like Qt for C++ or Tkinker. These maintain the same look across different platforms.
I wouldn't really recommend making your user interface different deliberately. You don't stand to gain much. Your controls are almost always going to be buggier than native controls, and you are requiring the user to learn something new. Now, if you're controls add a large enough value to be worth the users' time it can be okay. But making them get used to different looking buttons is rarely worth it.
I`m not sure if this answer your question.
You can use third party skinning controls like from Infragistics, or SkinSoft for example. But like Bubba said I`d recommend going for WPF.
Model-View-Controller! It's as valuable here as in web apps or anywhere else. Be sure to keep the part of your program that generates the custom UI separate from the part of your program that flashes the BIOS.