If you don't need any caption buttons at all, you can achieve this by setting some window flag hints:
setWindowFlags(Qt::Dialog | Qt::CustomizeWindowHint | Qt::WindowTitleHint);
Qt's Demo application has a sample application that lets you experiment with these flags (Qt Demo->Widgets->Window Flags) if you want to see what different combinations do.
On the other hand, if you want any of the Minimize/Maximize/Close buttons, you will notice Qt forces the system menu and window icon to show up. I think this is Qt generalizing the platforms a bit, as it's very easy to find examples of native Windows dialogs with a Close button but without the system menu and window icon.
In that case, you will need some Windows specific code, similar to this (untested):
#if defined(Q_WS_WIN)
// don't forget to #include <windows.h>
HWND hwnd = winId();
LONG_PTR style = GetWindowLongPtr(hwnd, GWL_STYLE);
style &= ~WS_SYSMENU; // unset the system menu flag
SetWindowLongPtr(hwnd, GWL_STYLE, style);
// force Windows to refresh some cached window styles
SetWindowPos(hwnd, 0, 0, 0, 0, 0,
SWP_NOMOVE | SWP_NOSIZE | SWP_NOZORDER | SWP_FRAMECHANGED);
#endif
Edit: As commented by swongu, this only works if you want to have a close button without a system menu. If you want a minimize/maximize button but no system menu, you're out of luck.