At its simplest, a shell is simply a program that reads a command line and then executes the command the user typed in using one of the exec() family of functions. If you want to support I/O redirection, you will have to do a little more work parsing the command, but it's not impossibly difficult - writing a simple shell is often set as a beginners programming exercise.
I'm not sure, but what be on your mind are the Win32 console APIs. This are what you have to call to interact in detail with the character-mode console window.
If you want something shell-like that isn't, in fact, a Console Window, then you need to learn about Win32 pipes and the CreateProcess API. You will use the pipes to capture output and to feed in input.
If you want to mix in GUI apps, then you've got a gigantic job ahead of you, if you really want to force their windows to be in your app's window tree.
Most DOS applications with a graphical user interface will not work if you simply intercept (redirect) output and input. These programs, which tyically write directly to the video hardware or use DOS interrupts, will need to run in a console window if you are using Windows. The Windows console will emulate enough of the DOS environment so that these types of programs will function properly.
As Neil Butterworth mentioned, you should be using one of the exec functions to launch programs. There's nothing special needed for DOS programs: the operating system can tell by looking at the executable if the program is a Windows program or a console program.
A complete, but complex, example of an interactive shell is available in Microsoft's F# distribution. Code for the F# Interactive (FSI) shell is in the C:\Program Files\FSharp-1.9.7.8\source\fsharp\Fsi folder.
Another example in C is available as part of the TCL distribution. The code that implements the TCL command shell is in the \win folder.