I am using layered windows and drawing a rounded rectangle on the screen. However, I'd like to smooth out the jagged edges. I think that I'll need alpha blending for this. Is there a way I can do this with GDI?
You can do this in C# using the LockBits method of the BitMap class (see this question for an explanation). You definitely don't want to use GetPixel and SetPixel for this, as they are hideously slow (even though you'd just be manipulating the edges and not the entire bitmap).
There isn't an easy way to do such drawing with just GDI calls. What you want isn't just alpha blending: you want anti-aliasing. That usually involves drawing what you want at a larger resolution and then scaling down.
What I've done in the past for similar problems is to use an art program to draw whatever shape I want (e.g. a rounded corner) much larger than I needed it in black and white. When I wanted to draw it I would scale the black and white bitmap to whatever size I wanted (using a variant of a scaling class from Code Project). This gives me a grayscale image that I can use as an alpha channel, which I'd then use for alpha blending, either by calling the Win32 function AlphaBlend, or by using a DIBSection and manually changing the appropriate pixels.
Another variation of this approach would be to allocate a DIBSection about four times larger than you wanted the final result, draw into that, and then use the above scaling class to scale it down: the scaling from a larger image will give the appropriate smoothing effect.
If all this sounds like quite a lot of work: well, it is.
EDIT: To answer the title of this question: you can create a bitmap with an alpha channel by calling CreateDIBSection. That won't on its own do what you want though, I think.
Any chance of using GDI+ instead of GDI? It supports antialiasing and transparency right out of the box.
CreateDIBSection. Fill in the BITMAPINFOHEADER with 32bpp. Fill in the alpha channel with pre-multiplied alpha and youre good to go.
AlphaBlend is the API to actually blit 32 bpp bitmaps with an aplha channel.