If you are interested in writing application using C#, do a little search for Compact Framework (CF). It is the stripped down version available in Windows CE.
Keep in mind that Windows CE is a highly customizable OS and you have no guarantee that it will have the component to support your application (that includes C#) - Windows Mobile (+PocketPC, +SmartPhone2003) is a different story.
When you develop for Windows CE you need to use Visual Studio 2005/8 Professional to have the Smart Device support. When you develop C# applications using VS2008 you target CF3.5 and when you use 2005 you target CF 2.0