Might be able to post more useful stuff later on, but for now I can tell you the term you want to search Google for is: "kiosk mode".
Update - Useful stuff (hopefully)
Now to be frank any kiosk mode is more or less a hack. Windows Mobile isn't designed for it and as you get into more and more edge cases you are going to find the odd gap, however for the purpose of most programs the following is sufficient:
Task 1 - Cover the UI and the taskbar so that it can't be accessed:
On your main form set the WindowState to Maximized and FormBorderStyle to None.
On older OSes you might need to actually disable the taskbar itself and move the form over the top of it. This is achieved by PInvoking:
FindWindow with the argument "HHTaskBar" (this may depend on platform, HHTaskbar works for Pocket PC 2003) and String.Empty
[DllImport("coredll.dll", EntryPoint="FindWindowW", SetLastError=true)]
private static extern IntPtr FindWindowCE(string lpClassName, string lpWindowName);
EnableWindow with the IntPtr from FindWindow and false
[DllImport("coredll.dll", SetLastError=true)]
public static extern bool EnableWindow(IntPtr hWnd, bool bEnable);
ShowWindow with the IntPtr from FindWindow and 0 (SW_HIDE)
[DllImport("coredll.dll")]
public static extern bool ShowWindow( IntPtr hwnd, int nCmdShow);
Task 2 - Prevent hard-wired app keys.
You know the ones, press orange and left button and it will automatically open Pocket Outlook.
To do this I'm going to break rank here and recommend the only viable way I know of doing this which is to use an undocumented Win32 API call. It's a perfectly stable call and I have a range of projects running every day that use it I just figure in some future upgrade I might need to modify the code if it gets removed, so bear that in mind.
You'll want to setup a low-level system wide keyboard hook via the PInvoke call:
[DllImport("coredll.dll")]
private static extern IntPtr SetWindowsHookEx(int idHook, HookHandlerDelegate lpfn, IntPtr hMod, uint dwThreadId);
This is reasonably complex and its probably better just to point to a guide like this one to explain the theory. The basic premise is to discover the keycode of the irritating "special keys" and then block them via the hook (i.e. don't pass them on).
If you're working on CF I suggest also digging into OpenNETCF as I believe it already has a global KeyHook inside it.
As I said before this isn't perfect and IIRC the volume control is not blockable and its possible that a notification such as a new wireless network might intrude upon your kiosk mode if you don't set various flags in the registry (to tell it not to do that :) ).
Still, it's not that much effort and it should be sufficient for most of your apps.
Task 3 - Make your app run from start up
This is the bit that can differ a fair bit depending on the device. If you want to stay in managed code the issue is that the NETCF doesn't come pre-installed on some devices. In most cases you can just write an unmanaged booter that sits in the autorun directory (there should be one, check the manufacturer's docs), and installs .NETCF, your app(s) and then runs your app(s). If you don't want to get your hands dirty with unmanaged code then most hardware manufacturers offer some kind of scripting system to setup a device as you see fit. However these may work with varying degrees of effectiveness.