If it's a small, fairly simple chunk of data, you can create an XML serializable class and very easily write it to disk on shutdown and read it back on startup. For a simple enough class, you can just add the [Serializable] attribute, and the XmlSerializer will automatically know how to serialize/deserialize it.
If you have enough data that a SQL database would be a better fit, look into SQL Server Compact Edition or the System.Data.SQLite binding for SQLite.
Both will let you create a database as a single file, without having to install any extra Windows services or configure anything. System.Data.SQLite doesn't even need to be installed - it's contained entirely with the .dll that your project references.
In either case, the best location for the file is probably SpecialFolder.CommonApplicationData - I think this ends up being C:\ProgramData\ on Vista, but avoids having to hardcode the exact path.