There is no DLL that you can use to access an NSF without Notes installed. I guess if you're only concern is that Domino Server is not installed, well, then just install the Notes client. But that's probably not what you mean. If you want to access an NSF programmatically, you'll need to have the client or server installed.
To answer the question of whether an NSF is password protected, you'll need a Notes ID that has access rights on the server or local client. You can get a handle to the database using NotesDatabase.Open, then use the db.QueryAccess("-Default-") method to determine what the access level is for the default user. That will tell you what the default access level is for all users no matter who they are.