My users like to upload password-protected Word documents into our custom document management system. I'd like to add a validation to check for the password and refuse the upload if it has a password.
Automating Word - with COM interop - is out of the question because this is a server side application.
Are you saying that you'd like to open/examine Word documents without using any Office interop at all? Good luck!
Possibly your only place to start would be to manually parse the docx yourself (the spec is published here). Really though, I'd suggest you bite the bullet and use the Office interop libraries, 'cause that's a much lighter weight requirement than parsing the document yourself. There's nothing that intrinsically prevents you doing that on a server application.
I think it is going to depend on the version of Microsoft Word. Older versions of Word (before 2007) will require some COM interop, because they are not saved in an open format. Nothing you can do to get around that, and I feel your frustrations with installing anything Office on a web server. Additionally, I believe the only way to detect password protection on these is to attempt to open/unprotect the file and catch a certain exception (you may have to further evaluate an error code within the exception as well). Not pretty!
However, for newer versions of word (2007+) saved in the open DOCX format (Standard ECMA-376), you can examine the XML and check for the existence of the DocumentProtection element with the w:enforcement attribute set to "on"...
<w:DocumentProtection
w:edit="read-only"
w:enforcement="on"
w:unprotectPassword="1FC6CBEB"/>
Note: The password seen here is encrypted (obviously); I saved this particular document with a password of "test".
Unfortunately it's not available directly from .NET or DSOFile.dll, but you could create a wrapper in .NET to read the PIDSI_DOC_SECURITY property of any Office file to find out if it is password-protected or not without opening the document. There are a number of C++ samples out there that could be ported with a definition of iPropertyStorage.
A wrapped example is on TechTarget, but the wrapper seems to be unavailable.