Is there a windows api to detect whether or not my dll is running in a system service or a normal user process?
Previously I was looking at the current username and ignoring "SYSTEM", "LOCAL SERVICE" and "NETWORK SERVICE". But now I am seeing GetUsername return machine_name$ in cases where svchost gets run in certain situations before anyone is logged in. Also, I cannot find any documentation for the machine_name$ result from GetUsername in vista, has anyone seen this behavior?
It is whatever user account was used to configure the service. Could be anything, it is up to the system admin to pick these accounts. Which of course means that you can not reliably use it to detect whether your code runs in a service.
You should let the service that uses your DLL tell you that it is a service, it always knows. Auto-detecting is tricky, GetProcessWindowStation() ought to be a lead. Call GetUserObjectInformation on the returned handle with the UOI_FLAGS flag and check if you get USEROBJECTFLAGS.dwFlags = WSF_VISIBLE. There could be some degenerate cases with RDP I suppose but you can see if a user has odds of seeing process output.
You can look in the process token to see if the well known SID NT Authority\Service is active in the token. If it is, then you're running in a service. You can use CheckTokenMembership to see if your process is running with a particular SID active in the token.
As the other answers have already stated, examining the current username is very unreliable. Services can potentially be set the use any account, even including the account of a real user.
I have researched a question similar to this in the past: Can an executable determine whether it's being run as a service or as a normal program? The conclusion there seemed to be that there is no official way to tell. The only reasonably reliable way I found involved examining the success or failure of one of the functions that a service program must call when it starts up. Obviously, that technique is not usable for a dll being used by the service.
So to my knowledge, there is no good way for a dll to determine this on its own. Either need the hosting program to tell the dll. Or else you need to re-examine your design. Why does the dll actually need to know this? And are there any alternative ways to make it work such that it doesn't need to know?
You can try call StartServiceCtrlDispatcher function (see http://msdn.microsoft.com/en-us/library/ms686324.aspx) and examine the error code. Inside a service it will be ERROR_SERVICE_ALREADY_RUNNING.