It's possible to get a handle to a database without opening it. If you try and call most methods without opening it first then you will get that error. The most likely explanation though is that you don't have access to open it.
What id signed the agent? Probably not the server. The agent will run with the authority of the agent signer, so that is the id that needs to be in the ACL of the database you are trying to open.
There are a whole bunch of rules about how agents can run under different authorities and on behalf of different users. That can get pretty complex if the situation requires it. But check the agent signer has rights to open the database first then look at any "run on behalf of" settings.