Storing private key information and passwords for ssh logins

Question

Alright I am working on a system for managing a bunch of vps. This includes mundane maintenance tasks as well as resource allocation.

In order to accomplish this my control server will need to be connecting to the various servers. I regularly use ssh public/private key and this seems like the most logical way to connect from the control server to the slave servers.

Now, lets pretend the control server is as secure as is reasonably possible. My question is, how can I store the private key to the control server, and the root password (assuming direct root log-in is disabled) to the slave servers securely.

I feel like a hardened server is the best defense since once the server is compromised the battle is probably lost. I know that no scheme will be foolproof, but what is the best practice?

Answer 1

One solution we have used previously is to acutally not let the control server execute commands on the slaves directly, but to have a daemon/script on the slaves pull a list of tasks from the control node and execute them. This way you do not need to store privileged information but only need to give the slave access to a "todo-list".

A basic implementation would just be a database which is accessed by the slave, but you could also do a client-server architecture.

If you want the server to SSH into the slave, you can elevate privileges using sudo which has additional features such as limiting the commands a particular user can run with elevated privileges. Storing the password is, as you already mention, very tricky since they will have to be available in plain at one point or another.