I am trying to secure online transaction between application server and database. One-way is SSL, but is there any other way of securing transaction between them. I am trying to reduce the overhead in securing the line. Are there any tool or api's which I can integrate with the application server or deploy on it.
/SR
Sure, but it requires a thoughtful review of your architecture. If you have a dedicated line between your application and your database, the connection could be encrypted at the network layer.
I'm a bit surprised that your application server and your database are not within the same trust boundary. Is this an app in the cloud?
SSL is by far the best way, its extremely secure and absolutely free. If your database or another application doesn't support SSL then you can use a free SSL VPN such as OpenVPN. This will create a secure transport layer connection between 2 machines.
If you are connecting to a database on the local machine then there is no point in a secure transport layer connection. However you should firewall off the database port, for instance MySQL uses tcp 3306.