Java Database Connection Pool for multiple databases (Shards)

Question

Hello,

I was wondering what the best technique for implementing a DB connection pool for a web application which uses shards. From what I can tell most (all?) open-source implementations only support a single database behind. At least, I have not found one that supports shards.

Also, even though I using shards not all of the database will have the same schema as I will have other databases too. I'm not sure if that's important to mention.

The only solution that I can come up with so far is to write a layer that sits on top of multiple and distinct pools. Each distinct pool can be any of the available single database implementations.

Are there already solutions for this? What would be the best technique otherwise?

Thanks in advance,

Stephen.

Answer 1

I don't think there is an open-source implementations that support sharding. Maybe, there isn't a real need since creating a layer on top of multiple database pools is not too hard. It only takes a shard mapping function(e.g. hash function) and a manager class to track multiple pools.

If you are worried that not all db have the same schemas, you can put additional schema tracking config into your manager class, so it knows which shards can serve the schema. That's, you need to track schema to shard info in additional to db pool. This is not really much additional work since you need the shard config anyway to determine how to pull the right shard from the pool (e.g. User id mod 10 = 1 should pull from Shard 1)

Good luck

Answer 2

What is the technology stack that you are using currently? I know that Hibernate has a sharding project, but I have not used it, just listened to some podcasts about it.

More information about it can be found here. Also, the previously mentioned podcast could be found here.

The podcast explains what a few of the issues with sharding in general, some of the hurtles the Hibernate plugin has taken care of, and then explains their anticipated path forward. Hope that helps a little bit!

Answer 3

There is the hibernate shards project you could take a look at.