Here's the problem I am trying to solve: I have recently completed a data layer re-design that allows me to load-balance my database across multiple shards. In order to keep shards balanced, I need to be able to migrate data from one shard to another, which involves copying from shard A to shard B, and then deleting the records from sha...
I've a model MyModel (extending Google's db.Model), and I want to keep track of the number of Models that have been created. I think the code at from Google's I/O talk on Sharding Counters is quite good, so I'm using that. But I'm not sure where I ought to be calling the increment when creating a new code. (I'm using Django, and I've ke...
Hello, I was wondering what the best technique for implementing a DB connection pool for a web application which uses shards. From what I can tell most (all?) open-source implementations only support a single database behind. At least, I have not found one that supports shards. Also, even though I using shards not all of the database w...
What is the concept of Sharding from Database Design perspecitve ? ...
Hi -- Looking to shard a database and to assign different users to different home servers based on their user id. User IDs are 10 character strings, e.g., "f4gKUKkj91" ... each server has an ID of 1 - 1000. How can I create a hash function in php to uniquely and consistently assign each user id to a specific shard ? If the user id were...
For an online marketplace product under construction, I have a situation which requires implementing a database sharding solution. I am new to sharding and after reading the posts in this forum I feel a directory based sharding strategy using business entities will be suitable. But I am still not clear about the denormalization and data ...
I am using MySQL to store my parent-child relation. The data is concentrated in one table and it is really elegant by design. I do not have any problem quering; however the table has grown in size dramatically. I would like to shard or use some techniques to improve the performance of my queries (and joins). How would I do it? Any poin...
I'm new to both memcached & sharding. I gone though some articles on both. I need to implement both in my application. Articles I'd gone through were good but none of them gave me how to use these technologies. Anyone who can suggest me the links from which I can actually configure them and run the samples. I want to implement memca...
I'm trying to run a sample program for hibernate shards. I'm all done with it but whenever I run the test program I get an Exception javax.naming.NoInitialContextException: Need to specify class name in environment or system property After googling I got to know that I've to set the JNDI properties. I did like this Hashtable env = ...
I have a problem where I need to assemble a Map whose eventual size is in the GBs (going on past 64GB) and I cannot assume that a user of the program will have this kind of monster machine hanging around. A nice solution would be to distribute this map across a number of machines to make a far more modest memory footprint per instance. ...
Does anyone have any good sources of information of using NHibernate with Sql Azure with the implications of sharding (because of the 10gb cap)? I know there are posts on the internet that reference a sharding project for NH but they are from 3rd quarter 09 and I haven't found any much more relevant on google. Related does anyone have i...
I have a database that is currently shard-ed. This is the first time that I have come across a shard-ed database and would like to know a bit more about them. Things like: When is it a good idea to shard a database? At what database size is it a good idea to shard a database? How easy it is to remove sharding? ...
When performing many inserts into a database I would usually have code like this: using (var connection = new SqlConnection(connStr)) { connection.Open(); foreach (var item in items) { var cmd = new SqlCommand("INSERT ...") cmd.ExecuteNonQuery(); } } I now want to shard the database and therefore need to choose the c...
We're looking to implement load balancing by horizontally sharding our tables across a cluster of servers. What are some options to implement live redundancy should a server fail? Would it be effective to do (2) INSERTS instead of one ... one to the target shard, and another to a secondary shard which could be accessed should the pri...
the first one is running at 73% memory utilization. The second one runs at 16% utilization. how can i balance the 2 out? ...
Say I have a "message" table with 2 secondary indexes: "recipient_id" "sender_id" I want to shard the "message" table by "recipient_id". That way to retrieve all messages sent to a certain recipient I only need to query one shard. But at the same time, I want to be able to make a query that ask for all messages sent by a certain sen...
To improve page render speed I am looking at implementing image sharding which serves images from the multiple subdomains. How can I ensure that the same image is retrieved from the same server? I want to avoid a scenario like the one below where the same image is served from multiple domains: images1.domain.com/image1.gif images2.domai...
I have two tables on different servers, and I'd like some help finding an efficient way to combine and match the datasets. Here's an example: From server 1, which holds our stories, I perform a query like: query = """SELECT author_id, title, text FROM stories ORDER BY timestamp_created DESC LIMIT 10 ...
Has anyone experience with custom Lucene sharding / paritioning using Hibernate Search? The documentation of Hibernate Search says the following about Lucene Sharding : In some cases, it is necessary to split (shard) the indexing data of a given entity type into several Lucene indexes. This solution is not recommended unless...
This sounds kinda dumb, but I have a sharded DB that I no longer think I need to run on 2 machines, and would like to run on one single machine instead. Any ideas on how that can potentially be done? There are lots of resources on how i can achieve the converse, but very little on how this can be done ...