I am using a sharded db - horizontally partitioned. I am using the DataFabric gem from FiveRuns. What would be the implications of using memcached on this? Would it work the same as if I was using ActiveRecord? ...
Does anyone have any information on the architecture of twitter? A few specific items I'm especially interested in: I know that they use message queues. But what exactly do they use queues for? Do they "duplicate" tweets? If so, how? For example, say a user has 10,000 followers and he makes a tweet "hello world". Does twitter store "...
There is a web application which is in production mode for 3 years or so by now. Historically, because of different reasons there was made a decision to use database-per customer installation. Now we came across the fact that now deployments are very slow. Should we ever consider moving all the databases back to single one to reduce en...
Hi everybody, I have thought a bit about sharding tables, since partitioning cannot be done with foreign keys in a mySQL table. Maybe there's an option to switch to a different relational database that features both, but I don't see that as an option right now. So, the sharding idea seems like a pretty decent thing. But, what's a good ...
Hi everybody, After getting some help on how to measure fysical/actual size of memcached objects to prevent them from growing too large, I have thought about the next step - implementing a sharding/splitting function that transparently splits large objects into smaller pieces upon storage and glues them together as one big object when ...
Does grails support sharding? I want a multi-tenancy database solution and I looking at grails currently. ...
I read today about sharded counters in Google App Engine. The article says that you should expect to max out at about 5/updates per second per entity in the data store. But it seems to me that this solution doesn't 'scale' unless you have some way of knowing how many updates you are doing per second. For example, you can allocate 10 s...
I am considering the option of neo4j for some of the new projects I am working for. For the given data needs (inherently graph based) neo4j fits well and a quick prototype is giving good response time for me. What I want to understand is how to scale a neo4j deployment. Specifically: How do I shard my data across neo4j deployments. Sin...
Hi everybody, I have thought a bit about making a somewhat lightweight consistent-hashing-like PHP function to shard uploaded files between different servers. Obviously, rand() would work to distribute the files among the servers somewhat evenly, but when requesting the files, no one will know which file lies on what server... I know ...
I've read some article say that RDBMS such as MySQL is not good at scalable,but NoSQL such as MongoDB can shard well. I want to know which feature that RDBMS provided make itself can not shard well. ...
Hi everybody, I did just come up with the following idea, but I lack the knowledge to say if it is applicable to a production application. We have a web application, built on PHP/mySQL to make it easy. A table in the database are prone to grow large - a couple million of records easily, so table sharding might be an option here. Here...
I have an application where the main entity is a Story and users can vote for each story. Each vote increments a vote_count for the story. I am concerned about write contention on the story so I plan to use a sharded counter for each story to track the votes. Now my question: how could I get a list of stories ordered by number of vo...
Say I have a MySQL table: CREATE TABLE tweets ( tweet_id INT NOT NULL AUTO_INCREMENT, author_id INT NOT NULL, text CHAR(140) NOT NULL, PRIMARY KEY (tweet_id) ) PARTITION BY HASH(tweet_id) PARTITIONS 12; All is good. The table lives on a single server - Server1. But eventually I may want to scale out. So I'd want to shard the table and...
I am having one heck of a time with my implementation. We have a solr server running. We have 3 cores, and for simplicty let's call them "Members", "Businesses", and "Products" I was attempting to use solrnet, but it does not support sharding. So, I am dreading that I will have to build the url myself. So, I used the admin tool to bu...
I am using the data-fabric gem, sharding between 2 machines. However, I would like to use the standard activerecord functions for some tables. Anyone has a similar experience achieving this? ...
I run a lyrics web site and I have a single table that basically keeps an index of songs; essentially primary key with auto increment (song_id), artist, and song title. The lyrics themselves are stored across 10 shards, with each shard based on song_id MOD 10. There are two columns: song_id and song_lyrics. The song_id field is primary ...
Hi every body! i want to split a product table into many smaller one, and put the table in diffrent server. but there has some problems with me: if i sharding the database with the product id. how can i get all product belong to some categorys when someone list one product category. anybody get a good ideals ? ...
Hi. I'm playing around with data replication in MongoDb, and I currently have a master and several slave nodes. All of these are running on different ports, but they could just as well be on different machines (I'm testing atm). My question is, how would I go about connecting to these from my code? Currently, I just have one instance o...
Consider the following setup: There a 2 physical servers which are set up as a regular mongodb replication set (including an arbiter process, so automatic failover will work correctly). now, as far as i understand, most actual work will be done on the primary server, while the slave will mostly just do work to keep its dataset in sync....
I'm having problems with what seems to be a simple sharding setup in mongo. I have two shards, a single mongos instance, and a single config server set up like this: Machine A - 10.0.44.16 - config server, mongos Machine B - 10.0.44.10 - shard 1 Machine C - 10.0.44.11 - shard 2 I have a collection called 'Seeds' that has a shard key '...