I am looking to get a random record from a huge (100 million record) mongodb. What is the fastest and most efficient way to do so? The data is already there and there are no field in which I can generate a random number and obtain a random row. Any suggestions?
+5
A:
Do a count of all records, generate a random number between 0 and the count, and then do:
db.yourCollection.find().limit(-1).skip(yourRandomNumber).next().
ceejayoz
2010-05-13 02:48:12
Unfortunately skip() is rather inefficient since it has to scan that many documents. Also, there is a race condition if rows are removed between getting the count and running the query.
mstearn
2010-05-17 18:49:17
A:
it is tough if there is no data there to key off of. what are the _id field? are they mongodb object id's? If so, you could get the highest and lowest values:
lowest = db.coll.find().sort({_id:1}).limit(1).next()._id;
highest = db.coll.find().sort({_id:-1}).limit(1).next()._id;
then if you assume the id's are uniformly distributed (but they aren't, but at least it's a start):
unsigned long long L = first_8_bytes_of(lowest)
unsigned long long H = first_8_bytes_of(highest)
V = (H - L) * random_from_0_to_1();
N = L + V;
oid = N concat random_4_bytes();
randomobj = db.coll.find({_id:{$gte:oid}}).limit(1);
dm
2010-05-13 13:48:41
A:
I'd suggest adding a random int field to each object. Then you can just do a findOne({random_field: {$gte: rand()}}) to pick a random document. Just make sure you ensureIndex({random_field:1})
mstearn
2010-05-17 18:47:23
A:
The $gte: rand funtion does not work with that! $gte will return the FIRST value that is greater than the random number, not the number that is closest to the random number!
Flov
2010-10-05 13:59:15