I was thinking of this the other day, apps like Twitter deal with millions of users. I was thinking how the functionality of 'following' would work, where the maximum amount of users in the database can follow the maximum amount of users in the database less one, (himself).
If this was a ManyToMany bidirectional mapping, it would create a relationship table with potentially billions of records. and how could an ORM retrieve such records? for example, if user A is following 20,000 users, would the ORM load 20,000 records for that one user, or would a pagination approach be used? I can deal with JPA / ORM with small recordsets (< 10 records, say), but beyond that blows my mind on how to write software to support large recordsets. Sorry this question is not specific, but I'm trying to get architectual ideas on this. In my spare time, I will run some tests against billions of records, but wanted to get some input from the community first.