Handling very large lists of objects without paging?

Question

Hi,

I have a class which can contain many small elements in a list. Looks like:

public class Farm {

    private ArrayList<Horse> mHorses;
}

just wondering what will happen if the mHorses array grew to something crazy like 15,000 elements. I'm assuming that trying to write and read this from the datastore would be crazy, because I'd get killed on the serialization process.

It's important that I can get the entire array in one shot without paging, and each Horse element may only have two string properties in it, so they are pretty lightweight:

public class Horse {
    private String mId;
    private String mName;
}

I don't need these horses indexed at all. Does it sound reasonable to just store the mHorse array as a raw Text field, and force my clients to do the deserialization? Something like:

public class Farm {
    private Text mHorsesSerialized;
}

then whenever the client receives a Farm instance, it has to take the raw string of horses, and split it in order to reinstantiate the list, something like:

// GWT client perhaps
Farm farm = rpcCall.getMyFarm();
String horsesSerialized = farm.getHorses();
String[] horseBlocks = horsesSerialized.split(",");
for (int i = 0; i < horseBlocks.length; i++) {
    // .. continue deserializing the individual objects ...
}

yeah...

so hopefully it would be quick to read a Farm instance from the datastore, and the serialization penalty is paid by the client,

Thanks

Answer 1

In general, it's not a good idea to use lists unless your lists are short (not the case here!) or need to be indexed (also not the case here). You also need to bear in mind that the maximum size for a serialized entity is 1MB - so whatever serialization mechanism you use needs to fit 15,000 list entries into 1MB.

If they do fit, then yes, using your own serialization into a Blob field (not a text field, unless you're using a text format like JSON) is your best option.

Answer 2

Another consideration you might want to take into account is that you will be making one big request from GWT, and presumably using the response of that request to build some sort of UI of 15,000 horses.

That will not be quick, and while it's chugging away on building your list of horses, the rest of the UI will be pretty much unusable.

And what's more, the UI will be nearly impossible to navigate once it is displayed. What is this UI for? Listing horses, so that one can be selected? Would you want to have to sift through 15,000 entries to find one that they like?

Pagination is not only a way to ease the stress on the server, it's a way to ease the stress on the browser, and on the user.

You may want to look into some way to allow users to search or filter through your list, otherwise it will be nearly impossible for them to use. And to do that, you'll probably want to restructure your data model.