What language(s) for very large lists?

Question

Java (and maybe the underlying C-ish code) has max capacity of Integer.MAX_VALUE (~ 2 billion) for arrays and containers in java.util. Are there other languages that feature containers with larger capacities?

Answer 1

You can write your own containers in both languages that support long indices.

Answer 2

Do you have a machine with enough RAM to use more? O_o If you do, I'd say you need your own collection, because performance of the builtin ones will doubtfully scale...

Answer 3

There is no limit if you write your own container.

Answer 4

I normally use a database to store that large amounts of data. Solves a lot of problems with scaling.

Answer 5

If you're starting to hit the 32-bit limit of a number in relation to the number of elements you can store in a list/array/collection, then I would seriously start finding a new way to implement your algorithm.

You're going to have lots of "we need this specialized hardware in order to execute our program" type of requirements.

Answer 6

You don't want languages, you want databases.

Answer 7

STL containers in C++ use size_t indices, which are 64-bit on a 64-bit machine.

Answer 8

An object database may suit your purposes better.

For example, db4o.

Or, for arrays of fixed size objects, it might be worth experimenting with a memory mapped file, but you will need a language interface to the OS API for that.

edit: or just use on ORM to map your collection to a standard SQL database. These exist for most languages. For example ruby has activerecord and Java has hibernate.

Answer 9

No body wants to cope with such large quantities of data in memory at once. I don't know what you're trying to do, but if you need to maximize the amount of resident data you must take in account:

• First use dynamic memory allocation.
• You may try (in your OS) to maximize the user-mode virtual memory addressing space, if this is an option. e.g. 32-bit Windows can use the typical 2GB/2GB addressing spaces for usermode/kernel or 3GB/1GB.
• Always monitor your commit memory/OS commit limit so you don't force the OS to thrash, slowing down the entire system.
• You can lock down a minimal amount of memory for exclusive use of your application. This varies from OS to OS, but e.g you can give the user to fix 256MB, 512MB, 1GB memory space for your application.
• If you need to surpass the available usermode address space, there are 64-bit systems at your disposal, or 32-bit with extensions such as PAE.

Well, there are a lot of things to research, but just my 2c.