Scalable stl set like container for C++

Question

Hi, I need to store large number of integers. There can be duplicates in the input stream of integers, I just need to store distinct amongst them. I was using stl set initially but It went OutOfMem when input number of integers went too high. I am looking for some C++ container library which would allow me to store numbers with the said requirement possibly backed by file i.e container should not try to keep all numbers in-mem. I don't need to store this data persistently, I just need to find unique values amongst it.

Answer 1

Since you need larger than RAM allows you might look at memcached

Answer 2

Take a look at the STXXL; might be what you're looking for.

Edit: I haven't used it myself, but from the docs - you could use stream::runs_creator to create sorted runs of your data (however much fits in memory), then stream::runs_merger to merge the sorted streams, and finally use stream::unique to filter uniques.

Answer 3

Have you considered using DB (maybe SQLite)? Or it would be too slow?

Answer 4

You should seriously at least try a database before concluding it is too slow. All you need is one of the lightweight key-value store ones. In the past I have used Berkeley DB, but here is a list of other ones.