matrix operations on large matrices on limited memory

Question

I need to do some matrix operations on my computer. These matrices are large 1000000x1000000 and more, some operations requiring TB of memory. Obviously these cannot be directly loaded into memory and computed. What approaches can I use to solve these matrices on my computer? Assuming that the matrices cannot be reduced further using matrix optimizations and are already stored in compact form. I am thinking about using some memory mapped scheme but need some ideas.

Answer 1

Two suggestions:

1. Use the mmap2 system call to map the files containing both the input and output data. This allows you to map files up to 2^44 bytes and treat them as if they were already in memory. I.e. you just use a standard pointer syntax to access the data and the OS takes care of either reading or writing it from / to disk without you needing to worry about it. Not only that, but mmap is many times significantly faster than manual file I/O - See this SO post.

2. Read "What every programmer should know about memory" by Ulrich Drepper. One of the example problems he deals with is highly optimizing matrix operations.