I use file mapping to read a 20 GB file. And when the main memory is exhausted, how is the kernel swapping the file mapped pages to the disk?
A possible way I guess is to set the page entry to NULL. Then next time if the page is accessed, the do_no_page() function will be called again to map the file to memory. Is it right?
Another question is what's the priority the kernel uses to swap the memory pages? Normal page or file mapped page?
There is an invalid bit for each entry in the page table. When a page is swaped to the hard drive, its invalid bit is set. Whenever the page is accessed the MMU (memory managment unit, inside the processor) raise an exception and the system is in charge of reloading the page into memory. Then the faulty instruction is re-executed.
Usually pages that are swapped are the pages that are not frequently accessed. There is another bit in each page table entry to know whenever a page has been accessed. How to choose the right page to swap with this single bit is a big problem, there is a whole chapter about this in the very good operating systems by Andrew Tanenbaum.
You can lock page into RAM memory with mlock() system call. If you use mmap() to map your file in memory it can be done with the flag MAP_LOCKED (see the manuals).