How to achieve low memory consumption?

Question

I want to know which technique antivirus programs use for scanning disk or files and maintaining low memory consumption. They don't affect the user activity either.

I am looking for an approach by which we can achieve disk scanning with low memory consumption.

Answer 1

They don't. Every scanner I know uses a lot of memory, and has impact on the performance.

Answer 2

I think you are overestimating the leanness of these scanning tools. I've seen them routinely take huge chunks of memory and occasionally spike the cpu for a while. They also hijack your startup to make sure they start up first, which holds up your startup.

Answer 3

NOD32 has a pretty small footprint, but still 10-20MB in memory.

Keep in mind what AV has to do for the most part- look at the executable part of each file for malicious bytes. A traditional virus is typically less than 1000 bytes, the identifiable patterns maybe only be 50 bytes. So for AV to protect you against 100K virus patterns, it only needs a pattern database of 50*100K=5MB.

Answer 4

You should explore memory mapped files. They allow one to process huge files without loading the entire file into memory at one time.

Answer 5

I agree to most people that antivirus software has never had low memory or CPU consumption. However, here are a few ideas off of the top of my head:

• Scan only the files the user opens, only when he opens them.
• Only scan risky files - like executables or scripts, not all files.
• The scanning is usually done by hashing the file and maching the hash against known virus hashes. To minimize memory usage you could just keep the known hashes on disk and search them when needed, but that would be very slow. The fastest way would be to keep them all in RAM and forbid the OS to swap them out, but that would use a lot of memory. A tradeoff can be achieved by several level of hash caches, like this:
  • 1st level cache contains 24-bit hashes as a bitmask. This occupies about 16MB of RAM and can be kept completely in RAM (forbidding the OS to swap it out). Checking this can be done very quickly.
  • 2nd level cache contains full 128-bit or larger hashes and is kept on disk. Only if first level cache gets hit, is the second level cache tested. Because the hash space of 1st level cache is small, it is likely to get a lot of false positives, so the second level cache has to be checked.
• Cache the results of the last, say, 1000 files scanned. This way you don't have to do all the hashing and checking over and over again for files that are often used.