If I reference a function named sqrt, how does the compiler know which file to look in, if I haven't specified it? It could be absolutely any file on my entire harddrive.
Unlike Java, C++ doesn't really consider any files "special". Java has its giant (bloated) class library, which is automatically made accessible to the programmer.
In C++, this concept doesn't exist. You tell the compiler which paths to search in, and whenever you #include a file, it will search for the filename in those paths.
If that happens to find a standard library file, it'll use that. If it happens to find a third-party file, it'll use that.
The compiler doesn't know that sqrt is defined in the header math.h. Or that it is also typically defined in cmath In fact, the functions defined by a header might vary. Perhaps, if I #define the appropriate preprocessor symbol, some functions will be removed from a specific header, and others will be enabled.
But unlike Java, where the functions and classes defined by a library can be determined just by examining the library file's metadata, in C++, the header has to be compiled. And the result of compiling it might vary depending on the context in which it is included.
So the C++ compiler cannot guess at which header should be included in order to define the function you just used.