I have a program that's statically linking to a library (libA.2.0.a) and also dynamically links to another library (libB.so). libB.so also dynamically links to an older version of libA (libA.1.0.so).
Is this configuration possible? And if so, how does the system know to use the symbols from libA.2.0.a for my program and the symbols from libA.1.0.so for libB.so?
Yes, this configuration is possible.
In answer to your question as to how the system knows how to use the symbols, remember that all of the links happen at build time. After it's been built, it isn't a question of "symbols", just calls to various functions at various addresses.
When building libB.so, it sets up it's links to libA.1.0.so. It does not know or care what other applications that use it will do, it just knows how to map its own function calls.
When building the application itself, the application links to libB.so. Whatever libB.so calls it completely unknown to the application. The application also statically links to a library, which libB.so does not care about.
One gotcha: if libA uses static variables, there will be one set of statics accessible to libB.so, and a different, independent set of statics accessible to the application.