If I compile branch A, then switch to branch B, compile, and switch back to branch A.
All the object files touched by the compile of branch B have to be recompiled!
Normally one does not check-in object files, but there seems little choice here.
What are the ideal working practices here?
Only the object files whose dependencies differ between the two branches need to be rebuilt.
When switching branches git doesn't touch files that don't differ between the two branches.
So long as you have appropriate dependencies expressed or deduced in your build system, an incremental build after a branch switch will work and be reasonably efficient. In many of my projects I switch branches often and incremental builds are fast and reliable.
If you want to work on two branches without wanting to rebuild at all when switching branches you should use a second clone and work in independent working areas.
Checking in object files into the same repository as the source that creates them is almost always a bad idea. For a start, it allows the possibility of checking in input and output that are inconsistent.