The key to the answer is in the protocol, just as you are hinting at. It may not always be pretty, and sometimes you are locking yourself into a design.
However, the PC architecture itself is proof of feasibility of forward detection.
It is often entirely possible to boot a Linux kernel compiled in say, 1996 on a system purchased today. The Linux system may not be able to use much of the hardware in that system, but it will find whatever subset is still the same.
It requires however, either a very wise hardware detection scheme or a willingness to live with kludges (see the line A20 setup on PCs, originally using the keyboard controller) or both at the same time. (Like how almost all PCs today both has the old ISA bus internally for things like PS/2 keyboard support, but also have the PCI protocol.)
For work, I designed an ID ROM format for daughter boards. They had a compact, binary XML or Lisp-like structure with a few mandatory fields, several optional, and a version number. Any completely new hardware can increment the version number and add any data necessary for new software, while still telling old software whatever appropriate in the old fields.
So that at the very least, the old software can detect the board ID, purpose, and then give up. This is much how USB devices work. All old software can enumerate new devices, but it is far from certain that the old software can drive the new hardware in any meaningful way. Exceptions are keyboards, mice and memory sticks, where everybody sticks to the old "application level, so to speak" protocols.