I believe Ext.Js does something similar. They have a concept of "adapters" which allow you to sit Ext.JS on top of any of the underlying libraries and it will just work. The key difference is that they are using a 1-to-1 model where you say which library you want to use and it joins the dots, whereas I believe you're trying to say "I don't know which one will be available but whatever one you find, go use it".
I don't think it's insane, but you might have some fun trying to work out which library to use, particularly if both are available.