My thoughts over this for a while now lead me to think the Controller part of the pattern is fairly redundant in this design. And again, It's MVVM not MVVMC or MVC :)
Abstractions need to be justified in the end, or it'll just be the redundancy of the service pattern all over again. Something that shouldn't be necessary with observers.
So my design as of right now looks like this. The hardware controllers are all event based and meant to do UI updates, so I think it really do make sense to access them directly from the viewmodel.
We'll probably still design it with the use of DI/IoC, so loose coupling could allow for a mediator at a later point, should it be necessary.
But I'll leave the question open for better answers for now :)