I'd rather get vaguey sane behaviour that something crazy.
Well, as a potential consumer of your library, here's what I'd like: if there's no performance cost to the documented usage, then see if one of the memory_order values provides a functional superset of the others, particularly something corresponding to what a caller might naively expect the unsupported modes to do (if any sensible expectation can be formed). The caller may get the slowest, safest mode, but that's better than something functionally wrong. You minimise the client code's dependency on getting everything perfect for your code. The problem with this - compared to an assert/exception - is that it can go unnoticed in a test environment, so consider also writing an explanation to std::cerr, using a static variable to limit the messages to one per process run. That's a very useful diagnostic.
An exception, fatal assertion etc. might bring down a client application at a very inconvenient moment.... Seems a bit draconian, and not something I'd appreciate particularly. Another option is to have an environment variable control this behaviour.
(There's presumably a similar issue for values that aren't even in your current enumeration.)