So it turns out the backticks method specifically maps to Kernel.`, which is fine until one looks at the FlexMock source to see what they consider to be valid method names. The regex they use essentially is checking for alphanumeric with ? or ! at the end, so backtick fails to match this regex.
Changing the code internally resolves the initial exception that is thrown, but doesn't actually make the code work as intended. I could have monkeypatched it as pierr suggested, but that would be extremely repetitive in my testing code, so I went the alternate route and just made a shell method in my library code that only does the backticks. This method then can be mocked out in the desired fashion, and since I only use the backticks in a small number of places, not a lot of my underlying code had to be changed.