Your code is not valid C, unless put in a specific (and quite artificial) context. Operator | is a binary operator. It is a bitwise-or, as you seem to know already. By itself, it cannot be used the way it is used in your code.
If one wanted to force this code to compile as C code, one'd probably have to define FBMap as a macro. Something like
#define FBMap(x) something_else(abs(0 x 0))
thus trying to emulate the mathematical "absolute value" operator | |. Your call will expand into
pFCT->something_else(abs(0 | ( VBQNum - 1 ) / 8 | 0))
thus making the application of | operator valid.
But even after that you'd need something_else to be a function pointer in that *pFCT struct, since the call looks awfully as a C++ method call. Your question is tagged C, so the only way to make it work in C is to introduce a function pointer member into the struct.