Does anyone know what the core drupal javascript functions are to force an error message? More importantly, it must work with Message Effects (http://drupal.org/project/messagefx).
With JS Theming you can call these: (this is javascript code)
Drupal.messages.set("The Message!", 'error');
But, it doesn't hook into Message Effects, and after disabling JS theming, this function no longer works.
It seems like you have asked the wrong question. The status, warning, error messages that you see, is not invoked by javascript. Instead they are generated by the drupal_set_message() function. It work, by saving the messages in the session table for that user, and then they get printed whenever that user (re)load a page. It ends up as $messages in the page.tpl.php. I believe that there are no javascript version of this functionality, but you could mimic it. It's no harder than to insert some html on the page, something like.
$("#message").append('<div class="message error">The error message</div>');
I'm not sure about the exact html structure, and it cal also vary from theme to theme, but you get the idea. If you do take this approach, you will also need to hook into the messagefx module's js, to add the effect, but this shouldn't be too hard.