From my investigation, all calls into the RIA service classes are forced through a custom IOperationInvoker class (enforced by a custom IOperationBehavior class). This invoker calls into DomainService to have the operation executed.
Before it is executed, the method call is validated any/all AuthorizationAttribute attributes marked on the operation in question. Each AuthorizationAttribute (two provided are RequiresAuthenticationAttribute & RequiresRoleAttribute) is given an opportunity to accept or reject the call via the abstract IsAuthorized method.
If any of these attributes returns something other than "AuthorizationResult.Allowed", an UnauthorizedAccessException exception is thrown with the ErrorMessage from the AuthorizationResult returned.