I wrote a program for retrieving mail from POP3 servers. One of its users encounters the following problem...
His mail server when my program connects to it issues a greeting:
+OK Lotus Notes POP3 server version X2.0 ready <PrintableCharacters.UnprintableCharacters>
The part in angle brackets is a so-called challenge for the APOP authorization. The challenge (and all other strings in POP3 protocol) typically only contains ASCII characters. But this time the challenge is composed of two parts - the second one contains non-printable characters. The pattern is the following: first 0x5, then some non-ASCII character. This looks like special encoding introduced by Lotus Notes developers.
When I extract the challenge as-is and use it for APOP authentication the server declines.
After carefully reading all I could find on LMBSC I deducted the following. The 0x5 is a prefix to indicate so-called "Group 5" characters. Each character is in fact a Win-1251 character but is prepended by 0x5.
With that I just stripped away all 0x5's only leaving legal ASCII and non-printable Win-1251 and passed those into usual APOP authentication algorithm. However the server declines.
Is that the right way to decode LMBSC with Win-1251 characters or is that a problem on the server side?