Getting random output from Crypto++

Question

Can't figure out why I am getting seemingly random output from the Crypto++ RC2 decoder. The input is always the same, but the output is always different.

const char * cipher     ("o4hk9p+a3+XlPg3qzrsq5PGhhYsn+7oP9R4j9Yh7hp08iMnNwZQnAUrZj6DWr37A4T+lEBDMo8wFlxliuZvrZ9tOXeaTR8/lUO6fXm6NQpa5P5aQmQLAsmu+eI4gaREvZWdS0LmFxn8+zkbgN/zN23x/sYqIzcHU");
int          keylen     (64);

unsigned char keyText[] = { 0x1a, 0x1d, 0xc9, 0x1c, 0x90, 0x73, 0x25, 0xc6, 0x92, 0x71, 0xdd, 0xf0, 0xc9, 0x44, 0xbc, 0x72, 0x00 };
std::string key((char*)keyText);

std::string data;
CryptoPP::RC2Decryption rc2(reinterpret_cast<const byte *>(key.c_str()), keylen);
CryptoPP::ECB_Mode_ExternalCipher::Decryption rc2Ecb(rc2);
CryptoPP::StringSource
    ( cipher
    , true
    , new CryptoPP::Base64Decoder
        ( new CryptoPP::StreamTransformationFilter
            ( rc2Ecb
            , new CryptoPP::StringSink(data)
            , CryptoPP::BlockPaddingSchemeDef::NO_PADDING
            )
        )
    );

std::cout << data << '\n';

Answer 1

RC2Decryption should have been defined as:

CryptoPP::RC2Decryption rc2(reinterpret_cast<const byte *>(key.c_str()), key.size(), keylen);

Answer 2

The parameters to the RC2::Decryption constructor are: (pointer to key-bytes, length of key-bytes). You are giving it a pointer to 16 bytes but using a length of 64 bytes. Crypto++ is reading uninitialized memory when reading the key, so you get random results.

If you want to indicate an effective key-length, you can use the other constructor like this:

CryptoPP::RC2Decryption rc2(keyText, 16, keylen);

Note that you should not use a std::string to hold your key. It is completely legal for a key to contain a 0x00-byte, and std::string is not designed to hold those.