Why is the same Public Key generated even with different key container names?

Question

private readonly string KeyContainerName = "MyKeyContainer";

public void GenerateKeyPair()
        {

            Debug.WriteLine("Generating Key Pair");

            CspParameters cp = new CspParameters();           

            cp.KeyContainerName = KeyContainerName;

            cp.Flags = CspProviderFlags.UseDefaultKeyContainer;

            RSACryptoServiceProvider rsaProvider = new RSACryptoServiceProvider(1024,cp);

            rsaProvider.PersistKeyInCsp = true;

            byte[] pubkey = rsaProvider.ExportCspBlob(false);

            string PublicKey = Convert.ToBase64String(pubkey);

            if (!x.checkSettingExists("PublicKey"))
                x.addSetting("PublicKey", PublicKey);
            else
                x.changeSetting("PublicKey", PublicKey);


        }

x is a XML Utility class that i created which will save the the Public Key to an XML file.

However my problem is that even if i change the Key Container name, the public key generated is ALWAYS the same... I am under the impression that changing the key container would generate a fresh pair of private/public key pair.

Note that i am using Dot Net Compact Framework 3.5 :)

What am i doing wrong? Any insight to it would be highly appreciated :)

To make it even worse, i tried with a physical device which actually generates the same public key... I really don't know how that is possible...