Chrisc,
To follow up on adding a new provider, you can add the files anywhere you want, including within an application. Android already has multiple providers registered in different packages. For example, in dalvik/libcore/security/src/main/java/java/security/Security.java
// Register default providers
private static void registerDefaultProviders() {
secprops.put("security.provider.1", "org.apache.harmony.security.provider.cert.DRLCertFactory"); //$NON-NLS-1$ //$NON-NLS-2$
secprops.put("security.provider.2", "org.apache.harmony.security.provider.crypto.CryptoProvider"); //$NON-NLS-1$ //$NON-NLS-2$
secprops.put("security.provider.3", "org.apache.harmony.xnet.provider.jsse.JSSEProvider"); //$NON-NLS-1$ //$NON-NLS-2$
secprops.put("security.provider.4", "org.bouncycastle.jce.provider.BouncyCastleProvider"); //$NON-NLS-1$ //$NON-NLS-2$
}
and also redundantly in libcore/security/src/main/java/java/security/security.properties
However, even if you add your own classes into the same package, it won't affect the existing provider. That is because the Provider classes such as in this case JSSEProvider are what define what classes are used to provide various algorithm implementations (through the properties you see put above)
In your case, you probably want to provide a new SSLContext implementation. You can see in JSSEProvider that it registers SSLContextImpl to provide a "TLS" (aka SSL) implementation of SSLContext:
put("SSLContext.TLS", SSLContextImpl.class.getName());
So you could add your own Provider that register an alternative implementation.
You do not need to to add your Provider to Security.java or security.properties, but you can use Security.addProvider to add a new one, or even Security.insertProviderAt to control where your Provider will appear in priority order.
Once you have your own Provider registered, you can ensure you get an instance from your new Provider by using SSLContext.getInstance("SSLContext", "your-provider-name"). Alternatively if you provider is made higher priority than the built in one, all programs calling SSLContext.getInstance("SSLContext") will get your new implementation.
One other note is that SSLSocket's can also be created with SSLSocketFactory.getDefault(). That can be overriden to use your own SSLSocketFactory with ssl.SocketFactory.provider=org.apache.harmony.xnet.provider.jsse.OpenSSLSocketFactoryImpl (looking at SSLSocketFactory.getDefault). I'm not sure that can be done within an application, unlike adding a new provider.
Finally, as far as the C++ code, you can obviously provide your own JNI native code. If you are just doing it in an app, you'd register it like any other JNI code (or have it autoloaded) but I don't have the details on that handy since I work with the built in code most of the time. As with most native code, you would need to avoid symbol name collisions if you are building your code into the VM like the current code. We try to make as much of the code staticly scoped to the file to avoid such issues. However, providing a different version of libssl.so itself is another matter entirely. If you need to do something like that, I don't have any tested device.
You might find this follow document useful as well for a background adding new providers:
How to Implement a Provider in the Java™ Cryptography Architecture
http://download-llnw.oracle.com/javase/6/docs/technotes/guides/security/crypto/HowToImplAProvider.html
-bri