I guess you have to consider what it really means for the application (or activity) to be running. If the user navigates away from your application, it could remain in memory or its process may be terminated to reclaim resources for foreground activities. This is all transparent to the user, since if they navigate back to your application and it was previously terminated it will be re-created for them.
If the expectation is that the password will always be remembered until the user manually "exits", then I believe you must accept that the password could be written to persistent storage (if the process is killed, for instance). This doesn't mean that the password has to be stored in clear text. However, then you have the additional problem of storing the key used to protect the password.