The nested ViewModel is set to the MainWindow DataContext:
var mainWindow = new MainWindow();
mainWindow.Show();
mainWindow.DataContext = new
{
    MyProperty = new
    {
        MySubProperty = "Hello"
    }
}
It is easy to bind to MySubProperty in XAML:
<Button Content="{Binding MyProperty.MySubProperty}"/>
How can I do this binding in code behind?
// MyButton.xaml.cs
public partial class MyButton : Button
{
    public MyButton()
    {
        InitializeComponent();
        // todo: add binding here
    }
    // I want this method called if this datacontext is set.
    // and called if MySubProperty changes and INotifyPropertyChange is implemented in the Datacontext.
    public void MySubPropertyChanged(string newValue)
    {
        // ...
    }
}
I have no access to MainWindow in MyButton.xaml.cs, so I cannot use it as a source.
The Button is just an example, but it would be a start. In my original scenario I have no useful dependency property for that. If a dp is necessary for such a binding, an example would be very helpful that includes the creation of a dp.