I have not noticed any slowdown. We are doing a LOT of binding to INotifyPropertyChanged ViewModels and the UI seems to be extremely responsive.
Sure, there will be a hit for data binding vs direct data access... but that hit is so small that the benefit you get from data binding makes the small hit inconsequential.
Something to remember: The data binding is happening in the UI. There isn't a lot of high intensity processing happening at that layer. In addition, the UI renders on a separate thread. Those two things together make for an experience that feels very responsive, in my opinion.
Erik asked if you have any Value Converters in place? I would ask the same thing. If so, are they doing a lot of work? In my experience with MVVM, value converters are rarely needed anymore. Just some food for thought.