IDataErrorInfo with ValueConverter

Question

I'm somehow doing it wrong, but I can't figure it out:

I have a model like this:

public class Person : IDataErrorInfo {
  public DateTime Birthdate {
    get { return _birthdate; }
    set {
      if (!Valid(value))
        AddError("Birthdate", "Birthdate not valid");

      _birthdate = value;
    }
  }
}

A ValueConverter like this:

public class DateToStringConverter : IValueConverter {
  public object Convert(object value, Type targetType, object parameter, System.Globalization.CultureInfo culture) {
    return date.Date.ToShortDateString();
  }

  public object ConvertBack(object value, Type targetType, object parameter, System.Globalization.CultureInfo culture) {
    DateTime result;

    if (DateTime.TryParse(text, out result))
    {
      return result;
    }

    return DependencyProperty.UnsetValue;
  }
}

And a View like this:

<TextBox Text="{Binding Person.Birthdate,
                        Mode=TwoWay,
                        Converter={StaticResource DateToStringConverter},
                        ValidatesOnDataErrors=True}" />

If someone modifies an valid date like "1.1.1950" into an invalid date like "1.1.abc", the value does not get through to the Person and doesn't invalidate it. But a red border around the birthdate-textbox is shown. How can I keep the invalid text (to be modified by the user) and register an error for IDataErrorInfo?