'0.0' is not a valid timestamp when accessing OldValue of TClientDataSet's TDateField

Question

When I use the following code in a newly inserted record in a TClientDataSet:

cdsMyDateField.OldValue <> Null

I get an EConvertError:

''0.0' is not a valid timestamp'.

Looking at Delphi's VCL's code, it tries to convert the value to TDateTime which results in this Exception because the value (Null) is an invalid DateTime, but as I'm comparing Variants I thought that it would return a variant, which would be Null in this case, but that doesn't happen, instead I get this exception.

I know that I can just check if the DataSet.State = dsInsert before comparing the values, as if the State = dsInsert every OldValue is Null, but I want to understand why the OldValue tries to convert the value instead of Just returning Null in all Fields when the State = dsInsert.

Can anyone please give me some light?

Answer 1

A TDateTime in Delphi is a double, where the date is stored in the whole number portion and the time is stored in the fractional portion. So a conversion of an empty date value to 0.0 is correct, to some extent. Since the underlying field you're accessing is a TDateField (or TDateTime field), it's probably doing the conversion internally.

In addition, checking a Variant against Null isn't the proper way in Delphi any more. A Null variant is still assigned, but has the value Null, whereas an unassigned variant has no value. (Think of an SQL database NULL value). Use the VarIsNull(const V: Variant) function found in the Variants.pas unit instead; it returns true if the variant is null, false if it has any other value.

Answer 2

I debugged the code below with Debug DCUs option activated and the weird thing is that SysUtils.ValidateTimeStamp evaluates a TimeStamp with date = 0 to be invalid and therefore throwing a EConvertError exception (instead of return Null or Unassigned).

So the end result is that doing an OldValue request to a null field in a dsInsert state is that its invalid. The variant is NEVER returned so it is irrelevant if you test it with (field.OldValue <> Null) or VarIsNull(field.OldValue). The exception gets thrown before.

The cds_something has two fields (created at design time):

• dt_Something
• num_something

Code:

var
  b: TClientDataset;
begin
  b := cds_somethin;
  b.Close;
  b.CreateDataSet;
  b.Insert;
  if b.FieldByName('DT_Sometinhg').OldValue <> Null then
    ShowMessage('Something wrong!!!')
  else
    ShowMessage('Normal');

  b.Cancel;

Note: I messed up with the original editon of this post. This now is the correct interpretation of what I find.

Addition: tested with some other field types (string, BCD, Float and Memo) and OldValue is Unassigned - so the test above will evaluate to false.

Appear that ONLY TDateField and TDateTimeField show that behavior. TTimeField and TSQLTimeStamp evaluates normal - but TSQLTimeStampField.OldValue does not equal neither to Null or Unassigned (wtf!!)...

The snippet changed a little:

var
  b: TClientDataset;
begin
  b := cds_somethin;
  b.Close;
  b.CreateDataSet;
  b.Insert;
  /*
  if (b.FieldByName('DT_Something').OldValue <> Null) 
     and (b.FieldByName('DT_Something').OldValue <> Unassigned)  then
    ShowMessage('Something wrong!!!')
  else
    ShowMessage('Normal');
  */
  if (b.FieldByName('ts_Something').OldValue <> Null) 
     and (b.FieldByName('ts_Something').OldValue <> Unassigned)  then
    ShowMessage('Something wrong!!!')
  else
    ShowMessage('Normal');


  b.Cancel;

Where ts_Something is a TSQLTimeStampField. The fields are created at design time.

Answer 3

Got it reproduced in Delphi 2009 update 2 and Delphi 2007 as follows:

uses DB, DBClient;
procedure TTestForm1.TestButtonClick(Sender: TObject);
const
  SMyDateField = 'MyDateField';
  SMyIntegerField = 'MyIntegerField';
var
  MyClientDataSet: TClientDataSet;
  MyClientDataSetMyDateField: TField;
  MyClientDataSetMyIntegerField: TField;
  OldValue: Variant;
begin
  MyClientDataSet := TClientDataSet.Create(Self);
  MyClientDataSet.FieldDefs.Add(SMyDateField, ftDate);
  MyClientDataSet.FieldDefs.Add(SMyIntegerField, ftInteger);
  MyClientDataSet.CreateDataSet();
  MyClientDataSetMyDateField := MyClientDataSet.FieldByName(SMyDateField);
  MyClientDataSetMyIntegerField := MyClientDataSet.FieldByName(SMyIntegerField);
  MyClientDataSet.Insert();
  OldValue := MyClientDataSetMyIntegerField.OldValue;
  OldValue := MyClientDataSetMyDateField.OldValue;
end;

You always get this error:

exception class EConvertError with message ''0.0' is not a valid timestamp'.

I'm not sure if this is to be regarded as a bug:

• upon insert there is technically no OldValue, so obtaining it can raise an exception
• MyClientDataSetMyIntegerField.OldValue returns 0, but MyClientDataSetMyDateField.OldValue raises an exception

Some more notes:

• TCustomClientDataSet.GetFieldData will get the actual underlying data
• TDataSet.DataConvert will convert the underlying data to the native data format, performing validity checks where needed

Edit: as a result of Fabricio's comment I have stressed that OldValue is technically invalid after an insert. So technically this might not be a bug.

His 'new evidence' can be verified by checking the VCL/RTL sources:

For fieldtypes ftDate, ftTime, ftDateTime, TDataSet.DataConvert calls its local NativeToDateTime function which fills a TimeStamp, then converts that using SysUtils.TimeStampToDateTime which in turn calls SysUtils.ValidateTimeStamp which raise an exception when the Time portion is less than zero, or the Date portion is less or equal to zero.

The Date portion can become zero only for fieldtypes ftDate and ftDateTime (TDateField and TDateTimeField), hence only those can raise an exception. For all other data types, NativeToDateTime will not have problems: those types all allow for a result filled with zero-bytes.

I just checked the RTL/VCL history: since Delphi 6 SysUtils.TimeStampToDateTime calls SysUtils.ValidateTimeStamp, so this behaviour has been the same since 2001.

That makes it really hard to regard this as a bug.

Answer 4

FWFIW, I've run into this same problem and it caused me several headaches. My opinion: The behaviour is inconsistent so even from that reason alone I class it as a bug. It's also a bug because raising an exception on reading a property is IMHO ridiculous and not in keeping with the intent of properties (principle of least surprise.) I'd expect OldValue to be Unassigned, not raise an exception on read. (Also, the fact that some issue has existed for a long time does not imply anything about whether or not it's a bug.)

(Edit: Updating my answer with more information including our workaround. Also posted same to the QC report.)

We've also had this same problem in an app that makes heavy use of datasnap/clientdatasets. The actual problem however is not with the clientdataset, but with the Timestamp validation routine in SysUtils that apparently erroneously validates a Timestamp with a 0.0 value as invalid.

The workaround/fix is hence to SysUtils and the validation routine, and not to Tclientdataset Specifically, in the validation routine "ValidateTimeStamp()", the Time portion is correctly compared < 0 but the Date portion is erroneously compared <= 0.

Consequently a (legitimate) 0.0 Datetime value will sometimes be converted to a Timestamp with datepart = 0, and when this value is again reverse validated (such as when the value is read from a dataset field as shown here and in the QC report), an exception is (erroneously) raised. Hence the simple fix of changing the validation for the Date portion of the timestamp to use strict less-than, fixes the problem exposed by TClientDataset.

Here's our workaround, based on Delphi 7.1

(* SysUtils.pas line 10934 (Delphi 7.1)  *)
(**)(* fix - Timestamp values 0.0 erroneously designated as invalid *)
(* D7.1 *)
(* Walter Prins, originally patched May 2005, submitted 4 June 2009 *)
procedure ValidateTimeStamp(const TimeStamp: TTimeStamp);
begin
  if (TimeStamp.Time < 0) or (TimeStamp.Date < 0) then (* Changed TimeStamp.Date <= 0 to TimeStamp.Date < 0 *)
    ConvertErrorFmt(@SInvalidTimeStamp, [TimeStamp.Date, TimeStamp.Time]);
end;

Answer 5

This is a bug on Midas/TClientDataSet related to record buffers. When you do AppendData many times, the InternalCalc fields appears as "not null" (wrong null flag in record buffer). In Delphi 2010 we can study Midas.dll sources (.cpp files).

http://img514.imageshack.us/img514/2840/wrongnull.jpg

I am researching how to implement a solution.

Al Gonzalez.