You will simply need to do a type check for each of the basic numeric types.
Here's an extension method that should do the job:
public static bool IsNumber(this object value)
{
if (value is sbyte) return true;
if (value is byte) return true;
if (value is short) return true;
if (value is ushort) return true;
if (value is int) return true;
if (value is uint) return true;
if (value is long) return true;
if (value is ulong) return true;
if (value is float) return true;
if (value is double) return true;
if (value is decimal) return true;
return false;
}
This should cover all numeric types.
Update
It seems you do actually want to parse the number from a string during deserialisation. In this case, it would probably just be best to use double.TryParse.
string value = "123.3";
double num;
if (!double.TryParse(value, out num))
throw new InvalidOperationException("Value is not a number.");
Of course, this wouldn't handle very large integers/long decimals, but if that is the case you just need to add additional calls to long.TryParse / decimal.TryParse / whatever else.