Since hexadecimal is usually used only for integer numbers, you can parse the string to the largest integer type (Int64) and change the type of the result to the desired type:
string hexString = "deadcafebabe0000";
long hexValue = long.Parse(hexString, NumberStyles.AllowHexSpecifier);
Type hexType = typeof(Int32);
object o = Convert.ChangeType(hexValue, hexType);
(Note that you need to strip out the 0x prefix before passing the string to the Parse method.)
Convert.ChangeType is essentially a big pile of if (type == ...) ... else if (type == ...) statements. You could create a dictionary that maps all integer types to their respective Parse method:
var dict = new Dictionary<Type, Func<string, object>>
{
{ typeof(byte), s => byte.Parse(s, NumberStyles.AllowHexSpecifier) },
{ typeof(sbyte), s => sbyte.Parse(s, NumberStyles.AllowHexSpecifier) },
{ typeof(short), s => short.Parse(s, NumberStyles.AllowHexSpecifier) },
{ typeof(ushort), s => ushort.Parse(s, NumberStyles.AllowHexSpecifier) },
{ typeof(int), s => int.Parse(s, NumberStyles.AllowHexSpecifier) },
{ typeof(uint), s => uint.Parse(s, NumberStyles.AllowHexSpecifier) },
{ typeof(long), s => long.Parse(s, NumberStyles.AllowHexSpecifier) },
{ typeof(ulong), s => ulong.Parse(s, NumberStyles.AllowHexSpecifier) },
};
string hexString = (-5).ToString("X");
Type hexType = typeof(Int32);
object o = dict[hexType](hexString);