The difference in behaviour you're seeing is the difference between identity and representation.
Unboxing is an identity cast, and a representation-preserving operation. Casting an int to a byte, however, is representation-changing (since there is a potential loss of precision).
You get an InvalidCastException when you try to unbox the int as a byte because the identity of the boxed value is not a byte, it is an int. When you write byte b = (byte)obj, you are telling the runtime, I know that what's in there is a byte, but what you really mean to say is, I think that what's in there can be converted to a byte.
In order to make the latter statement, you first have to declare the identity of the object, which is an int. Then and only then can you make a representation-changing conversion to byte.
Note that this applies even if the target type is "larger" - i.e. an Int64. All explicit conversions for which the destination type is not in the inheritance tree of the source type are considered to be representation-changing. And since all types derive from System.Object, unboxing by definition cannot change the representation.