This is a fairly frequently asked question.
In C#, we almost always reason from inside to outside. When you see
x = y;
we work out what is the type of x, what is the type of y, and whether the type of y is assignment compatible with x. But we do not use the fact that we know what the type of x is when we are working out the type of y.
That's because there might be more than one x:
void M(int x) { }
void M(string x) { }
...
M(y); // y is assigned to either int x or string x depending on the type of y
We need to be able to work out the type of an expression without knowing what it is being assigned to. Type information flows out of an expression, not into an expression.
To work out the type of the conditional expression, we work out the type of the consequence and the alternative expressions, pick the more general of the two types, and that becomes the type of the conditional expression. So in your example, the type of the conditional expression is "int", and it is not a constant (unless the condition expression is constant true or constant false). Since it is not a constant, you can't assign it to byte; the compiler reasons solely from the types, not from the values, when the result is not a constant.
The exception to all these rules is lambda expressions, where type information does flow from the context into the lambda. Getting that logic right was very difficult.