The various answers here are generally correct, but have a bunch of different facts spread out all over the place. The relevant points are:
1) The result of byte | byte is an int, because there is no | operator defined on bytes.
2) Computations involving only integral compile-time constants are treated as "checked" arithmetic; that is, the compiler verifies that the constant result does not overflow, and so on. A pleasant consequence of this fact is that an assignment of a compile-time constant integer to a variable or constant of a smaller type automatically verifies that the constant integer fits into the smaller type. If it does, then the assignment is allowed without an explicit cast. If it does not, then a compile-time error occurs. (Use the "unchecked" expression if you want to override this behaviour.)
3) Assignments from non-constant expressions of type int to byte require casts, because the compiler has no way of knowing that the result definitely fits in the byte.
4) The compound assignment operators automatically insert a cast to the operand type as part of their operation, precisely so that expressions like b |= whatever work as you'd expect.
Those four facts should explain all the behaviour that you guys have pointed out.