It's all about static type safety - whether, at compile time, we can safety know the type of an expression. With a literal, the compiler can correctly tell that if it can be converted to a byte. In byte a = 20, 20 is convertible, so it all goes through fine. byte a = 257 won't work (257 can't be converted).
In the case byte b = a, then we already know a is a byte, so type safety is assured. b = 100 is again fine (it's statically known that 100 is convertible).
In b = a + 100, it is not statically known if a + 100 is a byte. a could contain 200, so a + 100 is not representable as a byte. Hence the compiler forces you to tell it "Yes, a + 100 is always a byte" via a cast, by appealing to your higher level programmer knowledge.
Some types of more advanced type systems don't suffer from this problem, but come with their own problems that most programmers won't like.