At the CPU level, there are no bytes, only words, which are 32bit or 64bit nowadays. Arithmetic units are usually hardwired to deal with word-sized numbers (or larger, in the case of floating point).
So there is no speed advantage in using types smaller than a word in regard to arithmetic operations, and there may be a speed penalty because you have to do additional work in order to simulate types that the CPU does not have natively, e.g. writing a single byte to memory requires you to first read the word it is part of, modify it and then write it back. In order to avoid this, most compilers will actually use a full word of memory for all smaller variables, so even a boolean variable takes up 32 or 64 bits.
However, if you have a large amount of data, such as a large array, then using smaller types will usually yield better performance because you'll have fewer cache misses.