JPEG is lossy compression. A bit of oversimplification, but it works by analyzing pixels around other pixels to see how similar they are. Not every pixel is stored, and that means it isn't possible to simply chop bytes out of the image data to perform the crop. If you are outputting JPEG, you will be re-compressing an already compressed image, and you will have some loss in quality. However, if you crop the image and your output is a non-lossy format, then you will not have loss of quality.
To be clear, the loss of the quality isn't in the crop operation. It is in the way the image is compressed itself. If the source image is compressed with JPEG, quality has already been lost. When you crop that image, you aren't losing anything more, but if you were to output JPEG again afterwards, this would require a re-compression, and thus more loss.
So in the end, make your final output PNG or something non-lossy and you have nothing to worry about.