divide the rect into 4 smaller rectangles. cut this in half diagonally (before rotation, from a corner to a center point) and you will have 8 triangles. You only need 4 of them. After your rotation, the hypotenuse of these triangles are sticking out of the original bounds box.
determine a formula for the hypotenuse (you have the original angle, 45, -45, 135, -135, and the starting point, so that's mx+b), transform those lines (modify their slopes by adding the rotation), intersect those lines with the boundary walls (y = 0, y = w, x = 0, x = h, distance formula, test for infinite cases) & figure out which hypotenuse is shortest (center to wall, at the corner). Since all the hypotenuse were the same length to start with, just resize all of them to this new value, you have your new rectangle.
Am I doing it right?