I post this as an answer because I can't put images into comments.
Sattle point
Here's an image of two brothers camping:
They placed their simple tents right beside each other in the middle of a steep walley (that's one bad place for tents, but thats not the point), so one end of each tent points upwards. At the point where the four squares meet you have a sattle point. The two edges on top of each tent can be rounded normally as well as the two downward edges. But at the sattle point you have different curvature in both directions and therefore its not possible to use a sphere. This rules out Svante's solution.
Selfintersection
The following image shows some 3D polygons if viewed from the side. Its some sharp thing with a hole drilled into it from the other side. The left image shows it before, the right after rounding.
.
The mass thats get removed from the sharp edge containts the end of the drill hole.
There is someething else to see here. The drill holes sides might be very large polygons (lets say it's not a hole but a slit). Still you only get small radii at the top. you can't just scale your polygons, you have to take into account the neighboring polygon.
Convexity
You say you're only removing mass, this is only true if your geometry is convex. Look at the image you posted. But now assume that the viewer is inside the volume. The radii turn away from you and therefore add mass.
NURBS
I'm not a nurbs specialist my self. But the constraints would look something like this:
The corners of the nurbs patch must be at the same position as the corners of the scaled-down polygons. The normal vectors of the nurb surface at the corners must be equal to the normal of the polygon. This should be sufficent to gurarantee that the nurb edge will be a straight line following the polygon edge. The normals also ensure that no visible edges will result at the border between polygon and nurbs patch.
I'd just do the math myself. nurbs are just polygons. You'll have some unknown coefficients and your constraints. This gives you a system of equations (often linear) that you can solve.