How to detect element transformation type (scale, move, rotate) for rotated elements depending on cursor position?

Question

The situation

I've written the following method (in actionscript 3, it's a flash application) which returns what transformation type is required for the current mouse position in relation to a given element position.

An element can be moved, scaled and rotated. This method returns if any of these transformations are applicable with the given coordinates:

(don't worry, you don't need to debug this method, it works - just added it here for you to understand what I am doing)

//returns whether the mouse is at a transforming position
        private static function mouseAtTransformPosition(cX:Number, cY:Number, eX:Number, eY:Number, eW:Number, eH:Number, margin:Number):String
        {
            //initialize the transformation type
            var transformType:String = null;

            //set the transformation type depending on the given coordinates
            if ((cX > eX) && (cX < eX + eW) && (cY > eY) && (cY < eY + eH))
            {
                transformType = "mover";
            }
            else if ((cX > eX) && (cX < eX + eW) && (cY <= eY) && (cY >= eY - margin))
            {
                transformType = "scalerUp";
            }
            else if ((cX >= eX + eW) && (cX <= eX + eW + margin) && (cY > eY) && (cY < eY + eH))
            {
                transformType = "scalerRight";
            }
            else if ((cX > eX) && (cX < eX + eW) && (cY >= eY + eH) && (cY <= eY + eH + margin))
            {
                transformType = "scalerDown";
            }
            else if ((cX <= eX) && (cX >= eX - margin) && (cY > eY) && (cY < eY + eH))
            {
                transformType = "scalerLeft";
            }
            else if ((cX >= eX - margin) && (cX <= eX) && (cY >= eY - margin) && (cY <= eY))
            {
                transformType = "scalerUpLeft";
            }
            else if ((cX >= eX + eW) && (eX <= eX + eW + margin) && (cY >= eY - margin) && (cY <= eY))
            {
                transformType = "scalerUpRight";
            }
            else if ((cX >= eX + eW) && (cX <= eX + eW + margin) && (cY >= eY + eH) && (cY <= eY + eH + margin))
            {
                transformType = "scalerDownRight";
            }
            else if ((cX >= eX - margin) && (cX <= eX) && (cY >= eY + eH) && (cY <= eY + eH + margin))
            {
                transformType = "scalerDownLeft";
            }
            else if ((cX >= eX - margin * 2) && (cX <= eX) && (cY >= eY - margin * 2) && (cY <= eY))
            {
                transformType = "rotatorTopLeft";
            }
            else if ((cX >= eX + eW) && (cX <= eX + eW + margin * 2) && (cY >= eY - margin * 2) && (cY <= eY))
            {
                transformType = "rotatorTopRight";
            }
            else if ((cX >= eX + eW) && (cX <= eX + eW + margin * 2) && (cY >= eY + eH) && (cY <= eY + eH + margin * 2))
            {
                transformType = "rotatorBottomRight";
            }
            else if ((cX >= eX - margin * 2) && (cX <= eX) && (cY >= eY + eH) && (cY <= eY + eH + margin * 2))
            {
                transformType = "rotatorBottomLeft";
            }

            //return the found transformation type
            return transformType;
        }

An elaboration of all the parameters required:

cX: cursor X position
cY: cursor Y position
eX: element X position
eY: element Y position
eW: element width
eH: element height
margin: how far the cursor can be removed from an exact transformation location to still be applicable

If no transformation is applicable, it will return null.

The problem

Now, my function seems to work great. The problem arises now that I am introducing rotation to the elements that need to be transformed.

When you rotate an element, the corners move to a different position. The cursor now has to be at different positions to get the same transforming types depending on the situation, just like my above method does (if there is no rotation):

cursor is...
  inside the element -> move
  at top of element -> scale up
  at right of element -> scale right
  at bottom of element -> scale down
  at left of element -> scale left
  at top left of element -> scale up-left
  at top right of element -> scale up-right
  at bottom left of element -> scale down-left
  at bottom right of element -> scale down-right
  further top left of element -> rotate right/left
  further top right of element -> rotate right/left
  further bottom right of element -> rotate right/left
  further bottom left of element -> rotate right/left

The question

My method does this perfectly, if there is no rotation. However, how do I alter this function to take rotation into consideration, for example the change of coordinates of the corners?

Update for Justin

//checks whether mouse transforming is applicable
        private static function checkMouseTransforming(cX:Number, cY:Number, render:Sprite):void
        {           
            //temp disable rotation to get accurate width/height, x and y
            var realRotation:Number = render.getChildAt(0).rotation;
            render.getChildAt(0).rotation = 0;

            var eX:Number = render.x;
            var eY:Number = render.y;
            var eW:Number = render.width;
            var eH:Number = render.height;
            var margin:uint = 10;

            //restore real rotation
            render.getChildAt(0).rotation = realRotation;

            cX -= eX + eW / 2;
            cY -= eY + eH / 2;

            var theta:Number = render.getChildAt(0).rotation * Math.PI / 180;
            var newcX:Number = Math.cos(theta) * cX - Math.sin(theta) * cY;
            var newcY:Number = Math.sin(theta) * cX + Math.cos(theta) * cY;

            newcX += eX + eW / 2;
            newcY += eY + eH / 2;

            var transformType:String = mouseAtTransformPosition(newcX, newcY, eX, eY, eW, eH, margin);
                        if (transformType)
                        {
                           //...
                        }

As you can see, the rotation is set and received from the child of the rendered object. This is because the actual object is a positionWrapper and inside this wrapper there's a rotationWrapper (which the rotation is set to) and inside that wrapper the actual visible element remains. The x, y, width and height of the position wrapper are always equal to those of the visible display object so this shouldn't cause problems.

Now, these are the results:

• Without rotation, everything works as expected
• With about 45 rotation, when I mouse-over at visual display object's:
  • bottom center: it returns "scalerLeft", while it should return "scalerDown"
  • right center: it returns scalerDown, while it should return scalerRight
  • top center: it returns scalerRight, while it should return scalerUp
  • left center: it returns scalerUp, while it should return scalerLeft (notice the offset pattern here)
  • the mover seems to return pretty accurately
  • the rotater seems to return at the direct bottom of a corner, not where it should
  • the bottomLeft scaler seems to return at the bottom right corner, I think this same offset is the problem and probably is the same for all other scalers too

This is a bugger to debug, but maybe this will help.

Update #2 for Justin The offset/bug seems to get stronger the higher the rotation is. With 0 rotation, this bug does not appear.

Does this make sense to you or anyone else? Thanks in advance.

Answer 1

If you use the mouseX/mouseY properties of the object you are transforming instead of those of it's parent (or the stage for that matter) you will get them in the objects coordinate space.
Thus, you will never have to worry about rotation or scaling, and can keep your code pretty much as it is.

I would also recommend using scaleX/scaleY to alter the size, since width and height will change as you rotate the object.

Answer 2

I've made a few assumptions here. I've assumed that the object is rectangular and two-dimensional (no 3D rotations). I have assumed that this is the only object that can be transformed to simplify things. Also, from the comments on the other answer, I understand that the dimensions of the object (eX, eY, eW, eH) are still listed as being the same after the rotation.

Now, here's my solution. You will definitely need to know the rotational angle (which I'll assume is positive for clockwise rotation) of the object which I will call theta. At first I thought that you should rotate the bounding boxes for the various areas (the area where the transformation is 'mover' for instance), but that is much more complicated. What you should do is rotate the x and y coordinates of the mouse around the center of object and in the opposite direction of theta. Here are some explicit steps:

1. Subtract the x-displacement and y-displacement of the object's center from the respective x- and y-coordinates (cX and cY). The resulting values give the x and y displacements away from the center of the object.

2. Rotate anticlockwise the resulting coordinates using the angle theta.

3. Add the x-displacement and y-displacement of the object's center to the resultant x and y coordinates from the previous step.

4. Use the resultant x and y values as the cX and cY values in your mouseAtTransformPosition function.

Now here's some code to illustrate this.

cX -= eX + eW/2;
cY -= eY + eH/2;

newcX = cos(theta)*cX - sin(theta)*cY;
newcY = sin(theta)*cX + cos(theta)*cY;

newcX += eX + eW/2;
newcY += eY + eH/2;

newcX and newcY will the coordinates actually used in mouseAtTransformPosition.

Let me know if this doesn't make sense or if you have any questions.

* Updated:

I think that you are rotating the coordinates the wrong way. To change the direction of rotation use:

newcX = cos(theta)*cX + sin(theta)*cY;
newcY = -sin(theta)*cX + cos(theta)*cY;

instead of what I have above. Try this and tell me how it goes.