Not possible with CGAffineTransform. An affine transform can always be decomposed into translations, rotations, shearing and scaling. They all map parallelograms into parallelograms, which your transform does not.
For your transform, it can be done in two steps. One to convert the square into a trapezoid.
p1-----p2 p1-----p2
| | --> | \
p3-----p4 p3--------p4'
Another to the vertical direction. A naive transformation rule is
y - c
x' = (x - p1.x) * ———————— + p1.x
p1.y - c
y' = y
where c is the y-coordinate of the intersection point of the lines joining p1 and p3, and p2 and p4.
Now notice the x*y factor in the transformation. This indicates such a transform is not linear. Therefore, CATransform3D cannot perform this as a 2D transform either.
However, the vector
[x, y, z, w=1]
will be converted to the actual 3D vector
(x/w, y/w, z/w)
before projection if CA follows usual 3D compute graphics rules, so you could "cheat" by using the transform
[ P . . Q ] [ x ] [ x' ]
[ . R . S ] [ y ] = [ y' ]
[ . . 1 . ] [ z ] [ z' ]
[ . T . U ] [ 1 ] [ w' ]
with appropriate P, Q, R, S, T, U that maps the 4 points to the expected locations. (6 unique coordinates and 6 variables should have exactly 1 solution most of the cases.)
When you have found these 6 constants, you can craft a CATransform3D. Notice the structure definition is
struct CATransform3D
{
CGFloat m11, m12, m13, m14;
CGFloat m21, m22, m23, m24;
CGFloat m31, m32, m33, m34;
CGFloat m41, m42, m43, m44;
};
typedef struct CATransform3D CATransform3D;
So you can directly change the matrix elements, instead of relying on the CATransform3DMake functions. (You may need to perform a transpose due to convention of using row or column vectors.)
To obtain the transform to convert a rectangle ((X, Y), (W, H)) to any quadrilateral ((x1a, y1a), (x2a, y2a); (x3a, y3a), (x4a, y4a)), use this function (you may need a transpose):
function compute_transform_matrix(X, Y, W, H, x1a, y1a, x2a, y2a, x3a, y3a, x4a, y4a) {
var y21 = y2a - y1a,
y32 = y3a - y2a,
y43 = y4a - y3a,
y14 = y1a - y4a,
y31 = y3a - y1a,
y42 = y4a - y2a;
var a = -H*(x2a*x3a*y14 + x2a*x4a*y31 - x1a*x4a*y32 + x1a*x3a*y42);
var b = W*(x2a*x3a*y14 + x3a*x4a*y21 + x1a*x4a*y32 + x1a*x2a*y43);
var c = H*X*(x2a*x3a*y14 + x2a*x4a*y31 - x1a*x4a*y32 + x1a*x3a*y42) - H*W*x1a*(x4a*y32 - x3a*y42 + x2a*y43) - W*Y*(x2a*x3a*y14 + x3a*x4a*y21 + x1a*x4a*y32 + x1a*x2a*y43);
var d = H*(-x4a*y21*y3a + x2a*y1a*y43 - x1a*y2a*y43 - x3a*y1a*y4a + x3a*y2a*y4a);
var e = W*(x4a*y2a*y31 - x3a*y1a*y42 - x2a*y31*y4a + x1a*y3a*y42);
var f = -(W*(x4a*(Y*y2a*y31 + H*y1a*y32) - x3a*(H + Y)*y1a*y42 + H*x2a*y1a*y43 + x2a*Y*(y1a - y3a)*y4a + x1a*Y*y3a*(-y2a + y4a)) - H*X*(x4a*y21*y3a - x2a*y1a*y43 + x3a*(y1a - y2a)*y4a + x1a*y2a*(-y3a + y4a)));
var g = H*(x3a*y21 - x4a*y21 + (-x1a + x2a)*y43);
var h = W*(-x2a*y31 + x4a*y31 + (x1a - x3a)*y42);
var i = W*Y*(x2a*y31 - x4a*y31 - x1a*y42 + x3a*y42) + H*(X*(-(x3a*y21) + x4a*y21 + x1a*y43 - x2a*y43) + W*(-(x3a*y2a) + x4a*y2a + x2a*y3a - x4a*y3a - x2a*y4a + x3a*y4a));
return [[a,b,0,c],[d,e,0,f],[0,0,1,0],[g,h,0,i]];
}