Have you seen this article on CodeProject?
From an admittedly quick look at the page it looks like 4D maths. You can adopt a similar approach to contstructing matrices as you would for 2D or 3D maths.
Take a series of source "points" - in this case you'll need 4 - and the corresponding target "points" and generate a matrix. This can then be applied to any "point".
To do this in 2D (from memory so I could have made a complete howler in this):
Source points are (1, 0) and (0, 1). The targets are (0, -1) and (1,0). The matrix you need is:
(0, -1, 0)
(1, 0, 0)
(0, 0, 1)
Where the extra information is for the "w" value of the coordinate.
Extend this up to {R, G, B, A, w} and you'll have a matrix. Take 4 colours Red (1, 0, 0, 0, w), Green (0, 1, 0, 0, w), Blue (0, 0, 1, 0, w) and Transparent (0, 0, 0, 1, w). Work out what colours they map to in the new scheme and build up your matrix as follows:
(R1, G1, B1, A1, 0)
(R2, G2, B2, A2, 0)
(R3, G3, B3, A3, 0)
(R4, G4, B4, A4, 0)
(0, 0, 0, 0, 1)
NOTE: The order you do you mulitplication (vector * matrix or matrix * vector) will determine whether the transformed points go vertically or horizontally into this matrix, as matrix multiplication is non-commutative. I'm assuming vector * matrix.