You'll have read the Wikipedia page on the subject, of course.
Once upon an aeon or so ago, I read Foley and van Dam in one of the predecessor versions (this would have been 1983 or 1984), and it covered techniques for manipulating 2D and 3D coordinates with augmented matrices and vectors as described in the question. However, enough time has lapsed since then that I've forgotten all the details (and no longer have the book--too many moves of house). There was also a book by Newman and Sproul, I seem to remember.
A = [ a b c ] B = [ 0 1 1 0 ] C = [ 0 1 2 1 ]
[ d e f ] [ 0 0 1 1 ] [ 1 1 2 2 ]
[ g h 1 ] [ 1 1 1 1 ] [ 1 1 1 1 ]
The columns of B represent the corners of the square; the columns of C represent the corners of the parallelogram; and the matrix equation A x B = C has to be solved. IIRC, the matrix A has a 1 in the bottom right corner; it is possible that the values c, f, g, and h also have presecribed values (they'd probably be zeroes). The non-zero values apply a linear (affine) transform, scaling, shearing and rotating the input shape.
You'd need to look for similar information in a text book. Or in the Wiki page - I didn't look hard at it (the information above is working from ancient memory).