The identity matrix, in terms of the projection and modelview matrices, essentially resets the matrix back to its default state.
As you hopefully know, glTranslate and glRotate are always relative to the matrix's current state. So for instance, if you call glTranslate, you are translating from the matrix's current 'position', not from the origin. But if you want to start over at the origin, that's when you call glLoadIdentity(), and then you can glTranslate from the matrix which is now located at the origin, or glRotate from the matrix which is now oriented in the default direction.
I think Boon's answer, that it is the equivalent of 1, is not exactly correct. The matrix actually looks like this:
1 0 0 0
0 1 0 0
0 0 1 0
0 0 0 1
That is the identity matrix. Boon is correct, mathematically, that any matrix multiplied with that matrix (or a matrix that looks like that; diagonal ones, all else 0s) will result in the original matrix, but I don't believe he explained why this is important.
The reason why this is important is because OpenGL multiplies all positions and rotations through each matrix; so when for instance you draw a polygon (glBegin(GL_FACE), some points, glEnd()), it translates it to "world space" by multiplying it with the MODELVIEW, and then translates it from 3D to 2D by multiplying it with the PROJECT matrix, and that gives it the 2D points on screen, along with the depth (from the screen 'camera'), which it uses to draw pixels. But when one of these matrices are the identity matrix, the points are multiplied with the identity matrix and therefore are not changed, so the matrix has no effect; it does not translate the points, it does not rotate them, it leaves them as-is.
I hope this clarifies a bit more!
-Ricky