The <p> element has a margin-top by default, which pushes the whole content down.  You can get rid of it using margin-top: 0 on that <p>.
It is because <p> is block element and always starts in new line.
if you want your code to work with the <p> tag then add to you CSS
p
{
    display:inline;
}
This is kinda lame, but I actually found it myself.
Once again it's the collapsing borders tricking me. Here is a good forum post that describes it.
In my design it was appropriate to remove the margin from the <p> element.
From what I read the next best thing is giving the parent a tiny margin of it's own.
All browsers load a default stylesheet before any styles you define.
This is the style you see when you load a "plain" HTML into your browser.
You should look into and use a CSS reset in pretty much all your projects.
Yep there will be a gap at the top, because paragraph leaves some space before and after. The following removes the space try it:
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
<html xmlns="http://www.w3.org/1999/xhtml">
        <head>
                <meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
                <title>Test Lay-Out</title>                     
                <style>
                        html{
                                overflow: auto; /*for IE*/
                                height: 100%;
                        }
                        body{
                                background-color: orange;
                                margin: 0;
                                height: 100%;
                        }
                        #page{
                                background-color: green;
                                height: 100%;
                                margin-bottom: -50px;
                        }
                        #page p.first{
                                margin-top:0;
                        }
                        #footer{
                                background-color: red;
                                height: 50px;
                        }
                </style>
        </head>
        <body>
                <div id="page">
                 <p class="first">test</p>
                </div>
                <div id="footer">footer</div>
        </body>
</html>