So the form should be processed right after user populates it? You don't want to wait until the user submits the form? Please elaborate.
Popping up can be accomplished easily. Have the form on the page from the start but hide it with CSS, for example like this:
.hidden-form {
height: 0;
text-indent: -999em;
}
Then, you can use click event to make the form appear:
$('#popup-link').click(function() {
$('#form').removeClass('hidden-form');
});
Displaying error messages is quite easy as well. Have a separate page that will process the form and print error messages if something goes wrong and success message when everything goes alright. Make an AJAX request with POST method of that page and just print the HTML output returned by the page on top of the form.
Show us some of your code (form and the server side processing page) for more specific help.