For window.open you could check to see if the page you are currently on has a parent.
function parentExists(){
return (window.opener != null)? true : false;
}
Call this when your login page loads. If it returns true, you are in a popup window (or modal). You can then close the page and redirect the parent.
It's a little more tricky for a modal box because you don't have access to the opener. First, make sure all modal boxes are opened similar to this:
window.showModalDialog('test.htm', self, <optional options>);
This will make sure something is passed into the window's arguements.
Now add the following code on your login page:
function parentExists()
{
var opener = window.dialogArguments;
return (opener == null)?false:true;
}
Edit: added information on modal boxes