First, let's think through what is happening. When you submit the original page, you are taking a "normal" Request/Response trip to get the code. On the page is a JQuery AJAX bit that fires off what is essentially a modal dialog. The desired effect is the user plays with the new page until they have figured out their filters and submits back. The problem is this "modal page" loses information when someone paginates.
The solution to this is fairly simple, in theory. You have to store the "filters" in the popped up page so they can be resent, along with pagination information. OR you have to cache the result set while the user paginates.
What I would do to solve this is create a static page that has the "filters" in place and work out the AJAX kinks separate from having the page post back to a parent page. Once you have all of the AJAX bits working properly, I would then link it into the popup routine and make sure the pagination is still non-problematic. THe final problem is creating a JavaScript routine that sends back to the parent page and allows the parent page to send its JQuery bits back to the server.
I am not sure about the HTML DIV part of the equation and I think you can solve the problem without this solution. In fact, I believe you can make the "modal popup" page without invoking AJAX, if it is possible to either a) submit the filters to apply via the querystring or b) fake a form submit to the second page. The query string is an easier option, but it exposes some info. Faking a form submit is not that difficult, overall, but could be problematic with a popup.
I am just firing off some ideas, but I hope it spurs something for you.