If you are simply wanting to create modal-popup and call a partial submit I'd suggest using the Richfaces tag library which you may already be using.
To create a modal dialog simply use the tag to create the modal panel:
<rich:modalPanel id="myModalPanel" minHeight="100" height="100" minWidth="100" width="100" zindex="2000">
Insert content for modal panel here
</rich:modalPanel>
Now you can add a tag to a button or link to hide and show the modal panel:
<h:outputLink value="#" id="showLink">
<rich:componentControl for="myModalPanel" attachTo="showLink" operation="show" event="onclick" />
</h:outputLink>
<h:outputLink value="#" id="hideLink">
<rich:componentControl for="myModalPanel" attachTo="hideLink" operation="hide" event="onclick" />
</h:outputLink>
And to do an ajax call you would use the tag inside the tag that you would like to make the ajax call. So, for instance, if you wanted an action to fire when the text of a field changes you would do the following:
<h:inputText id="myField" value="#{myBean.myField}">
<a4j:support ajaxSingle="true" event="onChange" reRender="list fields you want to rerender when the methods completes" action="#{myBean.methodToCall}" />
</h:inputText>