Unfortunately all of my AppEngine knowledge is using their Python SDK but it "should" translate to Java.
You need to build a user system to start with. I have built a few as a wrapper around the Google account system but if you don't want Google accounts then you can build a simple User table and session system yourself. The concepts are pretty simple but you can see how one is built on top of AppEngine by taking a look here: http://github.com/aht/suas
That example user system has bugs (the cookie stuff) but you won't care as you won't be using cookies. Really you just need the ability to store user accounts, authenticate against those accounts, start a session with an authenticated account, and pass the session key back to the client app. The client app then uses that session key to authenticate in the future. The parts to do most of that can be gleamed from the suas example I gave above.
Once you have the ability to start and authenticate sessions then really you just need the ability to do RPCs to the server from the app. How you initiate the RPC depends on the app platform but when talking to AppEngine you need to talk HTTP. Basically the client will be making HTTP POST requests to the server with the body of the POST being a JSON/XML object containing the session key, the name of the function you wish to call, and any arguments to that function. The response from the server will be a HTTP response with the body of the response being just a JSON/XML object. In Python you can use the simplejson API to convert Python dicts to JSON easily and there is an XML lib that works similarly. There must be an equivalent Java API.
An example of doing RPC requests over HTTP to AppEngine (again in Python, sorry) can be fond here:
http://code.google.com/appengine/articles/rpc.html
You can skip all the client stuff unless you are using JavaScript (which is actually a great way to prototype a test client for this). The part you will be interested in is how the server figures out which function to call and how it responds:
self.response.out.write(simplejson.dumps(result))
Hopefully some of that gets you jump started on this project. Good luck!