Best option for Google App Engine Datastore and external database?

Question

I need to get an App Engine app talking to and sharing data with an external database,

The best option i can come up with is outputting the external database data to an xml file and then processing this in my app engine app and storing it inside the datastore,

although the data being shared is sensitive data such as login details so outputting this to an xml file is not exactly a great idea, is it possible for the app engine app to directly query the database? or is there a secure option for using xml files?

oh and im using python/django and the external database will be hosted on another domain

Answer 1

You may want to consider exposing a set of web services on the external domain where your database is hosted, and then use the App Engine's URL Fetch API to communicate with your external domain via HTTPS.

Answer 2

Google Apps' Secure Data Connector (SDC) is designed for this kind of tasks -- indeed, it even works when the "other database" lives behind a firewall (a common case for enterprise data), and for other Google Apps (Docs, Spreadsheets, ...) as well as App Engine.

As the docs summarize things, the flow is:

1. Google Apps forwards authorized data requests from users who are within the Google Apps domain to the Google tunnel protocol servers.

2. The tunnel servers validate that a user is authorized to make the request to the specified resource. Google tunnel servers are connected by an encrypted tunnel to SDC, which runs within a company's internal network.

3. The tunnel protocol allows SDC to connect to a Google tunnel server, authenticate, and encrypt the data that flows across the Internet.

4. SDC uses resource rules to validate if a user is authorized to make a request to a specified resource.

5. An optional intranet firewall can be used to provide extra network security.

6. SDC performs a network request to the specified resource or services.

7. The service validates the signed request, checks the credentials, and if the user is authorized, returns the data.

If you don't have to worry about firewalls, and have no security worries whatsoever, you can simplify things (as Daniel's answer suggests) by just using urlfetch directly (no tunnels, no validation, no encryption, no filtering, ...) -- but your worry about "the data being shared is sensitive data such as login details" suggests that this is not the case.

It's not a problem of XML vs other formats -- the problem is that sensitive data should not travel "in clear" over unprotected channels, nor be made available to all and sundry, and it's often nicer to have specialized infrastructure deal with encryption, filtering, and authorization problems, as the SDC does, rather than having to code all of this (and make it totally secure and locked-down) in your own app or specialized infrastructure middleware. For these purposes, the SDC can be very helpful, even if you only need a fraction of its functionality.