Which credentials should I put in for Google App Engine BulkLoader at development server?

Question

Hello everyone, I would like to ask which kind of credentials do I need to put on for importing data using the Google App Engine BulkLoader class

appcfg.py upload_data --config_file=models.py --filename=listcountries.csv --kind=CMSCountry --url=http://localhost:8178/remote_api vit/

And then it asks me for credentials:

Please enter login credentials for localhost

Here is an extraction of the content of the models.py, I use this listcountries.csv file

class CMSCountry(db.Model):
  sortorder = db.StringProperty()
  name = db.StringProperty(required=True)
  formalname = db.StringProperty()
  type = db.StringProperty()
  subtype = db.StringProperty()
  sovereignt = db.StringProperty()
  capital = db.StringProperty()
  currencycode = db.StringProperty()
  currencyname = db.StringProperty()
  telephonecode = db.StringProperty()
  lettercode = db.StringProperty()
  lettercode2 = db.StringProperty()
  number = db.StringProperty()
  countrycode = db.StringProperty()

class CMSCountryLoader(bulkloader.Loader):
  def __init__(self):
    bulkloader.Loader.__init__(self, 'CMSCountry',
                           [('sortorder', str),
                            ('name', str),
                            ('formalname', str),
                            ('type', str),
                            ('subtype', str),
                            ('sovereignt', str),
                            ('capital', str),
                            ('currencycode', str),
                            ('currencyname', str),
                            ('telephonecode', str),
                            ('lettercode', str),
                            ('lettercode2', str),
                            ('number', str),
                            ('countrycode', str)
                            ])
loaders = [CMSCountryLoader]

Every tries to enter the email and password result in "Authentication Failed", so I could not import the data to the development server.

I don't think that I have any problem with my files neither my models because I have successfully uploaded the data to the appspot.com application.
So what should I put in for localhost credentials?
I also tried to use Eclipse with Pydev but I still got the same message :(
Here is the output:

Uploading data records.
[INFO    ] Logging to bulkloader-log-20090820.121659
[INFO    ] Opening database: bulkloader-progress-20090820.121659.sql3
[INFO    ] [Thread-1] WorkerThread: started
[INFO    ] [Thread-2] WorkerThread: started
[INFO    ] [Thread-3] WorkerThread: started
[INFO    ] [Thread-4] WorkerThread: started
[INFO    ] [Thread-5] WorkerThread: started
[INFO    ] [Thread-6] WorkerThread: started
[INFO    ] [Thread-7] WorkerThread: started
[INFO    ] [Thread-8] WorkerThread: started
[INFO    ] [Thread-9] WorkerThread: started
[INFO    ] [Thread-10] WorkerThread: started
Password for [email protected]: [DEBUG   ] Configuring remote_api. url_path = /remote_api, servername = localhost:8178

[DEBUG   ] Bulkloader using app_id: abc
[INFO    ] Connecting to /remote_api
[ERROR   ] Exception during authentication
Traceback (most recent call last):
  File "D:\Projects\GoogleAppEngine\google_appengine\google\appengine\tools\bulkloader.py", line 2802, in Run
request_manager.Authenticate()
  File "D:\Projects\GoogleAppEngine\google_appengine\google\appengine\tools\bulkloader.py", line 1126, in Authenticate
remote_api_stub.MaybeInvokeAuthentication()
  File "D:\Projects\GoogleAppEngine\google_appengine\google\appengine\ext\remote_api\remote_api_stub.py", line 488, in MaybeInvokeAuthentication
datastore_stub._server.Send(datastore_stub._path, payload=None)
  File "D:\Projects\GoogleAppEngine\google_appengine\google\appengine\tools\appengine_rpc.py", line 344, in Send
f = self.opener.open(req)
  File "C:\Python25\lib\urllib2.py", line 381, in open
response = self._open(req, data)
  File "C:\Python25\lib\urllib2.py", line 399, in _open
'_open', req)
  File "C:\Python25\lib\urllib2.py", line 360, in _call_chain
result = func(*args)
  File "C:\Python25\lib\urllib2.py", line 1107, in http_open
return self.do_open(httplib.HTTPConnection, req)
  File "C:\Python25\lib\urllib2.py", line 1082, in do_open
raise URLError(err)
URLError: <urlopen error (10061, 'Connection refused')>
[INFO    ] Authentication Failed

Thank you!

Answer 1

I recommend you follow the advice given here, and I quote:

add this to app.yaml file:

-- url: /loadusers
 script: myloader.py
 login: admin

Note that if you run it on local development machine, comment off the last line login:admin so that you don't need a credential to run the bulkloader.

(my emphasis).

Answer 2

Next worked for me:

• removing the line

login:admin

from app.yaml

• Updating app cofig:

appcfg.py update [app-id]

• (Re)starting local server:

appserver [app-id]

• Upload data:

appcfg.py upload_data --config_file=album_loader.py --filename album_data.csv --kind Album --url=http://localhost:8080/remote%5Fapi ../[app-id]

Ask for mail and password type anything

Answer 3

Edit: Look at my new solution

This problem is still present. I have opened a ticket to ask if the authentication could be bypassed on the local dev server. Please vote for this issue so that we can have it resolved [quickly].

I have been able to upload data to the dev server by:

• leaving the "login:admin" line in app.yaml
• adding "[email protected]" to your command
• pressing Enter when prompted for a password (nothing required)

Leaving the "login:admin" line is a good thing, as you will not upload your app on the production servers without this line, which could expose you to someone adding data to your datastore...

Blockquote

Answer 4

Yes, comment out the admin requirement for the remote_api:

[app.yaml]

- url: /remote_api
  script: $PYTHON_LIB/google/appengine/ext/remote_api/handler.py
 # login: admin

Then run this command:

$ bulkloader.py --dump --kind=DbMyKind --url=http://localhost:8080/remote%5Fapi --filename=export.csv --app_id=my_appid --auth_domain=localhost:8080

Note: verify that --auth_domain is passed, and proper port is passed for localhost.

Answer 5

You need create an admin credential in your local development server first.

With your firefox (or chrome safari etc), open http://localhost:8178/remote_api, you will be asked to login (without password), enter an email as your login, and tick the login as administrater box, login. This will create you a local admin credential for you, use this when bulkloading locally.

It applies to other admin required local access.

Leaving (or commenting) out login:admin is a bad practice, since you might deploy that into production, too. Take care!

Answer 6

EUREKA: I found the way to use the bulkloader.py tool without having to manually enter login credentials.

Here are the 2 steps:

---

Set your app.yaml file up. Example:

- url: /remote_api
  script: $PYTHON_LIB/google/appengine/ext/remote_api/handler.py
  login: admin

You should put it BEFORE your - url: .* line in app.yaml, otherwise you will never access the /remote_api url.

Note that I've left the login: admin part, as removing it is a VERY BAD practice, since you might deploy that into production...

---

2 Launch this command (adapt it to your needs).

echo 'XX' | python2.5 ../google_appengine/bulkloader.py --dump --kind=NAMEOFMODEL --url=http://localhost:8080/remote_api --filename=FILENAME --app_id=APPID [email protected] --passin .

The trick is to use the combination of those 2 parameters:

• --email= (you can put whichever email address you want, I use [email protected])
• --passin

Specifying --email= will suppress the "enter credentials" prompt, and --passin will allow to read the password from stdin (that's where the echo 'XX' | comes into play!)

---

Enjoy!

P.S.: Don't forget to vote so that Google can provide an easier to use way to do that: Issue 2440.

Answer 7

Hi, Another solution is to stub out the auth function with lambda: http://www.carlosble.com/?p=603