User Authentication And Text Parsing in Python

Question

Well I am working on a multistage program... I am having trouble getting the first stage done.. What I want to do is log on to Twitter.com, and then read all the direct messages on the user's page.

Eventually I am going to be reading all the direct messages looking for certain thing, but that shouldn't be hard.

This is my code so far

import urllib
import urllib2
import httplib
import sys

userName = "notmyusername"
password  = "notmypassword"
URL = "http://twitter.com/#inbox"

password_mgr = urllib2.HTTPPasswordMgrWithDefaultRealm()
password_mgr.add_password(None, "http://twitter.com/", userName, password)
handler = urllib2.HTTPBasicAuthHandler(password_mgr)
pageshit = urllib2.urlopen(URL, "80").readlines()
print pageshit

So a little insight and and help on what I am doing wrong would be quite helpful.

Answer 1

Twitter does not use HTTP Basic Authentication to authenticate its users. It would be better, in this case, to use the Twitter API.

A tutorial for using Python with the Twitter API is here: http://www.webmonkey.com/tutorial/Get_Started_With_the_Twitter_API

Answer 2

The regular web interface of Twitter does not use basic authentication, so requesting pages from the web interface using this method won't work.

According to the Twitter API docs, you can retrieve private messages by fetching this URL:

http://twitter.com/direct_messages.format

Format can be xml, json, rss or atom. This URL does accept basic authentication.

Also, your code does not use the handler object that it builds at all.

Here is a working example that corrects both problems. It fetches private messages in json format:

import urllib2

username = "USERNAME"
password  = "PASSWORD"
URL = "http://twitter.com/direct_messages.json"

password_mgr = urllib2.HTTPPasswordMgrWithDefaultRealm()
password_mgr.add_password(None, "http://twitter.com/", username, password)
handler = urllib2.HTTPBasicAuthHandler(password_mgr)
opener = urllib2.build_opener(handler)
try:
  file_obj = opener.open(URL)
  messages = file_obj.read()
  print messages
except IOError, e:
  print "Error: ", e