Python's robotparser ignoring sitemaps

Question

I've the following robots.txt

User-agent: *
Disallow: /images/
Sitemap: http://www.example.com/sitemap.xml

and the following robotparser

def init_robot_parser(URL):
    robot_parser = robotparser.RobotFileParser()
    robot_parser.set_url(urlparse.urljoin(URL, "robots.txt"))
    robot_parser.read()

    return robot_parser

But when I do a print robot_parser Above return robot_parser all I get is

User-agent: *
Disallow: /images/

Why is it ignoring the Sitemap line, am I missing something?

Answer 1

Sitemap is an extension to the standard, and robotparser doesn't support it. You can see in the source that it only processes "user-agent", "disallow", and "allow". For its current functionality (telling you whether a particular URL is allowed), understanding Sitemap isn't necessary.