Python's robotparser ignoring sitemaps
I've the following robots.txt User-agent: * Disallow: /images/ Sitemap: http://www.example.com/sitemap.xml and the following robotparser def init_robot_parser(URL): robot_parser = robotparser.RobotFileParser() robot_parser.set_url(urlparse.urljoin(URL, "robots.txt")) robot_parser.read() return robot_parser But when...