Why is my hgweb server so slow?

Question

I am serving up access to many mercurial repositories using hgweb, providing them as a collection:

[collections]
/home/me = /home/me/projects

This serves them up at localhost/projects

I have around 30 repositories at that location, in a source tree with a fair number of other, non-mercurial-managed projects.

hgweb is really slow to respond; it takes about 30 seconds to provide a listing at http://localhost/, and about 30 seconds to open a project, making it painful to use this for sharing purposes.

How can I tune this to make it faster?

I'm running on OSX, if it makes a difference.

Answer 1

AFAIK, hgweb will scan all subdirectories of the [collections] entry in its configuration file. Since you've got a lot of non-Mercurial directories in there, it has to do a scan of each subdirectory of each of them. In contrast, it can stop scanning at the top level of a directory tree containing a Mercurial repository because it will see the .hg directory there.

If you're using a newer Mercurial (after 1.1, it looks like), try changing the hgweb.config to use a [paths] section instead, and provide explicit entries for each of the Mercurial repositories.

Answer 2

The problem is probably the server searching recursively for repos during every request. Sounds like you've got a pretty big directory there so this makes sense.

This notation will work with the preferred [paths] attribute but I am not sure about if it will help the [collections] attribute. Try changing to

[collections]
/home/me = /home/me/projects/*

so it will only search one level down.

Check here for more on the issue: http://mercurial.selenic.com/wiki/HgWebDirStepByStep

If that doesn't work it definitely will if you change to [paths] and use the * notation.

Answer 3

Following up on Niall's very helpful answer, above, I realized that I needed a tool to maintain this [paths] section. I ended up going with this (which uses configobj by M. Foord.

#!/usr/bin/env python
from __future__ import print_function

import os

from configobj import ConfigObj

hgweb_path = os.path.join(os.path.expanduser("~"), "Library", "Preferences", "hgweb.config")
projects = os.path.join(os.path.expanduser("~"), "projects")

config = ConfigObj(hgweb_path)

paths = []

def add_mercurial(arg, dirname, names):
  if '.hg' in names:
    paths.append(dirname[len(projects) + 1:])

os.path.walk(projects, add_mercurial, None)

config['paths'] = {}

for path in paths:
  config['paths']["projects/" + path] = os.path.join(projects, path)

config.write()

This script is run by OS X's equivalent of cron every 15 minutes and ensures that my hgweb never gets out of date.

Answer 4

As an alternative, you can use hg-app http://hg.python-works.com it's hgwebdir replacement written entirely in pylons.

• has it's own middleware to handle mercurial protocol request each request can be logged and authenticated + threaded performance unlikely to hgweb
• full permissions per project read/write/admin access even on mercurial request
• mako templates let's you cusmotize look and feel of application.
• diffs annotations and source code all colored by pygments.
• mercurial branch graph and yui-flot powered graphs
• admin interface for performing user/permission managments as well as repository managment.
• Additional settings for mercurial web, (hooks editable from admin panel !) also manage paths, archive, remote messages
• backup scripts can do backup of whole app and send it over scp to desired location
• setup project descriptions and info inside built in db for easy, non file-system operations
• Full search text on repository codes
• added cache with invalidation on push/repo managment for high performance and always upto date data.
• rss / atom feeds, gravatar support
• based on pylons 1.0 / sqlalchemy 0.6