Mercurial The Definitive Guide, also known as the hg book, contains a section on writing extensions for Mercurial. The book is available to view for free at http://hgbook.red-bean.com/.
Edit: My apologies, the hg book did only describe using extensions not writing them. The section on writing hooks in the book may still be useful though.
The best way to learn how to write an extension is probably going to be reading extension code. Focus the most attention on extensions that perform functions similar to what you want to implement.
e.g. If your interested in converting from one SCM system to another take a look at the hg-git extension.