Some of the reasons why you may want to provide a custom sorting function are:
*
You are using a little-endian system (such as x86) and you are using integers as your database's keys. Berkeley DB stores keys as byte strings and little-endian integers do not sort well when viewed as byte strings. There are several solutions to this problem, one being to provide a custom comparison function. See http://www.oracle.com/technology/documentation/berkeley-db/db/ref/am_misc/faq.html for more information.
You set a BTree's key comparison function using DB->set_bt_compare().
For example, an example routine that is used to sort integer keys in the database is:
int
compare_int(DB *dbp, const DBT *a, const DBT *b)
{
int ai, bi;
/*
* Returns:
* < 0 if a < b
* = 0 if a = b
* > 0 if a > b
*/
memcpy(&ai, a->data, sizeof(int));
memcpy(&bi, b->data, sizeof(int));
return (ai - bi);
}