If our server (running on a device) starts before a DHCP lease had been acquired then it can never connect using a hostname.
If that happens it can find hosts by IP address but not by DNS.
I initially thought that the Curl DNS cache was at fault as the curl connections failed. But I used CURLOPT_DNS_CACHE_TIMEOUT to prevent curl from caching address but connections still failed.
It turns out that glibc gethostbyname_r won't automatically reload it's configuration if that configuration changes. You have to manually call res_init. See bug report below.
Note: Neither the man page for gethostbyname_r nor for rer_init mentioned this limitation.
My solution is very specific. It works for our long running server but it is not my ideal solution.
I have a function that checks the mtime of the /etc/resolv.conf against the last known mtime (0 for DNE). If the two mtime differ then I call res_init. This function is called on program startup and then periodically to optionally reload the configuration.
The glibc bug report
libc caches resolv.conf forever
...
That's what res_init() is for, call it.