Drupal will offer you more flexibility and a better administrative experience than Joomla. WordPress is the least flexible, but the most elegant administrative interface. If you see this site growing in ways that are difficult to foresee, I would definitely go with either Drupal or a custom CMS built on a framework like CakePHP (or Django, CodeIgniter, etc). I've seen people invest a lot in a Joomla site only to jump ship because it couldn't grow with their needs, and it is relatively awkward to use.
One advantage of using an existing CMS is that it may be easier to replace your developer if necessary, since a lot of people are familiar with, or could easily become familiar with WordPress, Drupal, or Joomla. If your developer is working from scratch, you may end up with a poorly documented codebase that would be difficult for someone else to jump into.
I don't totally agree with Nik's comment regarding security. If you (or the developer) stay on top of upgrades for the CMS you choose (and the installed modules), that is probably the most secure option, since security experts are regularly testing and fixing holes in these systems. The danger is in using something like Drupal (or Joomla or WordPress) but failing to install security updates -- then you have a system with known and published vulnerabilities.