UnauthorizedAccessException when timer job tries to access SharePoint term store

Question

I am building a timer job to import terms into the SharePoint term store form an external database. When I run the code manually (using a test web part), everything works fine. However, When the timer job executes the same code, it gets UnauthorizedAccessException when it tries to enum the term store groups. If I login locally as the account the timerjob is running under, I can execute the code no problem.

The following is the code that throws the exception:

taxonomySession = new TaxonomySession(site);
termStore = taxonomySession.TermStores[mmdServiceName];

Microsoft.SharePoint.Taxonomy.GroupCollection tsGroups = termStore.Groups; //This will return null when run from timerjob

int ptcGroupCount = (from Microsoft.SharePoint.Taxonomy.Group tGroup
                                in termStore.Groups
                                 where tGroup.Name == groupName
                                select tGroup).Count();

if (ptcGroupCount == 0)
{
...
}

Any thoughts as to why this is happening? This is a showstopper for me...