InlineFormSet - Get a User ModelForm when the relationship is in the User.Profile model

Question

My end result is to provide a User Permissions Matrix. (django.contrib.auth.models.Permission)

Working with three models in this scenario: Office, User, and UserProfile. Users are related to an Office model through the UserProfile model. See below for my model setup:

class User(models.Model):
# the included django User model

class UserProfile(models.Model):
    user = models.ForeignKey(User, unique=True)
    office = models.ForeignKey(Office)

class Office(models.Model):
    name = models.CharField(max_length=64)
    phone_number = models.CharField(max_length=11, blank=True)
    fax_number = models.CharField(max_length=11, blank=True)
    address = models.ForeignKey(Address)

I'd like to organize this by Office and allow the Office and User models to be editable (provide a form) as well.

[Office Name]
    [username1] | []perm1 | [x]perm2 | []perm3
    [username2] | []perm1 | [x]perm2 | [x]perm3

My initial thought is to use an Office ModelFormSet with a User InlineFormSet but I run in to issues because the relationship is established in UserProfile. Any suggestion on how to go about this?