ActionFilter is probably a good starting point, but depending on your architecture, you may want to consider whether perimeter defense is good enough.
If you are essentially building a single-layer ASP.NET MVC application (and there may be perfectly reasonable reasons to do this), an ActionFilter will provide defense that is good enough while at the same time being very simply to apply.
On the other hand, if your application is a multi-layer application, Defense in Depth is more appropriate. In that case, you should consider applying the authorization logic in the Domain Model, or perhaps even in the Data Access layer. This will ensure that if you ever develop another application based on the same Domain Model (e.g. a web service), the authorization logic would still apply.
No matter what you do, I strongly recommend that you base the actual authorization implementation on IPrincipal.
On a more specific note, what you are asking about here is best modeled with ACL-based authorization: Set an ACL on each user profile that by default grants access to only the user him/herself and the administrator. If/when you later need to expand the application to allow delegated access to other users' profiles (I don't know whether that's even remotely realistic in your specific case), you can simply do that by adding a new entry to the ACL.
In such a case, evaluating access involves retrieving the ACL for the requested resource and checking whether the current user (IPrincipal) is included in that ACL. Such an operation is very likely to involve out-of-process operations (looking up the ACL in a database), so having it as an implicit part of an application by hiding it behind an ActionFilter sounds like it could potentially hide some performance issues. In such a case, I would consider making the authorization model a bit more explict/visible.