If you put your roles in a SQL database, lookups will perform substantially as you describe. I can help you with the database structure, if you're interested.
You need to reverse your pointers.
"Harry" is a member of "Site2 Admins" which has "Administrators" access to "Site2", so he can thus "Delete," "Write" and "Read that content.
Why "Administration" should be a common thing between "Harry" and "Joe" I'm not clear. Harry is an administrator on one site, but just a user on another, and Joe vice versa.
Have you measured this and determined that this traversal is a performance bottleneck?
I've never seen a system with so many roles / levels that the cost of traversing this kind of structure would become an issue. And if the tree really is that large, I'd be more concerned that administrators would have difficulty in understanding who is authorized to do what.
Regarding scalability, I would typically use the ASP.NET cache to cache the complete tree that maps between resources and roles, with a suitable cache timeout. And separately cache the mapping from Users to Roles (e.g. in Session or with a user-specific key in the ASP.NET cache).
Accessing the information from the cache will typically be blindingly fast compared with going to the database each time.