AFAIK you cannot filter the graph - the TortoiseSVN documentation says:
In order to generate the graph, TortoiseSVN must fetch all log messages from the repository root. Needless to say this can take several minutes even with a repository of a few thousand revisions, depending on server speed, network bandwidth, etc. If you try this with something like the Apache project which currently has over 500,000 revisions you could be waiting for some time.
The good news is that if you are using log caching, you only have to suffer this delay once. After that, log data is held locally. Log caching is enabled in TortoiseSVN's settings.
So - make sure that log caching is enabled and future graph views will be faster.