There are so many variables that could be involved that I find this question too general to answer. Some things you might want to check out are:
But since security is a pretty complex topic and important to get right (it is only as strong as the weakest link) and WCF has so many different options, configuration properties etc... it would be smart to get some books and really get aquinted with the intracies.
I can personally recommend Programming WCF 2nd edition and Learning WCF: A hands-on guide
After reading your previous question I also suggest you give tracing a try. WCF logs quite a bit about whats going on internally and when certain limits are hit. See http://msdn.microsoft.com/en-us/library/ms733025.aspx for more information about how to configure it. Pay special attention to the part about how XmlWriterTraceListener isn't thread-safe and might block execution flow and this severly impact performance (not something you want when debugging performance) and how to overcome this.
Lastly I think John Saunders his suggestion is very valid and if you do start with a trivial example and do not encounter this problem then I suggest you re-add your current functionality in small increments (or in a binary sort type of fashion) to see what it is that is causing the issue.