Is it somehow possible to "detect" that the reactor is overloaded and start dropping connections, or refuse new connections? How can we avoid the reactor being completely overloaded and not being able to catch up?
+1
A:
If I understand Twisted Reactors correctly, they don't parallelize everything. Whatever operations have been queued is scheduled and is done one by one.
One way out for you is to have a custom addCallback which checks for how many callbacks have been registered already and drop if necessary.
jeffjose
2010-08-08 06:33:55
+1
A:
No easy way, but here's some suggestions: http://www.mail-archive.com/[email protected]/msg00389.html
Matt Williamson
2010-08-08 06:36:38
+1
A:
I would approach this per protocol. Throttle when the actual service requires it, not when you think it will. Rather than worrying about how many callbacks are waiting for a reactor tick, I'd worry about how long the HTTP requests (for example) are taking to complete. The number of operations waiting for the reactor could be an implementation detail - for example, if one access pattern ended up with callbacks on long DeferredLists, and another had a more linear chain of callbacks, the time to respond might not be different even though the number of callbacks would be.
This could be done by keeping metrics of the time to complete a logical operation (such as servicing a HTTP request). An advantage of this is that it gives you important information before a problem happens.
Karl Anderson
2010-08-10 00:02:55
Yes, one way I've done in the past was to check response time (for e.g http request) and drop new connections when this increases too much. But this is not perfect, as it's possible that by the time we detect it, already thousands of deferreds have piled up, which all have to be processed, thus slowing the reactor even more.
Tommy
2010-08-10 07:01:37
Sure, if you want to degrade service gracefully, be smart about it and do it before your server is too thrashed to respond. My opinion is that the number of waiting callbacks is not what you want to be monitoring, I'd monitor other resources (SQL query time, memory, load, disk?) first, but I'd still prioritize request time. Twisted should make it easy to efficiently deny when your requests are taking too long - just keep track of the number of outstanding requests and their start times. Or, do it in a proxy so you don't have to worry about an unresponsive server at all.
Karl Anderson
2010-08-11 23:32:13