It seem what i can do fast crawler with python in two ways:
thread pool with block sockets
non block sockets select,asyncore,etc..
i thnk where is no real need in thread here, and solution #2 better.
which is better and why?
views:
42answers:
1
+1
A:
Twisted is usually preferred to asyncore. It is an asynchronous I/O framework that can also work with thread pools.
In Python, you should prefer asynchronous IO to threads, simply because threads are a second class citizen in its canonical implementation (CPython) due to GIL.
Alex B
2010-09-09 01:13:32
I am intrested in implementation on low level at this moment, how twisted do it internaly with asyncore?
Evg
2010-09-09 01:16:29
@Evg Twisted and asyncore are two different implementations, they are not related.
Alex B
2010-09-09 01:25:07
i understand) i mean what twisted internally can do it only by threads or asyncore at low level, i am right? And my question what better threads with blocking sockets or asyncore.
Evg
2010-09-09 01:33:05
or as you say "Twisted and asyncore are two different implementations" how twisted do it internaly? i mean python interface to non blocking sockets at low level used by twisted.
Evg
2010-09-09 01:40:46
@Evg seems from the source it uses `select` module, which uses platform-specific polling mechanism (epoll, kqueue, etc).
Alex B
2010-09-09 02:22:45
thanx for you help, i edit question according to you, what do you think about prefered solution #1 or #2?
Evg
2010-09-09 02:47:18
@Evg I've updated the answer.
Alex B
2010-09-09 11:30:33