Hi
i need to create multiple TCP connections simultaneously to some custom TCP-server application for its performance testing. I know a lot of such for Web (i.e. curl-loader based on libcurl), but I didn't found some general one.
Scenario for client is the simplest: create connection, send special data, read the answer and close connection. In every step there's timestamp. All timestamps should be written to file for the further calculations. I need about 10 000 such connections in parallel.
I'd prefer some ready solution but there's nothing found in Google, so I'm ready to write this one with Python. If so, can you recommend me a suitable python modules that could produce this amount of connections? (multiprocessing, twisted..?)
views:
140answers:
2
+1
A:
Handling 10k connections is a tough problem (known as C10K problem). If you need real numbers, stick with C++(Boost/POCO libraries or OS native API), or distribute clients across 10 load-generating clients.
No way should you try this with Python (handle 10'000 connection on 1 CPU core - not realistic).
BarsMonster
2010-09-03 13:27:05
`socket` is a [C extension](http://svn.python.org/view/python/tags/r31/Modules/socketmodule.c?view=markup) in Python (at least in Python 3.1.2); there shouldn't be too much of an overhead in terms of CPU usage, unless you spawn a new thread per connection as Python's threading library is not implemented natively.Or is C++ that much more efficient?
Beau Martínez
2010-09-03 13:33:20
C++ is MUCH more efficient, and could you all CPU cores. Python due to GIL work only on 1 core at any moment. With such bunch of connections tiny overheads here and there take most of the time.
BarsMonster
2010-09-03 13:39:49
Python multi-threading is not the option, sure. But maybe multiprocessing will help..?
DominiCane
2010-09-03 13:44:20
Yes, it's better. But you are still 10-100 times slower than C++. I would go with Python on 100 connections, but not 10'000.
BarsMonster
2010-09-03 13:48:19
A:
My two cents:
- Go for Twisted, or any other asynchronous networking library.
Make sure you can open enough file descriptors on the client and on the server. On my Linux box, for instance, I can have no more than 1024 file file descriptors by default:
carlos@marcelino:~$ ulimit -a core file size (blocks, -c) 0 data seg size (kbytes, -d) unlimited [..] open files (-n) 1024 [..]It may pay to run the client and the server on different machines.
Carlos Valiente
2010-09-03 18:23:23
How can I make a lot of connections with twisted? Is there some working example/reference?
DominiCane
2010-09-06 04:22:59