I'm building a socket application that need to shuffle a lot of small/medium sized files, something like 5-100kb sized files to a lot of different clients (sort of like a web server, but still not quite).
Should I just go with the standard poll/epoll (linux) or async sockets in winsock (win32), or are there any methods with even more performance around (overlapped i/o on win32 for example) ?
Both Linux and Windows are possible platforms!
On Linux, demultiplexing multiple sockets using epoll is the fastest possible way to do parallel I/O over TCP.
But I'll also mention that in the interest of portability, (and since you seem to be interested in either Linux or Windows), you should look into Boost.Asio. It has a portable API, but uses epoll on Linux and overlapped I/O on windows, so you can built highly efficient and portable networking apps.
Also since you're working with files, you should also implement double buffering when performing I/O for maximum performance. In other words, you send / recv each file using two buffers. For example, on the sending side, you read from disk into one buffer and then send that buffer over the network, while another thread reads the next block of data from disk into the second buffer. This way you overlap disk I/O with network I/O.
On windows you may try using TransmitFile, which has a potential of boosting your performance by avoiding kernel space <-> user space data copying.
On Linux, sendfile() is a high performance API specifically for sending data from files to sockets (you will still need to use poll to multiplex, it is just a replacement for the read/write part).
In addition to epoll it looks like Linux sendfile(2) would be a good fit for your needs on the server side.
Unfortunately, if you want maximum possible performance, you will still have to hand-craft your I/O code on Windows and Linux as currently available abstraction libraries don't scale that well to multiple threads (if at all).
Boost asio is probably the best option if you want portability (and ease of use), but it does have it's limitations when it comes to multithreaded scalability (see C++ Socket Server - Unable to saturate CPU) - I guess the main problem is to integrate timeout handling without excessive locking into a multithreaded event loop.
Essentially, what you would want to use for maximum performance is I/O completion ports with a pool of worker threads on Windows and edge-triggered epoll with a pool of worker threads on Linux.
Don't optimise your program prematurely.
Assuming it isn't a premature optimisation, the easiest thing to do is just keep all the data in memory. You can mmap() them if you like, or just load them in at startup time. Sending stuff that's already in memory is a no-brainer.
Having said that, trying to multiplex lots of things with (e.g.) epoll can be a bit of a headache, can you not use something someone's already written?