Edit: Due to the extremely subjective nature of comparing programming languages, in particular the open hostility many in the C and C++ camps have towards each other, I would like to make it absolutely clear that I am attempting to explain a point of view rather than champion one language over the other.
Fast servers try to minimize the amount of copying done. This means that any strings you parse out of an IO buffer are represented using pointers. Using std::string would mandate copying the string, which is slow. You could create a new string class which doesn't copy its contents, but this string class would be no better than a C structure. Edit: Sometimes, modifying code to use C++ structures will require either additional copying, additional indirection, or writing the code in the same manner as the C code. For example, std::list<T> will copy any T that you add to it, while std::list<T*> requires an additional object for each member and has an additional layer of indirection. The least expensive option might be to put next and prev fields inside the T structure itself. I think Boost has a library for handling this case, but linked lists are second nature to experienced C programmers so there's not a huge benefit to the Boost option and some programmers stylistically prefer handling linked lists themselves. Some of the people commenting below suggested that passing a std::string& or my_string& is as cheap as passing around pointers to a buffer — this is not entirely true, since in order to access the buffer one now has to dereference two pointers, where the C code would only have to dereference one. The differences vanish if my_string is passed by value, as long as it doesn't do anything too fancy in its copy constructor... however, such an object would no longer be any safer than the equivalent C construct (since in this case, my_string can now outlast the buffer to which it points).
Fast servers try to minimize the amount of allocation done. So while a programmers are writing fast servers, it helps if none of the allocations and deallocations are hidden from view. This means that all of those helpful ways to make allocations unobtrusive in C++ get in the way. Edit: A smart C++ programmer will be very aware of where the code performs allocations. Nonetheless, a C programmer doesn't have to think quite so hard about it.
Fast servers need to work with syscalls very closely, and keep data in the format that the syscalls expect whenever possible. Syscalls have a C interface. Edit: It's very easy to call them from C++, too, but wrapping all of the objects in C++ classes is a lot of work for not much benefit.
Exception handling causes code size to increase. This decreases the locality of the program code and slows it down, even if no exceptions are being thrown. Edit: I am aware that this is a negligible (but measurable) difference in speed. I've heard the difference in executable size is something like 5-15%, which matches the results I got using quick tests with and without -fno-exceptions using g++. I'd say it's more relevant that some of these servers use a somewhat unusual error handling technique: if code is handling a request and there's an error, you can just jump out to an outermost error handling routine and let it handle cleanup. If you're using memory pools, such as Apache's apr_pool_t, then there's no need to call destructors as you unwind the stack and therefore the advantages of C++ exceptions over longjmp are somewhat few.
Template instantiations cause code size to skyrocket. This also decreases the locality of the program code. Edit: The instantiations are exactly what give C++ templates their speed advantage over near-equivalent C code. For example, if you want to sort an array of X, you can get a fully inlined, rock-solid sorting function in C++ using templates in a single line of code. In C, you would have to write it yourself. But these servers do not use very many different kinds of data structures, often they only need some simple queues and an associative map from strings to void*.
It is relatively easy to estimate how long it will take a particular piece of C code to run, it is more difficult to estimate the run time for C++ code. There are a lot of extra things that can happen in C++ code: temporary objects can be created to pass between functions, operators can be overloaded to perform complex tasks, and exiting a scope can cause expensive destructors to run. I've seen C++ programs hang in poorly written destructors. On the other hand, C maps very cleanly to machine code, with relatively few surprises.
(Edit: new) There's also the issue of the performance compiling a given piece of code using a C compiler versus compiling it with a C++ compiler. Due to the stricter aliasing rules of C++, the performance benefit often goes to the code compiled with the C++ compiler. However, C99 has introduced the new restrict keyword which is not yet available in C++. It allows the user to declare that there are no active aliases to a given region of memory. This allows the compiler to perform some types of optimizations previously only available to Fortran programmers (as Fortran has stricter aliasing rules than C++).
So, if you were trying to write a very high performance server in C++, you'd probably not use the STL, exceptions, templates (or sparingly), sophisticated constructors or destructors, or operator overloading. You'd also probably not wrap the C structures you use. What are you left with? You're left with 90% C and 10% C++. At this point, you say, "why bother?" and write the whole thing in C. There are added benefits to using C:
C compilers are available in places where C++ compilers are not. On some platforms, C++ compilers are available but not as mature as the C compiler. Conditions were worse in 1995, which was when Apache was written.
If you write your app in C, people can write glue to call into their favorite dynamic language (Python, Ruby, Perl) without much trouble. It is easier to interface with those languages if your application is written in C.
I think the biggest part of it all is the syscalls. When designing a fast server, you basically plan how the bytes travel from one syscall to another syscall as quickly as possible. I would guess that high performance non-server apps that do more number crunching are more likely to be written in C++, for example, many fast AI libraries seem to be written in C++.
Edit: There's also a sampling error here. The projects listed are open-source projects only, and certain parts of the open-source community, especially the systems programming part, use C. Part of this is cultural, as perhaps the most visible open-source project is the Linux kernel, which is written in C. Linus Torvalds in particular is noted for his rants against C++, you can also look at the Linux kernel mailing list FAQ with a section on why Linux does not support C++ in the kernel. The Linux kernel has some even better reasons not to use C++, especially that exception handling in kernel code is generally regarded as a very bad idea (Apple's IOKit, which requires you to write drivers in C++, disables exception handling). I would say that C is much less common outside of the open-source community.
As for Python modules, most of the ones written in C are just interfaces to some other library (and a large portion of these modules are written in Python itself, natch). Most of the C code is spent converting between the formats that the library expects and formats that Python can understand. C++ doesn't deliver any real benefits for this kind of glue code. Fortunately, the glue code is very small (I wouldn't want to write very much of it).
As for a recommendation about which to learn: The nicest thing about programs like nginx, varnish, apache, git, subversion, python, etc. is that they're already written and already ready for a production environment. I would definitely not recommend either C or C++ over the other unless I knew what kind of applications you wanted to write, as I'm guessing you're not going to write the next web server (we already have a good selection).