tracing a linux kernel, function-by function (biggest only) with us timer

Question

Hello

I want to know, how does the linux kernel do some stuff (receiving a tcp packet). In what order main tcp functions are called. I want to see both interrupt handler (top half), bottom half and even work done by kernel after user calls "read()".

How can I get a function trace from kernel with some linear time scale?

I want to get a trace from single packet, not the profile of kernel when receiving 1000th of packets.

Kernel is 2.6.18 or 2.6.23 (supported in my debian). I can add some patches to it.

Answer 1

I can't immediately see a way to trace only a single packet at a time. In particular, it may be difficult to get such fine grained tracing since the upper-half of an interrupt handler shouldn't be doing anything blocking (this is dead-lock prone).

Maybe this is overly pedantic, but have you taken a look at the source code? I know from experience the TCP layer is very well commented, both in recording intent and referencing the RFCs.

I can highly recommend TCP/IP Illustrated, esp Volume 2, the Implementation (website). Its very good, and walks through the code line by line. From the description, "Combining 500 illustrations with 15,000 lines of real, working code...". The source code included is from the BSD kernel, but the stacks are extremely similar, and comparing the two is often instructive.

Answer 2

You want oprofile. It can give you timings for (a selected subset of) your entire system, which means you can trace network activity from the device to the application and back again, through the kernel and all the libraries.