high speed tracing

Question

I have an embedded board with a 32 Micro-controller, and an custom built OS,

• Unfortunately, as of now, connection to the PC is only through serial port,
• Internal memory is limited to 512KB.
• There are at-least 10 tasks in the system

Question,

• I want to capture the sequence in which task switch happens,
• And when I try to write into the RAM, it overflows ~~
• And when I try to send it though serial port, the system behavior changes (as serial port is slow)

There is no persistent storage like NAND FLASH, or something.

• Can you guys think of some Idea?

If there is no way with Serial port,

• Can you guys recommend some other interface or than Serial port.

Answer 1

First calculate how many task switches you will have and compare with speed of your serial. Maybe it is just not possible to push this amount of data? Consider than

• USB - up to 480Mb/s
• RS485 - should give you from 100Kb/s to 35Mb/s
• I2C - gives around 100Kb/s

If serial is enough, than maybe go with cache? First write to memory and is separate task get data from memory in blocks send to serial. Look for things like circular buffering to avoid locking.

Answer 2

You probability want to determine why your RAM overflows when logging, you don't need much logging if you log only what you need to see. You can log into a circular buffer to prevent overflow. With Ram logging you probably can run at close to true speed. Logging to a communications link added latency, interrupts and task switches to the system.

Don't log everything from the start. Log only enough to understand when your problem occurs. Once you know when your problem occur log more detail as soon as the problem section is entered.

If you really want to solve the problem in no time get a Green Hills Trace pod. Your hardware must be designed to allow the Pod to be connected and it is terribly expensive. However the results is incredible ...

Answer 3

I don't know what platform you are using but...

The ARM:s have a block called ETM, that solves your problem. And with a hardcore debugger from Lauterbach you can use that block.

The downside is that the cost is high, approximately the same as a small new car :)

And I don't know if your hardware has a ETM block...

Answer 4

Do you need real-time behavior when running the traces? I mean, is it an option to log into RAM until the buffer is (nearly) full, then enter a critical section preventing the application from task-switching and servicing interrupt and dumping the buffer over the serial line. This will block the operation of the application for a while, but depending on the test you're running it may be acceptable. Any real-time tracing over serial, USB,... will influence the behavior of the application so it's not sure that what you are measuring is relevant.

Another thing you can do (if you don't already) is make the logging as small as possible, for example: 1 byte per task switch with the most significant nibble the task ID of the old task and the least significant nibble the task ID of the new task. This way you should be able to cover a lot of task switches in your 512KB of memory.

Answer 5

If you can use an output port on the microcontrollers without disturbing other hardware too much you can output the current task number and capture it with a logic analyzer.

Answer 6

Somewhat more light-weight than the ARM ETM highlighted by Johan, the MIPI System Trace Protocol has been designed for just this sort of trace activity. It is designed for instrumentation trace, and typical implementations offer around 500 Mbit/s of trace bandwidth over a four-bit port.

However, it is unlikely your board has support for it. :-( Also, you need a trace receiver, which again can cost the price of a small car (Lauterbach have one).

Answer 7

• And when I try to write into the RAM, it overflows ~~

What are you logging and how large a buffer do you allow? Without knowing how you implemented this it is hard to advise, but there is probably much you could do to optimise in-memory logging.

If at each context switch you log task ID and time-stamp (say 3 bytes per event) you should get 341 context switches per Kbyte. In many systems that would be a significant period, and remember that is for just 1K of buffer. If you are logging interrupts as well that may be more expensive, as would logging all system calls rather than just context switches. Perhaps you could implement a filter in the logging, so it only logged tasks or events of interest. You might also implement event triggers so that the logged data is automatically dumped to your serial port when such an event occurs (and when the event of interest has occurred, so the act of transmission is not intrusive to your investigation). You should also implement the buffer as a circular buffer so that rather than overflowing, the oldest data is simply discarded to make room for the new, so that on occurrence of a trigger event, you have all the event information leading up to it.

Answer 8

Do you have access to any gpio's or test points? Depending on how many tasks you are really actually switching between, you maybe able to set a gpio for each task switch, and observe with an oscilloscope or logic analyzer. It will be enough to understand the basic switching and performance of the problem tasks, and it will be cheaper than a debugger.(at least parts cost, and labor if you have access and know how to program the gpio's) This may be enough info to troubleshoot the problem.

Answer 9

when I try to send it though serial port, the system behavior changes (as serial port is slow)

That makes it sound as though you're doing blocking writes to the serial port (my apologies if my guess is wrong).

The serial port may be slow, but if you use interrupt-based TX, it should have a minor impact on your system. That is, write your data into a circular buffer, and then have a serial TX interrupt routine to grab the bytes from the buffer and transmit them in the background.

At 57,600 bps, 8-N-1, you could transmit up to 5,760 bytes per second. If your task switcher generates a 2-byte timestamp plus a 1-byte task ID, then you can trace up to 1,900 task switches per second. But you may want to frame it, e.g. using COBS, which would mean 5 bytes per record, so you can trace up to 1,100 task switches per second.