Since a few years, there is quite good support for using performance counters on Linux machines. Examples are OProfile (which has been included in the kernel since 2.6, I think), Perfctr, and Perfmon (not to be confused with the other Perfmon, which is a SNMP based performance monitoring tool). I think Perfmon is destined to make it to the kernel source tree as well, or so I've heard. Yet, I have been using Perfctr since I started my research, so this post is only about that tool.

There has been talk on the Perfctr mailing list (which gets hopelessly spammed these days) for including support for multi-threaded processes, but thus far I've seen nothing that does what I want. So, without further ado, here's how to patch your kernel to support multi-threaded applications.

I assume you know how to install the basic Perfctr driver, and compile your kernel to add support for it. If possible, compile it as a module, as this is easiest if you need to change things (unless you also change stuff in the kernel header files, in which case you probably want to recompile the complete kernel). Let's further assume your kernel source lives in /usr/src/linux, further referred to as toplevel. I'll also assume we're building the Perfctr module here.

The first thing that needs to be done is make sure that child processes set up their kernel data structures such that performance counter data can be stored at context switch (you want to use virtual counters, i.e., per-process counters). Therefore you need to add a function that does exactly this. We'll call it __vperfctr_set_child_perfctr(struct task_struct*, struct task_struct*). You also want to be able to set up and empty vperfctr structure, by simply allocating space for it. So, add their existance to your toplevel/include/linux/perfctr.h file (which should be there if you patched the kernel and copied the relevant files when installing Perfctr), which should then read:

In the same file you should add some code to the perfctr_copy_task(struct task_struct *, struct pt_regs *) function. Otherwise it only contains a comment stating that nothing should be done until inheritance is implemented and sets the vperfctr structure to NULL. The code for that function should become the following:

The above will get a new structure set up (through __vperfctr_get_empty) and copy the existing settings (i.e., for the control registers etc.) to that structure (through __vperfctr_set_child_perfctr). Before we can use these functions, we need to define them, which is done in toplevel/drivers/perfctr/virtual.c.

I think the Perfctr patch uses p->thread as the first argument, so change this accordingly.

Now, to get the counter values assembled in one central spot, I'm using a second module that accumulates these values. Upon exit, each thread will pass on their values to that module through a hook. The code has protection for usage on multicore processors, through a spin_lock.

Here's the code you should add to toplevel/drivers/perfctr/virtual.c to be able to access the accumulating module through the hook.

The final piece then is the read_counter module where the data is accumulated.

If you wish to read the data, you must create the device /dev/read_counters with major number 120 and minor number 0. Code that does the reading for you might look like this: