Klogger is logging framework, and as such, needs some efficient way to save runtime events onto persistent storage. This section will try to overview the basic mechanism used by Klogger. (defining events and logging them is described in Section 3).
Klogger employs per CPU memory resident buffers, to which logging actually takes place. The logging code first checks if Klogger is enabled. If so, space is allocated on the relevant CPU's buffer, and the event's payload is saved there in the form of a C structure. The buffer is configured to not only be memory resident, but also uses Intel's write-combining cache policy [5], allowing efficient sequential writes and preventing the data from polluting the caches.
Each CPU is also attached with a special thread in charge of dumping the memory buffer to disk whenever the latter fills. These threads run at the highest real-time priority in the system, under the SCHED_FIFO scheduling policy [4,1]. Klogger does not actually wait for the buffer to fill, but rather has a user settable low-water mark indicating what percent of the buffer should be used as a reserve. Whenever the low-water mark is reached, Klogger signals the Linux scheduler to run the appropriate CPU's thread, in order to empty the buffer.
When using Klogger to analyze a subsystem that can be affected by the memory dump operation, events caused by Klogger's own action might cause the buffer to fill up, thus causing the loss of other events. Such cases can be identified by gaps in the logged events' serial numbers. Another possible problem is if logging occurs too often the overhead caused by emptying the buffer might affect the measurement. For such cases, Klogger provides the user with an interface to increase the buffer size (defaulting at 4MB per CPU) and change the low-water mark. This interface is described in Section 5.5. Experience shows that increasing the buffer is sufficient even in the more extreme cases and allows for logging millions of events per second.
One warning though: albeit a very efficient framework, we must strictly advise against using Klogger in production systems. Klogger is designed to be a framework used by researchers and developers. As such, some guidelines that are usually required by subsystem designers were relaxed -- for example: the decision to allow some minimal (and even negligible) overhead even when logging is disabled.
A more comprehensive discussion of Klogger's design and implementation is available in Section 8 and in [3].