linux kernel hacking methods

http://la-samhna.de/library/rootkits/basics.html



2. The basics
2.1. Subverting the kernel
There are basically two ways how a rootkit can subvert the kernel to perform actions on behalf of an intruder:
Loading a kernel module
The Linux kernel (and many other operating systems) can load kernel modules (e.g. device drivers) at runtime. This allows an intruder to insert a module that overrides kernel syscalls in order to return incorrect values (e.g. does not list certain files), or provides new functions useful for an intruder (e.g. give root privileges to certain processes).
It is possible to compile a kernel where the interface for loadable kernel modules is disabled.
Writing to /dev/kmem
The special device /dev/kmem is basically a device that gives access to the memory region occupied by the running kernel. By writing to /dev/kmem it is possible to overwrite the kernel at runtime, and thus perform any arbitrary modification. The only problem may be to find the correct location in /dev/kmem, if one wants to overwrite some particular part of the kernel. However, this is usually a solveable problem.
Access to /dev/kmem can only be denied by patching the kernel. A patch to make /dev/kmem non-writeable has been published in Phrack issue 58, article 0x07 ("Linux on-the-fly kernel patching without LKM", by sd & devik). As noted in a bugtraq post by Guillaume Pellat, this patch still allows to write to /dev/kmem by using mmap() rather than direct file I/O.
2.2. Suitable places in the flow of execution
When a system call (e.g. open() to open a file) is made by an application, the flow of control looks like this:
An interrupt is triggered, and execution continues at the interrupt handler defined for that interrupt. On Linux, interrupt 80 is used.
A rootkit could replace the kernels interrupt handler by an own function. This requires a modification of the Interrupt Descriptor Table (IDT). A discussion of this method, as well as proof-of-cocept code, is published in Phrack issue 59, article 0x04 ("Handling the Interrupt Descriptor Table", by kad). This article also provides source code for a utility CheckIDT that can be used to list the IDT and save the current state to check its integrity later on.
The interrupt handler (named system_call() on Linux) looks up the address of the requested syscall in the syscall table, and executes a jump to the respective address.
A rootkit may (a) modify the interrupt handler to use a (rootkit-supplied) different syscall table, or (b) modify the entries in the syscall table to point to the rootkits replacement functions.
Method (a) is currently used by one rootkit only, the SucKIT rootkit presented in Phrack issue 58, article 0x07 ("Linux on-the-fly kernel patching without LKM", by sd & devik). This is a fully working rootkit that is loaded through /dev/kmem (i.e. it does not need a kernel with support for loadable kernel modules. It provides a password protected remote access connect-back shell initiated by a spoofed packet (bypassing most of firewall configurations), and can hide processes, files and connections.
Method (b) is used by a large number of rootkits, because it is very easy to code. For a list of these rootkits, see Section 3>.
The syscall function is executed, and control returns to the application.
A rootkit may overwrite the syscall function to place a jump to its own replacement function at the start of the syscall function.
No currently known rootkit uses this method.