Chkrootkit LKM Trojan installed warning - What now?

Updated: December 14, 2011

First, a fact. There's no malware for Linux. Why? The primary reason is neither financial gain, nor interest, nor market share, nor the user skill, not even the security defaults built into the operating system. It is the simple fact that Linux code, while extremely highly portable, is in fact, not at all portable. The delicate combination of ever so slight differences between distro flavors, the headers, the libraries, and the variety of kernels and compilers makes executing random code on random machines extremely difficult. It is one thing to bundle a static application and get it running. Planting a module into the kernel, live and without errors, well, that's quite another.

But as it happens, many Linux users are also Windows users. And what do you do in Windows? You scan your system for malware. Can you name some malware scanners for Linux? Sure. There's chkrootkit and rkhunter. So you run them. And then you see chkrootkit report a warning about a possible LKM Trojan installed. Fear. What now?

Teaser

Understand the problem

Let's begin with an even more basic step. Do you even have a problem? A scanner says so. Not only that, it complains about LKM, which stands for Linux Kernel Module. This is not just some runaway process running as root, this is a kernel module, resting in the very heart of the system. This is the mother of all problems. But do you trust this scanner? Is it good enough, accurate enough? How do you start debugging something that sounds so wicked?

What you need to understand are many many things. The first one is that linear translation of the security model from one operating system to another is not the right thing to do, most of the time. In other words, Linux behaves differently than Windows, hence your examination of the process table in search for baddies might be misplaced.

The second thing to understand is that either of the two scanners mentioned might be actually wrong. False positives are not uncommon with blacklist scanners, which makes them tricky and somewhat unreliable. Third, if you go through the logs created by either of the two scanners, you will notice a finite and rather short list of results, indicating a very basic check for common issues, some of which are old, antiquated leftovers from years gone past. Therefore, the scanner logs might be indicative, but they sure are not a doomsday verdict. Now, let's approach the problem.

Analyze the problem

If you get the warning from chkrootkit, what is the next logical step to do? Rerun the software again and verify the results. Then, compare with the other scanner. If only one is reporting the problem, if the issue does not show up consistently in the results of either or both, you might be facing a bogus warning. Now you see me, now you don't, that's usually the system playing silly on you.

lkm warning

Clean scan

Moreover, ask yourselves - is your system working properly? Does the system monitor show any network activity even when you shut down all your downloads and uploads. Do you suffer from crashes and applications errors, slowness, bugs, etc? Usually, none of these will be related to malware, even if chkrootkit shows something in one of its logs. In the best case, you will have misconfigured software, in the worst, some failing hardware. Malware should be your last option, after you have exhausted everything else.

But assuming your system works dandily, without any issues, then you are most likely ok. Now, the astute paranoids among you might claim that you are having a case of perfect malware that cooperates well with your hardware. So how do you go about separating fiction from reality?

You have three long and healthy articles on Linux security waiting for you just a few scrolls down. Read these first to get an inkling of understanding why your paranoia level should be less than what it is. After you've done that, here's the one practical thing you will want to do to banish the spell of anguish forever.

Kernel modules, how to dissect them

To follow the next section, you will need some rudimentary understanding of the Linux bootloader, the boot process, runlevels, startup scripts, services, how the dynamically loadable kernel modules work, how the system uses them, and a little bit about compilation. None of these topics are simple, but if you've followed my tutorials over the years, you will know how to approach the problem, step by step, without getting lost in the sea of details and strangle lingo.

In general, kernel modules are loaded into matching kernels. If the two are not compatible, the modules will not load. plain and simple. You can determine this by checking your running kernel with uname -r, examine the available kernels under /boot, which are loaded on demand by different entries in your bootloader menu, and then compare against the kernel modules, which you can find under /lib/modules, stored in directories that match specific kernel versions.

Libs

So, what we need is try to load a different kernel than the one you have. If we assume you have a bad kernel module polluting the current kernel, starting the system with a different kernel will cause your bad module to fail to load. This should render it unusable and your scanner will return a clean slate. This is presented in great detail in my crash book. Now, let's go step by step.

Step 1, edit your GRUB or GRUB2 menu and start an older version of the kernel. Most likely, there will be one or more left over under /boot. If not, download one using the package manager and install it. However, make sure you do not download the kernel headers for that version, and if they are present, uninstall them. You want to do this last step to make sure no software on your system will be able to compile kernel modules for the older kernel version. While this could cripple benign software from starting, for the sake and duration of our troubleshooting, it will warrant no undesired modules are compiled and loaded.

A perfectly legitimate and benign example is the re-compilation of the VMware Workstation modules after kernel updates. This allows your system to seamlessly resume working, without you tampering manually.

Compiling

Step 2, reboot. Your would-be malware kernel module, if present, will have a hard time booting into a kernel that it has not been compiled for. Like mentioned before, this is the built-in safety mechanism of Linux and has nothing to do with malware. The simple fact is, if you try loading an incompatible kernel modules, you will most likely crash your host with a lovely panic. In general, if you try inserting a bad module, you will get a nice error:

insmod: error inserting <name> ... Invalid module format

Now, the bad module might be smart enough to detect new kernels under /boot, similar to what some programs do, like for example, VMware Workstation. But without the userland component, any such kind of calls from the kernel space into the user space is virtually impossible. Therefore, the bad module will not be compiled, as it does not have the correct build environment, which includes the proper headers, as well as its own source code somewhere on the filesystem, to say nothing of an actual mechanism to figure out what components are missing. In other words, not bloody likely. Hence, that's all you need to do to solve your problem. Of course, run chkrootkit and check again.

Now, since kernel to userland communication is tricky, the bad module might also come with a service script of its own, which launches a userland program that might help regenerate the bad kernel module and insert it into any which kernel is currently running. Not to worry. We solve this by being geeks. And you learn being geeks by reading my rather useful Linux commands article.

System services, startup scripts and shell login

Step 3, head under /etc/init.d/ and check for unknown or weird services. This directory holds a series of scripts that start and stop system services. For example, the network script will be there, named something like /etc/init.d/network. If you were to run this script with the correct argument, you would be able to start, stop, restart, or reload the service.

/etc/init.d/ dir contents

The services can be invoked manually, but they are normally called by another set of scripts, called Start scripts and Kill scripts, which are located under /etc/rcX.d directories, where X represent the correct runlevel. When the system boots, based on the default runlevel choices, Start scripts in the relevant rcX.d directory are executed, in numerical order. Likewise, when the system goes down, Kill scripts are executed. Notice that the order of services loading is reversed for Start and Kill scripts. The first to load will be the last to go down and vice versa.

rcX.d directory

Step 4 is therefore to examine the startup scripts in the relevant runlevel directory and check for weird items. I must apologize for the word weird, but having a startup script called S43foo is most likely weird, whereas items like firewall, network, ntp, cron, and others are perfectly legitimate.

Step 5 is to examine your shell login scripts in your home directory and comment out any strange entries. For most people in the Linux world, the default shell will be BASH, therefore, you need to take a look at the hidden BASH files in your home directory. There are several other login related scripts, which might also contain startup entries for your own custom software as well as planted badware. It is impossible to list all of these files here, but some of the more important once include:

/etc/profile - Login script that setups your environment

/etc/bashrc - Login script that setups your shell

~/.profile - Your own customizations, plus it sources the above

~/.bashrc and ~/.bashrc.$USER - Your own customizations, plus it sources the above

And many others, including /etc/bash.bashrc, ~/.bash_profile and others.

You may have some or all of these files on your system. You may have many others. Some distributions will use their specific schemes and changes. You might be using a different shell. Some programs may use a subset of these files, skip some, use others. Interactive and non-interactive programs will use different paths to spawn their shells. You might have a long and convoluted setup with lots of sourced files and linked scripts. The full hierarchy of execution and sourcing is beyond the scope of this article.

Bash scripts

That's all of the magic. There's nothing special or sinister at hand. Once again, this same procedure has nothing to do with malware. It's common troubleshooting. The real skill is knowing what services and scripts are normal and what don't belong there. This is not something you can master overnight, but if you use the Web to search for script names and commands, you will quickly learn what each entry does.

Finally, if you have more than one system or boot instance available, do compare between the results to see whether one of the systems has entries that should not belong there.

Get on with your life

Bottom line, you will see this message if you try to run both chkrootkit and rkhunter scanners at the same time. You will see the message any time the 'lkm' check reports hidden processes inaccessible by the readdir command. You will see this message if you use the BTRFS filesystem. There are several other scenarios that will lead to this message being shown. It is a bug that has existed for several years.

The important thing is not what the scanner reports - it's how you respond to the report. We had a message, we analyzed it, we find it bogus. Therefore, it's time to get back to enjoying the web.

More reading

Some security related tutorials and rants:

Do I need an anti-virus in Linux?

Linux = Windows anti-virus? Not!

Linux security - A few useful, tactical tips

Conclusion

The point of this article is not to focus on the security and how to battle the malware ghosts. That's overrated, boring and pointless. What is important is to know how to respond to error messages and warnings. That's the core issue at hand.

This short article might be too full of itself, but it does teach you a few neat things. How to correlate results and not blindly jump to conclusions; how to examine problems and evaluate their severity as well as validity; how to boot into another instance of the kernel to examine if one of the loaded modules is faulty or misbehaving, which applies equally well to any update and is the main reason why older kernels are kept under /boot as the normal part of the Linux maintenance cycle; how to examine services startup scripts, runlevel scripts and login files. And finally, how to address our real problem at hand.

Well, that would be all. I hope you find this article worthy and useful. Don't panic. Don't blindly follow online advice. Malware is not the reason why the Internet exists. Your coffee mug resting by the keyboard is a greater threat to the longevity of your data than any evil virus. Now, this is a tremendous opportunity to remind you of the importance of system imaging and data backups. Stay cool.

Cheers.