Updated: December 29, 2010
How do you audit your Linux environment? How do you track after changes to your files? What kind of processes are running on your system at any given moment? What uses the most resources? Valid questions, all. Some, we have already answered in a number of articles, where we covered cool hacks and tricks and handy tools for in-depth administration and debugging work.
We monitored system activity with sar. We used iostat, vmstat and dstat to collect statistics on resource usage. We learned about pacct for process accounting. We also worked with top in batch mode and atop. Finally, we worked with some high-end debugging tools like OProfile and strace. Links below. But we did not use any utility for file auditing.
Today, we do that. The name of the game: Linux audit.
Linux audit is a built-in mechanism in the kernel, which allows you to intercept changes to monitored files and write them to a log on the disk. The mechanism consists of three major components: auditd, the audit daemon (background service), auditctl, the program used to control the auditing system, and ausearch, a tool to query audit daemon logs.
To get audit configured, you will need to install the audit (or auditd) package, if it's not already installed. Once installed, you will probably want to setup audit to run on startup, using a system command-line tool like chkconfig, for instance. You can also manually control the service via /etc/init.d.
That's a good question. The big difference between audit and version control is that the first is mandatory. Version control works only if people want to cooperate. Let's see an example.
Take subversion (svn), for instance. You can have all kinds of files and directories under version control. Indeed, every time you update your files from the central repository, you will override any local changes done to your files. While svn blame will tell you about file edits against the subversion server, it won't tell you anything about local changes. Furthermore, this requires another layer of setup.
An alternative is configuration management, like cfengine. This could work, too. You will have a static baseline to revert to, deleting any unwanted changes to your files. However, you will not know, in between period runs, who made changes to your files - or why.
Both version control and configuration management are excellent resources, but they rely on good will. They work extremely well in an environment where you have cooperative users working toward a common goal. If you merely want to police your files against all and any desired and undesired changes, then you want auditing.
So auditing it shall be.
Let's see how this thing works. First, let's make sure the service is running:
Now, we can start working on collecting audits. You can configure auditing rules on the fly via auditctl, plus you can set some defaults using the configuration files.
/etc/audit/auditd.conf is used to configure log file location, size and flushing (rotation), and various actions like mail and disk space events. /etc/audit/audit.rules lists auditing rules that are loaded on the service start, restart or reload.
But we will start with using auditctl from the command line. You will need root or sudo to be able to conduct system-wide auditing. Let's try an example, which is pretty much what you will find in the man page. We will configure a watch using auditctl.
Or in text, if you can't see the image:
auditctl -w /etc/hosts -p war -k hosts-file
What do we have here?
-w flag specifies the file to be watched. -p sets the permission filter; in other words, what activity you want to audit. You have four options, r = read, w = write, x = execute, a = attribute change, which could be owner, permissions or one of the special file attributes, depending on the filesystem. -k is a filter, which you can later use to search through your records in a more meaningful manner. Call it a tag if you will.
Other useful flags
There are many more flags for auditctl. For instance:
-W will remove the watch on a specified file.
-S allows you to trace specific system calls. You can specify them individually or use all for all system calls. This is somewhat similar to -e flag for strace, as we've seen in the separate tutorial.
Flags -a and -A allow you to add rules to the end or the beginning or the rules list, respectively. Now, -d will delete a specific rule. -D will delete all rules.
Adding rules uses a list,action pair, where you first specify the list entry, which could be one of the following: task, which is generated when a process is created, entry, when you enter a system call, exit when you exit a system call, and so forth. Action can be never and always.
We will see more examples later on. Now, we have a watch in place, let's edit our /etc/hosts file and then see what ausearch gives us.
ausearch can be run any time you want, while auditd is running. The output produced is extremely detailed and should not be read at face value. You may probably want to parse the information, to grab only whatever interests you.
The basic usage is:
But this will produce an ugly output. You can use the -i flag to make it more reasonable to read; even so, you will have a lot of data to process.
What do we have here?
type tells us what kind of record we have. For file watch rules, it will normally be PATH. For system calls, it will be SYSCALL. msg is the record timestamp. name tells us the filename for files. You also have file type, permissions, owner, group, device major and minor numbers, as well as the inode number.
For system calls, it's a little more complicated. syscall tells us what system call was, uh, well, called. success tells us if it worked. You also get the process id (pid) and the parent process id (ppid). There's a long set of real and effective user and group ids. You also have the console number. exe specifies the command that was run. And so forth. Detailed and only useful for experienced administrators knowing what they need.
This is very similar to going under /proc and burrowing into data for specific processes, but the thing is, it's done automatically and faster than a human could possibly log information.
Other useful flags
-k will search by specific keys (tags) only. Otherwise, why did we use them in the first place? -m will filter only specific types of records. -sc will match only specific system calls. -p will only match specific processes.
You can also redirect your output into text editors or files.
There are quite a few examples in the man page. Let me be a monkey and repeat some of them here, for the sake of education.
To see files opened by a specific user:
auditctl -a exit,always -S open -F auid=510
We will always monitor the exit from the open system call for the audit uid (auid) 510, which might correspond to some user, like roger. auid is the original ID the user logged in, so if he/she changes his effective UID while working, it won't be masked in the log. We want to monitor the exit from the system call, because if it fails, we will know the exact error.
Another example is:
auditctl -a exit,never -S mount
It says, don't log any mount system calls.
Here, rather than collecting as much as we can, we suppress output. This could be useful if you know you might have too many logs or the information is just not interesting. For example, in the NFS environment, you may not want to do any auditing on the client side and will instead focus on doing it on the server side. Besides, in this case, there are other tools you may want to use. For example, pacct, access control via access.conf, traffic logging, export options for your NFS areas, and more.
One more for ausearch:
ausearch -ts today -k hosts-file -x vi
-ts tells us the timestamp we want to search through our logs; today, only. -k narrows down our search to the hosts-file key only. -x specifies the exact name of the executable run against the watched file. In this regard, our example looks for any attempts to open or modify the file with vi text editor today.
That's enough, I think.
There are more helper tools that can help you work with auditing. One such tool is aureport, which you can use to generate summary reports of collected logs. This is somewhat similar to -c flag in strace.
A handful of my system administration oriented articles.
Some external links:
Finally, man pages:
And that would be all folks!
You have just gained another powerful tool that you can use to make your Linux environment smarter and better controlled. Alongside an ever-growing repertoire of powerful utilities, audit will help you build your Linux street credit and make your work easier. audit is very simple to use, although processing the collected logs can take time and effort. You need to master all the little flags and understand how the system works before you can delve deeper.
If you have any useful tools you considered using but didn't dare try, feel free to ping me and I will testdrive them for you, trying to provide you with the best and most modular advice while striving for strategic imperatives rather than tactical keyboard hammering.