Updated: March 31, 2010
We have already learned how to configure Kdump for kernel crash dumping using Kdump in my long, detailed tutorial. While the article is more than sufficient to get you started, there's a number of changes introduced in the latest releases of openSUSE and Kdump that require additional comments.
In this article today, I would like to point out my observations in Kdump setup on openSUSE. Later, we will do the same thing with CentOS 5.4. These two articles are mandatory preparation for the long-awaited crash analysis tutorial.
Observations
Kdump works pretty much without any problems. Still, you may encounter a few odd issues here and there. I would like to help you understand these potential problems and provide you with the right tools to overcome them.
32-bit architecture
On 32-bit openSUSE 11.2 (Gnome), which I used for the setup, the configuration of Kdump was pretty straightforward. I downloaded and installed the required packages using YaST and then launched the YaST Kernel Kdump menu.
Kdump service
Kdump service has changed. It is no longer called kdump, it's called boot.kdump and is invoked during the boot. This means that you will have to adjust the usage of chkconfig for enabling/disabling Kdump. The new Kdump startup scripts make it more similar to LKCD.
You can also use the System Services module in YaST:
Memory allocation syntax
The memory allocation syntax has also changed. Although you can use the old crashkernel XM@YM syntax just like before, you will notice the default written in the GRUB menu.lst configuration file is slightly different.
The new syntax specifies a range rather just the starting point for the allocation. It's nothing cardinal, but worth paying attention to.
64-bit architecture
Memory allocation
Similarly, installing and configuring Kdump on 64-bit openSUSE 11.2 takes as much effort as doing that on the 32-bit machine. However, when I tried to dump the memory, instead of booting into the crash kernel via the Kexec mechanism, the system simply got stuck.
I realized the default allocating is incorrect. There are several ways you can ascertain this. First, when the system boots, you can hit Escape button to switch to verbose mode and then watch the console for Kdump error messages.
Alternatively, you can run Kexec and see if it throws any errors. Just execute:
kexec -p
We've seen this kind of message before, in the original tutorial and it tells us that the memory has not been reserved properly. Either you have used a bad offset or none at all.
The thing is, by default openSUSE, both 32-bit and 64-bit are configured to use the 16MB offset. You can check this value under /proc/config.gz, which contains the list of all parameters the kernel has been compiled with.
Unfortunately, while 16MB works for 32-bit, it is incorrect for 64-bit architecture. Furthermore, the CONFIG_PHYSICAL_START value set under /proc/config.gz is incorrect.
On my 64-bit openSUSE, it shows:
CONFIG_PHYSICAL_START=0x200000
If you translate this into decimal, it's only 2MB, below the 16MB value, an impossible allocation, when it should really read 0x2000000 or 32MB. Indeed, changing it to 32MB solves the problem. Of course, making the right choice from the start would be even better.
Other changes
Uncompressed kernel images
The new Kdump can worked with compressed kernels, so you will no longer require vmlinux under your /boot directory. Furthermore, the crash mechanism has also undergone some changes, allowing you to process memory cores in several different ways. But more about that is a separate tutorial.
debuginfo package missing
One more problem I've encountered is that there is no debuginfo package for the latest kernel available.
This means you will not be able to process your cores. Again, we will talk more about this in the crash tutorial, but for now, you should carefully inspect what your running kernel version is and what debug packages are available in the repositories.
Now, I may be mistaken, but here's what it looks like:
The running kernel is at version 31.8-0.1, but the debuginfo is only at version 31.5-0.1.1. For 64-bit systems, there's kernel-desktop-debuginfo version 31.8-0.1.1, but not 31.8-0.1, which again, poses a problem, as the two do not match.
I did not let zypper get in the way, so I did a manual check in the Update repository, looking for the RPM package that matches my running kernel and could not find it, in either 32-bit or 64-bit directories.
I hope this gets sorted soon.
References
Please refer to the following articles for more information:
Collecting and analyzing Linux kernel crashes - Kdump
Conclusion
Well, there isn't too much to say. These are all empirical observations, plus some notable differences in the basic functionality. Skimming through this article should help you setup Kdump more easily on the latest release of openSUSE.
We will soon have a similar article for CentOS 5.4, after which we will turn to working with the crash utility, overcoming all sorts of obstacles that may arise, and understand how to read the memory core dump analysis.
Cheers.
Download
This tutorial is a part of my Linux Kernel Crash Book. The book is available for free download, in PDF format. Please check the book article for more details.
