Kernel panic - not syncing: VFS - How to solve


Updated: December 12, 2011

Say you want to use Kdump in your environment. You setup your machines. Then, you crash your system for testing purposes, in order to see whether Kdump can collect a memory core successfully. However, during the Kdump kernel startup, a panic message appears on the console. It reads: Kernel panic - not syncing: VFS: Unable to mount root fs on unknown-block(0,0). What do you do now?

There are two variations to this message and many possible reasons to why it shows up. The solution is very simple, but it may not appear that way if you try to debug on your own or hit the Web search for solution. You will find a million threads, all discussing this item, most of them half-baked guesses. Like the Ubuntu initrd issue, the solution is none of the suggested items, so please read and learn how to handle this properly.

Error message

Like I said, two possible error message types. Let's see them here:

VFS: Cannot open root device "LABEL=/" or 00:00
Please append a correct "root=" boot option
Kernel panic: VFS: Unable to mount root fs on 00:00

Or the other one:

VFS: Cannot open root device "sda1" or unknown-block(0,0)
Please append a correct "root=" boot option
Kernel panic - not syncing: VFS: Unable to mount root fs on unknown-block(0,0)

You may see different devices and blocks listed. It can be sdb4, sdc6 or something else. Likewise, the unknown-block can be any sequence of numbers, like (8,3) or (0,0). Whatever the exact match, the problem is identical.

So what gives?

Before you can solve the problem, you need to understand what it is. The reason why your Kdump kernel is panicking is because it is trying to load with bad parameters. Whatever strings are passed to the Kdump kernel are bad and interfere with the loading.

Normally, Kdump gets its information from two sources. One, the KDUMP_COMMANDLINE directive, as specified in the /etc/sysconfig/kdump configuration file. By default, the directive is left empty, which means that Kdump parses the standard kernel command line. You can check what this is by issuing cat /proc/cmdline.

But that's not all. Kdump mangles this line to make it useful. The init script located under /etc/init.d/kdump performs some search & replace changes on the collected string so it is good enough for the Kdump kernel.

Here's a sample:

if [ -z "$KDUMP_COMMANDLINE" ]; then
  KDUMP_COMMANDLINE=`cat /proc/cmdline | \
     sed -e 's/crashkernel=[0-9]\+[mM]\(@[0-9]\+[Mm]\?\)\?//g' \
        -e 's/ *memmap=[^ ]*/ /g' \
        -e 's/ *splash=[^ ]*/ /g' \
        -e 's/ *mem=[^ ]*/ /g' \
        -e 's/ *BOOT_IMAGE=[^ ]* / /g' \
        -e 's/ *showopts/ /g'`
  # Use deadline for saving the memory footprint
  KDUMP_COMMANDLINE="$KDUMP_COMMANDLINE elevator=deadline sysrq=1
  reset_devices"
  case `uname -i` in
  i?86|x86_64|ia64)
     KDUMP_COMMANDLINE="$KDUMP_COMMANDLINE irqpoll"
     ;;
  esac
fi

As you can see, all kinds of items are replaced and cleaned to avoid conflicts. Now, you understand how Kdump assembles its command line, piece by piece, taking from your stock kernel, removing lots of stuff and adding its own. But maybe it's not removing enough?

This is your problem 1: Check whether the parsed Kdump command line is clean of all kinds of weird items that might conflict with the loading of the kernel. Pay special attention to items like mem, memmap and others, which hardcode memory allocations.

How can you know if memory allocations are ok?

Again, we need to refer to /proc in our running kernel and see what is currently in use. Under /proc/iomem, you will see the memory reservations, which define blocks of kernel memory that can or cannot be allocated. While some of the items may not be applicable for the Kdump kernel, you will get a pretty good indication if you're trying to do something that conflicts with reserved segments. Here's an example:

cat /proc/iomem
00000000-0009dbff : System RAM
0009dc00-0009ffff : reserved
000a0000-000bffff : Video RAM area
000c0000-000cafff : Video ROM
000f0000-000fffff : System ROM
00100000-bf61ffff : System RAM
00100000-002f684d : Kernel code
002f684e-003db957 : Kernel data
01000000-08ffffff : Crash kernel
bf620000-bf63bfff : ACPI Tables
bf63c000-bf63cfff : System RAM
bf63d000-cfffffff : reserved
d0000000-d7ffffff : PCI Bus #0e
d0000000-d7ffffff : 0000:0e:00.0

After the cleanup performed in the init script, if your Kdump command line contains hard-coded items that might conflict with some of the allocations, like for instance Video RAM area or maybe one of the PCI buses, then your kernel might panic. You must make sure you do not override essential parts necessary for proper functioning of the system.

You will notice I am not giving you any specific instructions, because every system is different. But the principle remains the same and should guide you regardless what computer you're fighting against right now.

Second part, initrd

Another possibility is that your system cannot open a root device because it does not have the proper drivers to do so. Normally, basic drivers for hardware initialization are included in the initrd image. We talked about this in the Ubuntu initrd saga. For example, you may have special hardware that needs special drivers.

Your next task to solving the problem would be to examine what modules are listed under /etc/sysconfig/kernel and then add the missing ones. For example, if you plan on using LVM, it seems quite necessary to add the lvm module.

Initrd configuration

After you make the changes, recreate the initrd image.

/sbin/mkinitrd

On some systems, initrd may also be named initramfs under /boot, so don't panic if you see a different naming convention. The basic principle remains the same. Moreover, some systems may not use either, so that's a possibility, too.

Conclusion

The VFS error boils down to two things: Kdump command line and initrd modules. Make sure your system is configured properly. Make the necessary changes, like cleaning the kernel command line a little more and/or adding the missing modules.

The problem will usually manifest after kernel upgrades. It may also occur in your normal kernel, but this usually happens because of bad hardware or hardware conflicts. Your best bet is to proceed slowly and use a checklist. First, see whether the problem can be worked around by changes to the kernel command line or the initrd configuration. After that, move to other items, like filesystem checks, hardware components, etc.

I hope this article sheds some light on your woes. At the very least, you learned a little bit more about systematic problem solving, Kdump command line and init script, memory allocations under /proc, how to edit the kernel configuration, and create the initrd image. Well, party on, fellas!

Many thanks to Mr. Avi for his ideas and help with this thingie!

Cheers.

RSS Feed icon

del.icio.us del.icio.us stumbleupon stumble digg digg reddit reddit slashdot slashdot



Advertise!

Would you like to advertise your product/site on Dedoimedo?

Read more

Donate to Dedoimedo!

Do you want to
help me take early retirement? How about donating
some dinero to
Dedoimedo?

Read more

Donate