Troubleshooting common Xen problems


Updated: September 8, 2011

When the karma takes a turn for the worst, your virtualization experience may turn sour. If you're a fresh new user of the Xen technology, you may learn the hard way that Xen error messages are extremely verbose and not quite useful, unless you're a Python developer. With a verbose trace and cryptic references to internal function calls, the errors can actually detract you from what are inherently simple and easy-to-solve problems.

While I'm no expert on either virtualization or Python, I've come across a handful of common problems that take only a few short minutes to resolve, but may seem like an eternity to less knowledgeable users. P.S. I am an expert, but it is usually better to claim you are not, because 1) errors are excused then 2) you can afford to fail 3) people love humility and all that underdog attitude. Anyhow, let me help you enjoy Xen.

Tip 1: Display

Before you even begin, make sure your display is configured properly. Whether you use BASH or TCSH, export or setenv the DISPLAY variable to the suitable device. This should be the first step.

export DISPLAY=hostname:port

Tip 2: No module named <insert here>

You may get a million messages like this one: no module name xen.xm, no module named gtk, no module named whatever. In most cases, the source of the problem is identical, even if error messages might appear unrelated.

Let's see an example of an xm traceback; try running /usr/sbin/xm something and fail:

Linux #xm list

Traceback (most recent call last):

  File "/usr/sbin/xm", line 8, in <module>

    from xen.xm import main

ImportError: No module named xen.xm

There are two options: 1) the module is really missing 2) something is wrong somewhere else. Let us begin by exploring the first option first.

Missing Python module

The module could really be missing. You can verify this by checking whether the relevant package is installed on your host. For example, if the gtk package is missing, you can query the installation database for gtk and python-gtk packages.

You can also run the failing command with strace and look for missing files or directories that the utility is expecting. This should give you more info about the problem at hand. It is quite possible that you may have an incomplete Xen installation on your box.

Something else is wrong

Something else refers to Python setup. You could have the packages installed on your machine, however Python may not be seeing them for some reason. To work around this issue, you can manually set the PYTHONPATH variable and point it to site-packages directory.

export PYTHONPATH=/path-to-python-version/site-packages

You should also check your PATH variable to make sure you are indeed using the right version of Python. Running commands with full path is also always quite recommended. This brings me to tip number three.

Tip 3: Explicit commands

Believe it or not, most of the Xen utilities are wrapper scripts. Underneath, you get Python code. For example, Virtual Machine Manager (VMM) is also a Python thingie. First, you need to find where the utility hides:

which <xen utility>

Then, you need to see the contents of that file:

cat <path to xen utility>

Then, rather than going through the wrapper, you can execute commands directly. These wrapper commands all have relative names, which is a good thing, since your tools will work regardless of which Python version you're using, however they mask the root cause when you have problems.

which, cat commands

To see whether you have a real problem or one with your environment setup, run the commands with full path to your Python installation. For instance, rather than executing virt-manager you should run:

/usr/bin/python "/usr/share/virt-manager ..."

Or even use a specific version of python, like python2.4 or python 2.7.

Tip 4: Change scripts

This is not recommended unless you're a skilled user, but you could change the wrapper scripts to point to specific paths and Python installations. Let's go back to our initial problem. Running xm create <vm name> throws the xen.xm module error.

Now, we need to see what exactly xm command is. Running which against xm tells us it lurks under /usr/sbin/xm. Now, open this file in a text editor. It is indeed a Python script.

#!/usr/bin/env python

#  -*- mode: python; -*-

import sys

 

# add fallback path for non-native python path installs if needed

sys.path.append('/usr/lib/python')

sys.path.append('/usr/lib64/python')

from xen.xm import main

 

main.main(sys.argv)

In some versions of Xen, the first line is #!/usr/bin/env python. You may want to change this to a specific Python version and see what gives. For example, Python 2.4 or maybe Python 2.6. One of these might be misconfigured or missing from the environment setup.

#!/usr/bin/env python <--- delete

#!/usr/bin/python2.6  <--- use

And then, you may see that everything works, as you expect. For example, on my openSUSE 11.4 machine, which also has a Xen kernel installed:

xm works

Similarly, on newer versions of Xen, which just list /usr/bin/python, you may want making the same change to see whether you have a problem specifically related to your setup.

xm path, changes

Don't forget to backup the original files before making any changes. Furthermore, never try more than a single change at a time, just to make sure you don't create new, compounded problems.

Tip 5: Check services

Sounds silly, but you will hit a windmill of bad breath with your virtualization stuff if libvirt service is not started. You won't be able to connect to your dom0 domain and whatnot. Therefore, you should make sure it is running, as well as enabled on startup.

And I guess that's enough for now.

More reading

A very worthy resource, over at Virtuatopia.

You may also be interested in RedHat Documentation, Common Xen Problems.

Conclusion

This article may not be the prettiest, then again, Python traceback messages are not exactly the work of beauty either. Too much verbosity can mask the real problems, which could be as simple as setting the correct environment variable or changing the Python version. Hopefully, this guide should save you many hours of unnecessary hassle and frustration.

You have a lot to read, so you'd better do it. Just head over to my virtualization section and starting clicking like mad. There's the introduction article, there's the Xen Cloud Platform and XenServer, there's the live CD for exploration. We'll get back to more Xen goodies. There's also KVM, if you fancy another open-source solution. Enjoy!

If you have other simple, global, universal, or common tricks you'd like to share, don't be a stranger, send an email. Dedoimedo out.

Cheers.

RSS Feed icon

del.icio.us del.icio.us stumbleupon stumble digg digg reddit reddit slashdot slashdot



Advertise!

Would you like to advertise your product/site on Dedoimedo?

Read more

Donate to Dedoimedo!

Do you want to
help me take early retirement? How about donating
some dinero to
Dedoimedo?

Read more

Donate