Updated: January 21, 2019
Virtualization is a nice, convenient technology that lets you incept your operating systems - you run an operating system on top of another, a computer inside a computer, and this means flexibility, legacy support, ability to test software on multiple platforms, isolation, and then, somewhat limited support for fancy graphics.
For a few years now, VirtualBox has had the option to enable 3D acceleration in virtual machines, which can help provide a smoother experience, as some of the graphics-intensive stuff is done more effectively. There has never been a perfect solution, but it's better than nothing, except those situations where things go wrong. Like enabling the VirtualBox Guest Additions and then getting a black screen on next login. Well, we should rectify that. This tutorial shows how to work out of the black screen situation without reinstalling, and even getting proper 3D acceleration enabled.
In a bit more detail, you have VirtualBox running, and you've installed an Ubuntu-based flavor as a guest operating system. Natively, Ubuntu does come with a basic set of drivers that offer mouse host-guest integration and screen resizing, but some features like shared folders and/or clipboard sharing might not work. So you will want to install the Guest Additions from the ISO for the full plethora of capabilities. You will then need to reboot or restart the graphical section to get it working. But the problem is, you might never get to the next login - you end up only seeing a black screen instead. What do we do now?
Introduction to Linux desktop sessions
Let's begin with some basics. First, create a snapshot of your virtual machine before installing the Guest Additions. This way, you can recover quickly and easily. In a way, this is not difference from having your data backed up and system imaged on a real, physical system before making any big, important changes.
Second, if there are problems with the desktop session login, you might want to get familiar with how the session management is done under the hood. Gnome-based desktops use the Gnome Desktop Manager (gdm) for logins. The actual configuration for each user is saved under:
The file (for each) user will contain declarations that define how the user session starts:
If you want the system to start with the default session, you can just use a blank XSession entry. Or use something like ubuntu. The names of these sessions correspond to the desktop session files under /usr/share/xsessions/. A typical file will contain the entries needed to start the right process (desktop environment). Something like:
Another thing to take into account: X11 vs Wayland. The latter may not work that well in a virtual machine environment, so you may want to test the desktop behavior (black screen included) by editing the necessary gdm configuration, and enabling/disabling Wayland. This is done through the /etc/gdm/custom.conf file. In there, look for the following line and comment/uncomment as needed:
These are your very first steps in troubleshooting any black screen or login issues. You may also want to consider disabling the automatic login, because if there are session-specific issues related to certain desktop environment, you don't want to be stuck in a constant login loop trying to reach a broken session. And now that we know all that, we can go back to doing our virtual machine stuff.
We need to recover the desktop first, to actually be able to work. First, to reach a virtual console in VirtualBox, press Right Ctrl (default Host button) + F1-7. This should give you a virtual console inside your guest without conflicting with the host's Ctrl + Alt + F1-7 combo. Unfortunately, you might not even have a virtual console, so this could be quite tricky, but there are a few ways around it, without a need to reinstall. Remember, we don't want to waste time with that.
Second, disable the 3D acceleration. Power down the virtual machine, open its settings, and then untick the right box. Start your machine, and you should be able to reach your desktop without any black screen. This means you have a working system, and you can commit to repairing the broken state properly. In general, the combination of having snapshots, using virtual consoles, and the 3D acceleration toggle should give you enough flexibility to troubleshoot issues of this nature without despairing.
Install VirtualBox Guest Additions from the repositories
All right, so, if you encountered problems using the drivers from the ISO, you will want to try the repo-provided content (while aware of the limitations). First, uninstall the drivers provided through the ISO by rerunning the script with the uninstall option:
sudo /media/(user)/(VirtualBox ISO path)/VBoxLinuxAdditions.run uninstall
Reboot. Now, run the installation from the repository. If you're running Wayland, you need:
sudo apt-get install virtualbox-guest-utils
If you're running X11 (most likely, especially inside VirtualBox), you need:
sudo apt-get install virtualbox-guest-x11
Power down the machine again. Toggle the 3D acceleration back on. Start the virtual machine, and it should work well. This is the first step toward achieving our goal. The settings are in place, and the desktop is working as expected.
Is 3D acceleration actually working?
But we still don't really know if we have the 3D acceleration enabled in the guest operating system. The easiest way to check this is with the glxinfo command, which is part of the mesa-utils package.
sudo apt-get install mesa-utils
And then run glxinfo, and check whether direct rendering is used:
glxinfo | grep direct
direct rendering: Yes
If the output says Yes, then you have 3D acceleration. Congratulations. Now, it might not be as smooth or fast as native drivers, but we're getting there. The most important thing is, no black screen. With this method, you won't have shared folders and the other advanced stuff I mentioned earlier, but there are ways around that, too, as we shall talk about in a future tutorial.
There was a lot of stuff here, but hopefully, all of that was good and fun. We focused on how modern Linux systems manage desktop sessions. We also learned how to toggle Wayland on and off. Then, we talked about the differences between the ISO and repo-provided drivers, how to use them, and various methods to keep your system in a robust shape, so if things do go wrong, you can recover quickly.
3D acceleration is slowly, gradually getting better in virtualization products, making them ever more useful. Sometimes, the complexity of the stack may rise against you. But even when things look rather bleak - and it looks like your desktop is busted, there are nice, elegant ways to work around them. We want to be efficient and non-destructive. Today, mission accomplished. See you later, virtual gator.