Updated: September 28, 2016
What is the most important component of an operating system? Well, arguably, it is the kernel. And if something goes wrong with the kernel and its associated files, you will not be able to your box, right? Example, the initrd problem we talked about a few years ago.
But what about glibc? What happens if you delete some of the important C libraries that power your booted system? How would you go about recovering from such a fiasco? In this tutorial, I am going to engage in some semi-mild hackery and teach you all sorts of nice self-rescue methods for fixing turbo-broken systems using busybox and a few other fancy tricks. If you're still wondering, this be a Linux guide. After me.
Scenario
All right. Let's talk about our system and how we can easily destroy it. To wit, a little bit of background on how Linux works under da hood. I've already explained this in detail in my fourth hacking tutorial, but here's a brief recap. And please excuse the lingo, I am not going to be 100% accurate here.
Linux programs normally come in two flavors - executables and shared libraries, usually dynamically linked. The programs are compiled in the Executable and Linkable Format (ELF), which defines how the program code should be loaded into memory and executed.
The loading of data into memory falls on the shoulders of two other programs, the standard C library (glibc) and the dynamic linker (ld), which calls on shared libraries with which the programs have been compiled. All the information on the file structure, including segments, sections, symbols, and such, are stored in the ELF, and can be parsed using the readelf command.
If you are to remove either glibc or ld, most programs will stop functioning, because they won't be able to load into memory and run correctly. So what happens if you delete, say /usr/lib64/glibc.so.6 file, which happens to be this highly critical library?
/bin/ld -v
GNU ld version 2.25-17.fc23
/usr/lib64/libc.so.6
GNU C Library (GNU libc) stable release version 2.22, by Roland McGrath et al.
Copyright (C) 2015 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.
There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
Compiled by GNU CC version 5.3.1 20151207 (Red Hat 5.3.1-2).
Available extensions:
The C stubs add-on version 2.1.2.
crypt add-on version 2.1 by Michael Glad and others
GNU Libidn by Simon Josefsson
Native POSIX Threads Library by Ulrich Drepper et al
BIND-8.2.3-T5B
RT using linux kernel aio
libc ABIs: UNIQUE IFUNC
For bug reporting instructions, please see:
<http://www.gnu.org/software/libc/bugs.html>.
Anyhow, this is what will happen - I will purposefully NOT write the command that moves the libc library away from its natural place so you do not accidentally copy it and ruin your box. But once you do that, any subsequent command you try to run that has not already been loaded into memory will fail:
ls
ls: error while loading shared libraries: libc.so.6: cannot open shared object file: No such file or directory
And if we try to fix by moving the glibc back into its place, again, no joy:
Catch 22. We need libc.so.6 to load other libraries, but it's gone. Hence, nothing will work. At this point, most people will panic, reboot and then panic some more. Again, no biggie, boot into a live session, copy the files into their right place, and Bob's your uncle. But we will discuss that separately. Let's focus on our special tools of the trade - Busybox and the LD_PRELOAD hack. But first, a disclaimer.
Caveats
Now, before we move forward, let's clarify a few points. One, you should always have full data backups and system images, for exactly this type of scenarios. Two, you should never delete critical system files, so be careful with your sudo or root rights. Three, you need to be comfortable working on the command line. Four, you do need an open root shell for the kind of work we will do today, otherwise it won't really work. Last but not the least, do not try this at home unless you really know what you're doing. Bonus, you can always fix broken Linux systems by booting into a live session. Now, onwards.
Busybox
If you're not familiar with busybox, it's a nice tool. First, it's a special snowflake. Unlike most other programs, it is designed to run on its own, without any dependency on shared libraries. The reason is, busybox is the default shell in small, embedded devices, and it needs to be tight and independent, so you can perform critical functions without relying on tons of data, which you can't store in the highly limited memory on embedded hardware.
If it's not installed, get it. For instance, on Fedora 23, it's not there, so you will need to install the whole 1.2MB worth of its equity. Then, let's explore the program's signature. Most importantly, it is not a dynamically linked executable.
ldd /usr/sbin/busybox
not a dynamic executable
file /usr/sbin/busybox
/usr/sbin/busybox: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), statically linked, stripped
The program is entirely self-dependent. It does not have any fancy symbols, group sections, dynamic section, relocations, or supported unwind sections. If you're not a developer, some of this stuff may look arcane, but what it tells us is that busybox can live on its own, without depending on glibc and ld.
...
Section to Segment mapping:
Segment Sections...
00 .note.gnu.build-id .init .text .fini .rodata .eh_frame
01 .data .bss
02 .note.gnu.build-id
03
There is no dynamic section in this file.
There are no relocations in this file.
The decoding of unwind sections for machine type Advanced Micro Devices X86-64 is not currently supported.
No version information found in this file.
Displaying notes found at file offset 0x00000120
with length 0x00000024:
Owner Data size Description
GNU 0x00000014 NT_GNU_BUILD_ID
Build ID: 7a9ff994b55730b373cabf88c8ca219d7b40f652
The program comes with tons of builtins. Basically, pretty much any command you can think of, Busybox has. For instance, if you want to see the list of loaded kernel modules, you will then run busybox lsmod. It also comes with important functions like su, mv, cp, ifconfig, and others. In essence, you should be able to do everything.
And then, the fix simply becomes (as defined in our example):
busybox mv /lib64/libc.so.6.bak /lib64/libc.so.6
Root permissions
Now, all of the above will work only IF you have a root shell inside which you want to run your busybox. It won't work if you try to do system changes from a user shell. The reasons are obvious, because that would enable pretty much anyone to gain root access.
busybox chown roger /usr/sbin/busybox
chown: /usr/sbin/busybox: Operation not permitted
/usr/sbin/busybox rm /usr/lib64/libc.so.6.bak
rm: remove '/usr/lib64/libc.so.6.bak'? y
rm: can't remove '/usr/lib64/libc.so.6.bak': Permission denied
busybox su
su: must be suid to work properly
A trick would be to setuid on the busybox binary - by root, BEFORE a fiasco. As it is owned by root, then in theory, yes, anyone running the shell should be able to do things as root. However, modern security mechanisms like SELinux and AppArmor will hamper your effort, and then, still, this is not something that should be too easily achievable, for obvious reasons. You really don't want to have a shell that allows users root access. Think about it. And be careful.
sudo chmod +s /usr/sbin/busybox
Not a silver bullet
Then, even if you do have root, you might not be able to do everything, especially if you've destroyed such a fine and delicate flower like glibc. I do admit it's not the typical usecase for busybox, but if you try to su into root with busybox on a system that doesn't have its glibc in the right place, you will fail. Rightly so.
Boot games
You may try to be extra clever and login into busybox in the early stages of the system boot. The same way you would declare init=/bin/bash on a typical and functional Linux system to reset the root password, you might go for this here. Except with no glibc, you will probably get a nice, juicy kernel panic. Then, the same goes if you try to invoke busybox as your init.What if you don't have busybox?
On some systems, busybox may not be available. Ideally, yes, if cacky hits the fan, your system, while trying to boot, will drop into a busybox shell, and let you do all sorts of nice magic. But if it's not there, there are still some cool hidden goodies to help you fix your broken boxen.
LD_PRELOAD
In our example, we only destroyed one file, but ld.so should still be functional. Which means it can load objects, but the problem is, glibc loads first. Luckily, you can use the LD_PRELOAD environment variable to tell your running shell to load specific objects first, before others, and this way, hopefully get around the glibc problem.
Speaking of glibc, there is more than one instance of the glibc shared library on the disk, and a few symbolic links, which means that even if you have moved the glibc.so.6 object away from its rightful place, we still have an ability to try to recover our system. What we essentially need is to be able to move or copy a file. Indeed, on most Linux systems, with a slight variation in the location and naming convention, there should be another copy of the standard library, with the full name. Using Fedora as an example, then you would have libc.so.6 and libc-2.22.so. Sometimes, the two could be actually symbolically linked. If you're lucky, then the copy or move fix becomes:
LD_PRELOAD=/usr/lib64/libc-2.22.so cp /usr/lib64/libc.so.6.bak /usr/lib64/libc.so.6
And you should be back to having a functional system.
Healthy practices
All right, and now that we've learned all of the above, there are some things you should remember and practice. Never delete system files, even if you do need them to go away. Move them aside, to start with, so you can move them back if needed. The exception to the rule are the glibc and the dynamic linker, which if gone, will cause a lot of problems with your system.
Never do fancy pipes and xargs removals of files when working under system directories. Always echo the outfit first, always type commands with the hash (#) key as the first character, so if you accidentally hit the Enter key, nothing bad happens. Try testing with innocent and unimportant files.
Never use environment variables to declare paths and filenames. You could end up deleting everything. Always go for full paths and names. Always.
Backups and system images, geddit?
More reading
Are you craving for more fancy knowledge? Read below then:
Highly useful Linux commands & configurations
Linux hacking guides one two and three - no. four linked earlier
System super debugging tutorial
How to use perf
And the book of course!
Conclusion
And here we are, at the end of this long and nerdy article. But hopefully, you have learned a lot, and I really mean, a lot of new stuff here: how Linux works, the intricacies of the ELF, what not to do with your system files and how to avoid tears and pain, and then a slew of recovery methods, including backups, system images, live CD games, and in more detail, our work with busybox and LD_PRELOAD.
We also learned a little more about the limitations of trying to recover from a badly botched situation, and why you should potentially have a root shell running on your box, and then all the reasons, why you really shouldn't let your users have any access to root files. 'Tis a double-edged sword. Anyhow, I'd like to believe you were enlightened as well as entertained. See you around for more gentle hackerology.
Cheers.