Repairing system after renaming

Posted on Monday, October 14, 2013

Repairing system after renaming

Well, we will notice the disaster we have caused because every time you try to run any executable you will get the following message:

No such file or directory.

What is not finding is, which is a library that is linked by all dynamic executables on the system, so the first thing we must do is not to close any session or any program you have open. Because, since we can't execute any dynamic executable, every program we close will be a program that we might need and won't be able to reopen.

If we have busybox installed, we will be lucky because it is a static executable and does not require any library to run. So now our problem is solved, just execute:

busybox sh

That launches a shell where we have a lot of standard UNIX commands implemented in a single statically linked executable from which we can restore library to its previous location and be more careful next time.

List of available internal commands in busybox
List of available internal commands in busybox

Unfortunately, this was not my case (mental note: install busybox in all servers), so I had to work a little more, but once you realize what you have to do, it's very easy.

Note that the case I am talking about is if you renamed library. If what you have done is to delete it, I'm afraid you have nothing to do. You will have to use a Live CD or something like this to recover the library. Okay, yes, the Live CD is usually a much faster and easier way than anything else I'm trying here, but it isn't the first choice when it's about a server that you're running remotely.

Okay, the first thing we have to realize is that we can't use any command external to the shell we're running, which I'll assume it's bash, because if you use other things you are complicating your life without a reason. But we have available a lot of bash internal commands but unfortunately none of them is directly able to copy or rename files.

But if you use your imagination, you will realize that sending the source file to stdin, running a command that takes stdin and sends it to stdout and then redirecting stdout to a destination file might be equivalent to copying a file.
That is, we know that:

command < file

uses file as standard input for command and

command > file

takes command output and saves it in file.

So, If we have a command that takes standard input and sends it as is to standard output, we can copy a file with

command < source > destination

And this is basically what cat does. That is, this:

cat < source > destination

is essentially the same as

cp source destination

But cat is an external command, we need to implement our own cat using bash internal commands. So we need a script to take standard input and show it on standard output.
Well, the bash internal command that shows what we want on stdout is classic echo, right? And to take something from standard input, the first thing that comes to mind is read command. Now, what we have to achieve is that what read takes from stdin echo has to be able to display it to stdout , and that is very easy if you know how read works. read reads a line from stdin saves it in a variable. And echo is able to display that variable on standard output so we have our "homemade cat":

while read line
  echo $line
done < source > destination

You may not know the exact path of some of the files and need the list of files in a directory. But ls is also an external command, so we neither have ls. However, we can get the list of files in a directory just with tab completion. Although if we have enabled any of these smart autocompletions we may not get all the files or even get a list that has nothing to do with the files in a directory. You can disable intelligent autocompletion by running:

complete -r

So, if we want to get the contents of /etc directory, we can type:

command /etc/

and press tab key twice. If we had disabled programmable completion, it doesn't care which command we use, even if we use a nonexistent one:

Using tab completion to get a directory listing
Using tab completion to get a directory listing

Well, with these two ideas we already have a kind of internal implementation of cp and ls.

This implementation could be improved since it doesn't take into account some special cases, but anyway, we will not improve it because it doesn't fit our needs. It doesn't fit because what we want to copy is a library, so it has to have execute permissions and as far as I know, no file created with bash internal commands will have execute permissions. And we can't change permissions because chmod is an external command.

So we will have to think of another option. And the other option is even more simple, to the point that you may feel a little stupid when you realize. Because obviously, you still have your library, what happens is that you have it in a wrong location.
As told above, is the dynamic loader, it's the library that loads all the other libraries and when our executable requests a library, usually asks for it just by the name of the library and is responsible for finding the location of that library. But path is hardcoded with its exact location in the executable, because until we load "nobody" is responsible to search libraries by name rather than by its full path.

This is like this with dynamically linked executables, that by having part of the code "outsourced" in libraries (.so on UNIX and .dll on Windows) allows us to share that code with other applications that also use it instead of repeating the same code in all applications that need to save disk space and memory. The problem is that if we remove some of those libraries the program will stop working, as is the case. Furthermore, a change in the version of the library we have installed can cause programs that use it to stop working.

On the other hand are statically linked executables, in which no code is outsourced in libraries but a single executable already contains all the code needed to run. They have the advantage of being more tolerant to errors and system disasters and incompatibilities, but occupy more disk space:

Static files have larger size but don't depend on external libraries.
Static files have larger size but don't depend on external libraries.

Since is the library that loads all others, it's statically linked. But, in addition, on Linux it is directly executable from command line:

This allows us to make the loading process in a slightly different way. Instead of launching our executable so it loads and then the loader loads the rest of the dynamic libraries, we will load first and then run the executable (which will no longer try to load because it is already loaded) and then the previously loaded will load the rest of the dynamic libraries.
The way to tell the application we want to run is by passing it as a parameter in the command line:

/lib/ /bin/bash

So now you can run cp to copy files (note that you have to type the full path of the file you want to run):

/lib/ /bin/cp source destination

Furthermore, once the loader has loaded, you can tell it a list of libraries you want to load before starting the program in the environment variable LD_PRELOAD separated by colons. So, if it turns out another required library have also been renamed, you can explicitly tell to the new path for this library:

LD_PRELOAD="/lib/" /lib/ /bin/cp source destination

Lessons learned:

  • Thou shalt not rename thy in vain.
  • With a little bit of imagination, we are able to copy files or get a directory listing using only bash built-in commands.
  • Statically linked executables may be much larger, but some of them may save your life, especially busybox.
  • Linux loader behaves like another executable that can be called from the command line passing the executable you want to run as a parameter. This even allows a particular executable to use a custom loader instead of the system one.

By the way, along this article I sometimes referred to Linux loader as and other times as is the generic name used by UNIX loaders, including Linux loader for old a.out executable file format. The Linux loader for current ELF executable file format is called