Running ARM binaries on Ubuntu x86 with QEMU

Posted on by Andrei Agape

For this article I’ll use the example of extracting the http daemon (build for ARM) from my home router and run it on my Ubuntu 18 on x86.

Resources

Steps

  1. Download the httpd and shared libraries from router
  2. Install QEMU
  3. Run httpd with QEMU
  4. Create a chroot jail
  5. Find all shared libraries + run bash in chroot
  6. Set LD_LIBRARY_PATH inside chroot jail
  7. Automate the LD_LIBRARY_PATH set

1. Extract binaries and libraries from router

First step is to extract the binary from the router. We can download it using nc static for ARM

y@router: cd /tmp
y@router: wget https://github.com/therealsaumil/static-arm-bins/raw/master/nc-arm-static
y@router: mv nc-arm-static nc
y@router: chmod +x nc

Set up nc listener on our machine

x@ubuntu: nc -lvnp 4444 > httpd
Listening on [0.0.0.0] (family 0, port 4444)

Send the binary from router to listener

y@router: ./nc -w 10 -n 192.168.0.17 4444 < /mnt/apps/bin/httpd

Kill the listener (ctrl+c) and check if the md5 sum of the binaries on both systems match. This is to ensure that the transfer was successful and my copy has not been corrupted

x@ubuntu: ls -l
total 0
-rw-rw-r-- 1 user group 0 Mar 14 08:52 httpd

x@ubuntu: md5sum httpd
cac768ae107614f54acd7dd40ca7d99b  httpd

y@router: md5sum /mnt/apps/bin/httpd
cac768ae107614f54acd7dd40ca7d99b  /mnt/apps/bin/httpd

To download a whole folder (i.e: all shared libraries from router) we can archive it using tar and exfiltrate with nc. 

y@router: cd /tmp
y@router: tar -cvf lib.tar /lib
y@router: tar -cvf mnt_lib.tar /mnt/apps/lib

y@router: ./nc -w 10 -n 192.168.0.17 4444 < /tmp/lib.tar
y@router: ./nc -w 10 -n 192.168.0.17 4444 < /tmp/mnt_lib.tar

2. Install QEMU

To run the ARM binary on a x86 processor, we can use QEMU

x@ubuntu: sudo apt-get -y install qemu qemu-system qemu-user-static qemu-user

3. Run httpd with QEMU

Copy the QEMU static binary to the same folder as the ARM binary we want to execute (httpd in this case), and run the binary. The “ld-linux-armhf.so.3” shared library cannot be found.

x@ubuntu: cp /usr/bin/qemu-arm-static ./
x@ubuntu: tree
.
├── httpd
└── qemu-arm-static

0 directories, 2 files
x@ubuntu: ./qemu-arm-static httpd 
/lib/ld-linux-armhf.so.3: No such file or directory

We can however find and download the library from router as well using nc and try again

y@router: find / -name ld-linux-armhf.so.3
/lib/ld-linux-armhf.so.3

<-- nc exfiltrare /lib/ld-linux-armhf.so.3 -->

x@ubuntu: tree
.
├── httpd
├── ld-linux-armhf.so.3
└── qemu-arm-static

x@ubuntu: ./qemu-arm-static httpd 
/lib/ld-linux-armhf.so.3: No such file or directory

The error complains that the library cannot be found in the lib folder (“/lib/ld-linux-armhf.so.3: No such file or directory”). If we copy it to the /lib folder, we can indeed see that the error message changes, however we don’t want to copy all required libraries for all our projects to the /lib folder as this can grow really fast.

x@ubuntu: ./qemu-arm-static httpd 
/lib/ld-linux-armhf.so.3: No such file or directory
x@ubuntu: sudo cp ld-linux-armhf.so.3 /lib/
x@ubuntu: ./qemu-arm-static httpd 
httpd: error while loading shared libraries: libcms_dal.so: cannot open shared object file: No such file or directory

4. Create a chroot jail

To avoid messing with the base OS libraries, we can use chroot. First we reproduce the same directory structure as on the router:

x@ubuntu: mkdir -p mnt/apps/bin/
x@ubuntu: mv httpd mnt/apps/bin/
x@ubuntu: mkdir lib
x@ubuntu: mv ld-linux-armhf.so.3 lib/
x@ubunut: tree
.
├── lib
│   └── ld-linux-armhf.so.3
├── mnt
│   └── apps
│       └── bin
│           └── httpd
└── qemu-arm-static

4 directories, 3 files

Now we if remove the from /lib directory, and run the httpd with qemu emulator, we see that we get the same error as before, so we know that qemu is now looking the the chroot lib folder, and not the /lib.

x@ubunut: sudo rm /lib/ld-linux-armhf.so.3
x@ubunut: sudo chroot `pwd` ./qemu-arm-static ./mnt/apps/bin/httpd
./httpd: error while loading shared libraries: libcms_dal.so: cannot open shared object file: No such file or directory

Now the error tells us that the libcms_dal.so is missing. If we search for this library on the router again, we can see that is located in another directory 

y@router: find / -name libcms_dal.so
/mnt/apps/lib/private/libcms_dal.so

If we proceed as before, an download the library from the router, create the same folder structure, and move it there we still get the same error even tho we followed the same structure. The reason for this that the ld.so does not know where to look for the shared libraries (except the common places like /lib)

x@ubuntu: mkdir -p mnt/apps/lib/private/
x@ubuntu: mv libcms_dal.so mnt/apps/lib/private/
x@ubuntu: tree
.
├── lib
│   └── ld-linux-armhf.so.3
├── mnt
│   └── apps
│       ├── bin
│       │   └── httpd
│       └── lib
│           └── private
│               └── libcms_dal.so
└── qemu-arm-static

6 directories, 4 files

x@ubuntu: chroot `pwd` ./qemu-arm-static ./mnt/apps/bin/httpd
./mnt/apps/bin/httpd: error while loading shared libraries: libcms_dal.so: cannot open shared object file: No such file or directory

5. Find all shared libraries + run bash in chroot

Before we fix this, we should make sure we have all needed shared libraries from the router and that we can more easily interact with our chroot (i.e.: getting a shell inside it).

We can download one library at a time, execute the binary and see the error message for the missing message. Alternatively we can run ldd on the binary and see all used shared libraries (example with bash on ubuntu)

x@ubuntu: ldd /bin/bash
	linux-vdso.so.1 (0x00007ffc2b46d000)
	libtinfo.so.5 => /lib/x86_64-linux-gnu/libtinfo.so.5 (0x00007f44ea3bb000)
	libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f44ea1b7000)
	libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f44e9dc6000)
	/lib64/ld-linux-x86-64.so.2 (0x00007f44ea8ff000)

Here we can see all shared libraries used by bash, along the path of these files. Now if we want to run bash in chroot, we would have to replicate the same folder structure and copy the libraries listed previously

mkdir -p lib/x86_64-linux-gnu/
cp /lib/x86_64-linux-gnu/libtinfo.so.5 lib/x86_64-linux-gnu/ 
cp /lib/x86_64-linux-gnu/libdl.so.2 lib/x86_64-linux-gnu/
cp /lib/x86_64-linux-gnu/libc.so.6 lib/x86_64-linux-gnu/

mkdir lib64/
cp /lib64/ld-linux-x86-64.so.2 lib64/

mkdir bin
cp /bin/bash bin/

Now the folder structure looks like this

tree
.
├── bin
│   └── bash
├── lib
│   ├── ld-linux-armhf.so.3
│   └── x86_64-linux-gnu
│       ├── libc.so.6
│       ├── libdl.so.2
│       └── libtinfo.so.5
├── lib64
│   └── ld-linux-x86-64.so.2
├── mnt
│   └── apps
│       ├── bin
│       │   └── httpd
│       └── lib
│           └── private
│               └── libcms_dal.so
└── qemu-arm-static

9 directories, 9 files

And we can run bash within the chroot using the following command

sudo chroot `pwd` ./bin/bash
bash-4.4#

6. Set LD_LIBRARY_PATH inside chroot jail

Now that we can have a shell inside the chroot, we can go back and specify what are other places to look for shared libraries (in our case /mnt/apps/lib as on the router)

From @ketansingh:

The programs ld.so and ld-linux.so find and load the shared libraries needed by a program, prepare the program to run, and then run it unless program was statically linked. Dynamic linker tries to load shared library (*.so) from :

1. directories listed in the LD_LIBRARY_PATH environment variable.

2. directories listed in the executable’s rpath.

3. directories on the system search path, which consists of the entries in /etc/ld.so.conf plus /lib and /usr/lib

Then, if we check each of these, we can see that the /mnt/apps/lib path cannot be find in any of them

bash-4.4# echo $LD_LIBRARY_PATH 
<empty>

x@ubuntu: objdump -x mnt/apps/bin/httpd |grep -A 2 -B 2 RPATH
  NEEDED               libgcc_s.so.1
  NEEDED               libcontainers.so
  RPATH                
  INIT                 0x00016fe4
  FINI                 0x0006b8dc

A solution for this is to set the LD_LIBRARY_PATH environment variable to look also in the mnt/apps/lib (and private and public) folders. By doing so, we can see that the error message changes – but this is because we didn’t copy yet all the libraries.

bash-4.4# ./qemu-arm-static mnt/apps/bin/httpd 
mnt/apps/bin/httpd: error while loading shared libraries: libcms_dal.so: cannot open shared object file: No such file or directory
bash-4.4# export LD_LIBRARY_PATH=/mnt/apps/lib/private/
bash-4.4# ./qemu-arm-static mnt/apps/bin/httpd         
mnt/apps/bin/httpd: error while loading shared libraries: libcms_msg.so: cannot open shared object file: No such file or directory

After we copy all shared libraries from the router (/lib and /mnt/apps/lib), and place them in the same folder structure within our chroot, we can finish setting up the LD_LIBRARY_PATH

x@ubuntu: tree 
<-- redacted -->
49 directories, 310 files

x@ubuntu: sudo chroot `pwd` ./bin/bash
bash-4.4# export LD_LIBRARY_PATH=/mnt/apps/lib/private:/mnt/apps/lib/public:/mnt/apps/lib/
bash-4.4# ./qemu-arm-static mnt/apps/bin/httpd -h
mnt/apps/bin/httpd: invalid option -- 'h'
mnt/apps/bin/httpd:error:667.627:main:374:bad arguments, exit
usage: mnt/apps/bin/httpd [-o] [-p port_num] [-m shmid] [-v num] [-x secs]
       o: open httpd server socket, normally server socket is inheritted from smd
       p: TCP port number for httpd to listen on
       m: shared memory id, -1 if standalone or not using shared mem.
       v: set verbosity, where num==0 is LOG_ERROR, 1 is LOG_NOTICE, all others is LOG_DEBUG
       x: set exit on idle timeout, in seconds.

7. Automate the LD_LIBRARY_PATH set

Instead of getting an interactive bash shell into the chroot and setting the LD_LIBRARY_PATH everytime, we can automate the process and set the variable transparently everytime we run the following script

export LD_LIBRARY_PATH=/mnt/apps/lib:/mnt/apps/lib/private:/mnt/apps/lib/public:/lib
./qemu-arm-static ./mnt/apps/bin/httpd $1

We save the script into a file (run.sh) next to the qemu-arm-static

x@ubuntu: tree -L 1
.
├── bin
├── etc
├── lib
├── lib64
├── mnt
├── qemu-arm-static
└── run.sh

5 directories, 2 files

And then we execute the httpd with arbitrary parameters

x@ubuntu: sudo chroot `pwd` ./bin/bash run.sh -h
./mnt/apps/bin/httpd: invalid option -- 'h'
./mnt/apps/bin/httpd:error:61.106:main:374:bad arguments, exit
usage: ./mnt/apps/bin/httpd [-o] [-p port_num] [-m shmid] [-v num] [-x secs]
       o: open httpd server socket, normally server socket is inheritted from smd
       p: TCP port number for httpd to listen on
       m: shared memory id, -1 if standalone or not using shared mem.
       v: set verbosity, where num==0 is LOG_ERROR, 1 is LOG_NOTICE, all others is LOG_DEBUG
       x: set exit on idle timeout, in seconds.



sudo chroot `pwd` ./bin/bash run.sh "-v 3"
./mnt/apps/bin/httpd:debug:92.440:main:381:using http serverPort=80 exitOnIdleTimeout=3600
./mnt/apps/bin/httpd:debug:92.445:main:382:using shmId=-1
./mnt/apps/bin/httpd:error:92.446:oalMsg_initWithFlags:115:connect to /var/smd_messaging_server_addr failed, rc=-1 errno=2
./mnt/apps/bin/httpd:error:92.446:main:386:cmsMsg_init failed, ret=9002

Published by Andrei Agape

One thought on “Running ARM binaries on Ubuntu x86 with QEMU”

  1. Hello,

    Your blog post was very helpful, so thank you. 🙂

    Im trying to reverse the httpd binary on a NowTV router popular in the UK. I actually get the same error you showed in your last paste which is:

    ./mnt/apps/bin/httpd:error:92.446:oalMsg_initWithFlags:115:connect to /var/smd_messaging_server_addr failed, rc=-1 errno=2
    ./mnt/apps/bin/httpd:error:92.446:main:386:cmsMsg_init failed, ret=9002

    Was you ever able to find why this happens and actually get httpd to run to the extent the web interface can be viewed?

    Many thanks,

    Like

    Reply

Leave a Reply

Fill in your details below or click an icon to log in:

Gravatar
WordPress.com Logo

You are commenting using your WordPress.com account. ( Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. ( Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. ( Log Out /  Change )

Cancel

Connecting to %s