Prerequisites

You need to download the latest Android tree from Google. For instructions on how to do this see How to build a custom Android emulator image.

You will also need the latest kernel version to use perf. perf will not work with kernel 2.6.29 since it does not have the CONFIG_HAVE_PERF_EVENTS option. The arm emulator does not work yet with kernel 3.4, so you will only be able to use perf with the x86 emulator for now. For instructions on how to use the emulator with the latest 3.4 kernel see Get the brand new 3.4 Android kernel working for the x86 emulator.

Once you have your Google tree and kernel compiled and the x86 emulator working, you can move on to the next steps.

Add needed kernel configuration options

One option you must enable in the kernel config is CONFIG_HAVE_PERF_EVENTS. By default this is enabled in the kernel. You can check your generated .config file to see if you have this option enabled:



$ cd ~/workspace/android/goldfish

$ cat .config | grep CONFIG_HAVE_PERF_EVENTS=y

CONFIG_HAVE_PERF_EVENTS=y

If you want to use kernel probes you must have CONFIG_KPROBES enabled. This is also enabled by default.

Unfortunately you cannot use user probes with the 3.4 kernel for the emulator. User probes have been integrated in the kernel starting with version 3.5.

If you want to get debug information from the kernel (and you probably do), you also have to enable CONFIG_DEBUG_INFO. You can do this by running make menuconfig and selecting Kernel hacking -> Compile the kernel with debug info.

After you have set these kernel config options you need to recompile the kernel.

Missing kernel symbols

In order to trace and profile kernel functions you need to have the kernel image installed in a predefined path. perf will look for vmlinux in predefined paths like the current directory, /boot and /lib/modules/3.4.0+/build. The easiest way is to just copy vmlinux from our compiled kernel to /data and run perf from there:



$ adb push vmlinux /data/

$ adb shell

# cd /data

Do not forget do go to /data directory once you connect to the emulator through adb. Otherwise you will get this error:

# perf probe -a schedule Failed to find path of kernel module. Kernel symbol 'schedule' not found. Error: Failed to add events. (-2)

Show kernel pointers

By default /proc/sys/kernel/kptr_restrict is set to 2. If kptr_restrict is set to 2, kernel pointers are printed as 0’s regardless of privileges. We need to set kptr_restrict to 1 to allow printing kernel pointers for root:



# cat /proc/sys/kernel/kptr_restrict

2

# echo 1 > /proc/sys/kernel/kptr_restrict

# cat /proc/sys/kernel/kptr_restrict

1

Run perf

Now you can use perf in the emulator. For example, you can count the number of read system calls in one second:

# perf probe -a sys_read Add new event: probe:sys_read (on sys_read) You can now use it on all perf tools, such as: perf record -e probe:sys_read -aR sleep 1 # perf stat -e probe:sys_read -a sleep 1 Performance counter stats for 'sleep 1': 22 probe:sys_read 1.027680611 seconds time elapsed

If you try to debug your running kernel in Ubuntu you will notice that you cannot use the full power of tracing and profiling tools. The main problem is that you do not have the debug symbols for the kernel. I will show you how to install the debug symbols for your Ubuntu distribution. I am using Ubuntu 12.04 but this should work on other versions as well.

First you must install the kernel debug symbols for your kernel. You should also install the kernel sources. Then you can use tracing and profiling tools like perf to analyse your running kernel.

Install kernel debug symbols

The packages with debug symbols have a -dbg suffix in Ubuntu. First you should check if you have such a package in your main Ubuntu repository. The name for the Linux kernel debug symbols package is linux-image-$(uname -r)-dbgsym:

$ apt-cache search linux-image-$(uname -r)-dbgsym

If you cannot find this package you need to add the ddebs repository. There are instructions on how to do this on https://wiki.ubuntu.com/DebuggingProgramCrash. Here are the steps, taken from the Ubuntu wiki site:

1. Create /etc/apt/sources.list.d/ddebs.list by running the following line at a terminal:

echo "deb http://ddebs.ubuntu.com $(lsb_release -cs) main restricted universe multiverse" | \
sudo tee -a /etc/apt/sources.list.d/ddebs.list

2. Stable releases (not alphas and betas) require three more lines adding to the same file, which is done by the following terminal command:

echo "deb http://ddebs.ubuntu.com $(lsb_release -cs)-updates main restricted universe multiverse
deb http://ddebs.ubuntu.com $(lsb_release -cs)-security main restricted universe multiverse
deb http://ddebs.ubuntu.com $(lsb_release -cs)-proposed main restricted universe multiverse" | \
sudo tee -a /etc/apt/sources.list.d/ddebs.list

3. Update your package list:

sudo apt-get update

4. You might get the following errors at package update:

W: GPG error: http://ddebs.ubuntu.com precise Release: The following signatures couldn't be verified because the public key is not available: NO_PUBKEY ECDCAD72428D7C01
W: GPG error: http://ddebs.ubuntu.com precise-updates Release: The following signatures couldn't be verified because the public key is not available: NO_PUBKEY ECDCAD72428D7C01
W: GPG error: http://ddebs.ubuntu.com precise-security Release: The following signatures couldn't be verified because the public key is not available: NO_PUBKEY ECDCAD72428D7C01
W: GPG error: http://ddebs.ubuntu.com precise-proposed Release: The following signatures couldn't be verified because the public key is not available: NO_PUBKEY ECDCAD72428D7C01

In this case, you need to install the GPG key:

wget -q http://ddebs.ubuntu.com/dbgsym-release-key.asc
sudo apt-key add dbgsym-release-key.asc

If you run apt-get update now you will no longer get the previous errors.
5. Now you can install the kernel debug symbols:

sudo apt-get install linux-image-$(uname -r)-dbgsym

6. You will find vmlinux in /usr/lib/debug/boot/vmlinux-$(uname -r).

Install kernel sources

The Ubuntu package for the kernel sources is linux-source:

sudo apt-get install linux-source

You will find the installed kernel sources in /usr/src. For example, for the 3.2.0 kernel version, the folder will be named /usr/src/linux-source-3.2.0. The sources are archived in this folder and you need to extract them:

cd /usr/src/linux-source-3.2.0
sudo tar xvfj linux-source-3.2.0.tar.bz2

Create a symbolic link so that you can easily refer the sources in /usr/src/linux-source:

sudo ln -s linux-source-3.2.0/linux-source-3.2.0 linux-source

How can you use this?

One tool you can use to take a look into your running kernel is perf. To get perf in Ubuntu you need to install linux-tools:

$ sudo apt-get install linux-tools-common linux-tools-3.2.0-31

Now you can do some cool stuff with perf, e.g. see the annotated code of the schedule function from the kernel:

$ perf probe -k /usr/lib/debug/boot/vmlinux-3.2.0-31-generic -s /usr/src/linux-source -L schedule

      0  asmlinkage void __sched schedule(void)
      1  {
      2         struct task_struct *tsk = current;
         
      4         sched_submit_work(tsk);
      5         __schedule();
      6  }
         EXPORT_SYMBOL(schedule);
         
         #ifdef CONFIG_MUTEX_SPIN_ON_OWNER

Yet another * programming blog

for * in Linux, Android, Kernel, Open Source

Tag Archives: tracing

Use perf for Android in the x86 emulator