Monthly Archives: December 2012

Use perf for Android in the x86 emulator

Starting with Jelly Bean, perf is integrated in Android. If you download Google’s source tree you will find perf in external/linux-tools-perf. This post explains what you need to do to use perf in the emulator.

Prerequisites

You need to download the latest Android tree from Google. For instructions on how to do this see How to build a custom Android emulator image.

You will also need the latest kernel version to use perf. perf will not work with kernel 2.6.29 since it does not have the CONFIG_HAVE_PERF_EVENTS option. The arm emulator does not work yet with kernel 3.4, so you will only be able to use perf with the x86 emulator for now. For instructions on how to use the emulator with the latest 3.4 kernel see Get the brand new 3.4 Android kernel working for the x86 emulator.

Once you have your Google tree and kernel compiled and the x86 emulator working, you can move on to the next steps.

Add needed kernel configuration options

One option you must enable in the kernel config is CONFIG_HAVE_PERF_EVENTS. By default this is enabled in the kernel. You can check your generated .config file to see if you have this option enabled:

$ cd ~/workspace/android/goldfish
$ cat .config | grep CONFIG_HAVE_PERF_EVENTS=y
CONFIG_HAVE_PERF_EVENTS=y

If you want to use kernel probes you must have CONFIG_KPROBES enabled. This is also enabled by default.

Unfortunately you cannot use user probes with the 3.4 kernel for the emulator. User probes have been integrated in the kernel starting with version 3.5.

If you want to get debug information from the kernel (and you probably do), you also have to enable CONFIG_DEBUG_INFO. You can do this by running make menuconfig and selecting Kernel hacking -> Compile the kernel with debug info.

After you have set these kernel config options you need to recompile the kernel.

Missing kernel symbols

In order to trace and profile kernel functions you need to have the kernel image installed in a predefined path. perf will look for vmlinux in predefined paths like the current directory, /boot and /lib/modules/3.4.0+/build. The easiest way is to just copy vmlinux from our compiled kernel to /data and run perf from there:

$ adb push vmlinux /data/
$ adb shell
# cd /data

Do not forget do go to /data directory once you connect to the emulator through adb. Otherwise you will get this error:

# perf probe -a schedule
Failed to find path of kernel module.
Kernel symbol 'schedule' not found.
  Error: Failed to add events. (-2)

Show kernel pointers

By default /proc/sys/kernel/kptr_restrict is set to 2. If kptr_restrict is set to 2, kernel pointers are printed as 0’s regardless of privileges. We need to set kptr_restrict to 1 to allow printing kernel pointers for root:

# cat /proc/sys/kernel/kptr_restrict
2
# echo 1 > /proc/sys/kernel/kptr_restrict
# cat /proc/sys/kernel/kptr_restrict
1

Run perf

Now you can use perf in the emulator. For example, you can count the number of read system calls in one second:

# perf probe -a sys_read
Add new event:
  probe:sys_read       (on sys_read)

You can now use it on all perf tools, such as:

	perf record -e probe:sys_read -aR sleep 1

# perf stat -e probe:sys_read -a sleep 1

 Performance counter stats for 'sleep 1':

                22 probe:sys_read                                              

       1.027680611 seconds time elapsed

Debugging your running kernel in Ubuntu

If you try to debug your running kernel in Ubuntu you will notice that you cannot use the full power of tracing and profiling tools. The main problem is that you do not have the debug symbols for the kernel. I will show you how to install the debug symbols for your Ubuntu distribution. I am using Ubuntu 12.04 but this should work on other versions as well.

First you must install the kernel debug symbols for your kernel. You should also install the kernel sources. Then you can use tracing and profiling tools like perf to analyse your running kernel.

Install kernel debug symbols

The packages with debug symbols have a -dbg suffix in Ubuntu. First you should check if you have such a package in your main Ubuntu repository. The name for the Linux kernel debug symbols package is linux-image-$(uname -r)-dbgsym:

$ apt-cache search linux-image-$(uname -r)-dbgsym

If you cannot find this package you need to add the ddebs repository. There are instructions on how to do this on https://wiki.ubuntu.com/DebuggingProgramCrash. Here are the steps, taken from the Ubuntu wiki site:

1. Create /etc/apt/sources.list.d/ddebs.list by running the following line at a terminal:

echo "deb http://ddebs.ubuntu.com $(lsb_release -cs) main restricted universe multiverse" | \
sudo tee -a /etc/apt/sources.list.d/ddebs.list

2. Stable releases (not alphas and betas) require three more lines adding to the same file, which is done by the following terminal command:

echo "deb http://ddebs.ubuntu.com $(lsb_release -cs)-updates main restricted universe multiverse
deb http://ddebs.ubuntu.com $(lsb_release -cs)-security main restricted universe multiverse
deb http://ddebs.ubuntu.com $(lsb_release -cs)-proposed main restricted universe multiverse" | \
sudo tee -a /etc/apt/sources.list.d/ddebs.list

3. Update your package list:

sudo apt-get update

4. You might get the following errors at package update:

W: GPG error: http://ddebs.ubuntu.com precise Release: The following signatures couldn't be verified because the public key is not available: NO_PUBKEY ECDCAD72428D7C01
W: GPG error: http://ddebs.ubuntu.com precise-updates Release: The following signatures couldn't be verified because the public key is not available: NO_PUBKEY ECDCAD72428D7C01
W: GPG error: http://ddebs.ubuntu.com precise-security Release: The following signatures couldn't be verified because the public key is not available: NO_PUBKEY ECDCAD72428D7C01
W: GPG error: http://ddebs.ubuntu.com precise-proposed Release: The following signatures couldn't be verified because the public key is not available: NO_PUBKEY ECDCAD72428D7C01

In this case, you need to install the GPG key:

wget -q http://ddebs.ubuntu.com/dbgsym-release-key.asc
sudo apt-key add dbgsym-release-key.asc

If you run apt-get update now you will no longer get the previous errors.
5. Now you can install the kernel debug symbols:

sudo apt-get install linux-image-$(uname -r)-dbgsym

6. You will find vmlinux in /usr/lib/debug/boot/vmlinux-$(uname -r).

Install kernel sources

The Ubuntu package for the kernel sources is linux-source:

sudo apt-get install linux-source

You will find the installed kernel sources in /usr/src. For example, for the 3.2.0 kernel version, the folder will be named /usr/src/linux-source-3.2.0. The sources are archived in this folder and you need to extract them:

cd /usr/src/linux-source-3.2.0
sudo tar xvfj linux-source-3.2.0.tar.bz2

Create a symbolic link so that you can easily refer the sources in /usr/src/linux-source:

sudo ln -s linux-source-3.2.0/linux-source-3.2.0 linux-source

How can you use this?

One tool you can use to take a look into your running kernel is perf. To get perf in Ubuntu you need to install linux-tools:

$ sudo apt-get install linux-tools-common linux-tools-3.2.0-31

Now you can do some cool stuff with perf, e.g. see the annotated code of the schedule function from the kernel:

$ perf probe -k /usr/lib/debug/boot/vmlinux-3.2.0-31-generic -s /usr/src/linux-source -L schedule

      0  asmlinkage void __sched schedule(void)
      1  {
      2         struct task_struct *tsk = current;
         
      4         sched_submit_work(tsk);
      5         __schedule();
      6  }
         EXPORT_SYMBOL(schedule);
         
         #ifdef CONFIG_MUTEX_SPIN_ON_OWNER

10 things you need to do before sending your patch to LKML

Finally you have your changes ready for the Linux Kernel and you want to share them with the community. For this you need to send them to the Linux Kernel Mailing List (LKML) and go through the review process. There are a lot of rules for sending your patches to this list and going through the process may be confusing for a newbie. This is a check list of things you have to do before sending you patches.

This is a long post so this is a short checklist to summarize it:

  1. subscribe to LKML and follow related discussions
  2. read Documentation/SubmittingPatches and apply what it says
  3. find and download the right git branch or tree for the subsystem you want to submit to
  4. rebase your patches to the latest version of the upstream branch
  5. fix your coding style using scripts/Lindent
  6. generate your patches with git format-patch -n -s --cover-letter --subject-prefix="PATCH" and fill in a description in the cover letter file; mention the git tree you based your work on in the cover letter
  7. check for style errors with scripts/checkpatch.pl.
  8. send the patches to yourself using git send-email and test if they apply cleanly on the upstream branch
  9. get a list of maintainers and main contributors using scripts/get_maintainer.pl
  10. sent the patches to LKML (linux-kernel@vger.kernel.org) and the maintainers using git send-email

1. Subscribe to LKML

It is not necessary subscribe to the list for sending patches, but it is a good practice. You should subscribe to the list and check out the way things are done. You can learn a lot by watching the discussions going on the list. Beware that the list has a lot of traffic (200-300 emails daily), so you will probably not be able to read everything. You can create an email filter to select emails related to the area you want to submit to.

To see what mailing lists are available you can check vger.kernel.org/vger-lists.html. You can find LKML as the linux-kernel mailing list. You can subscribe by having an email conversation with the majordomo. You should also take a look at the other lists. If you find one for the subsystem you want to submit your patch to, you should also subscribe to that.

LKML is archived on various sites and one of them is lkml.org. You should first search for related discussions and patches in the archives; you can find somebody else has submitted a similar patch as yours or that there was already a discussion on this and the decision was different from your approach.

2. Read the kernel documentation on sending patches

The very first thing you need to do is to read the kernel documentation on submitting patches:

Documentation/SubmittingPatches also has a list of things you have to do before submitting your patch. I will not repeat the things mentioned there in this post, so you must read and follow what it says.

3. Get the right branch or git tree

The Linux Kernel is divided into several subsystems and areas of development. Each such area has its own maintainer and most often its own tree or branch. You need to find the tree or branch you need to rebase your patches to. Since the maintainer will be merging your patches, you need to rebase your patches on his branch so that there will be no conflicts.

You can find the maintainers and their trees in the MAINTAINERS file from the Linux Kernel Sources. Beware that some information on the git trees is deprecated. The best thing to do is to check LKML to see what branch other people are rebasing to. You can also check the list of kernel git trees from git.kernel.org.

If you cannot find anything relevant, you can rebase your change to the tip tree or linux-next.

4. Rebase your changes to the latest tip tree

Before submitting your change, make sure you have the latest changes from upstream. Update your git tree using git pull and rebase if necessary.

5. Is your patch whitespace-damaged?

The Linux Kernel has specific rules when it comes to coding style. This is the first thing reviewers will look at and the easiest thing to check and fix. These rules are described in the Linux Kernel Documentation in Documentation/CodingStyle. Even if you read and follow all that says there, you might miss some obvious things anyway. Hopefully, you have a helper tool in the Linux Kernel: Lindent.

Lindent is a script that uses the GNU indent to automatically fix your coding style. You can find it in your kernel sources in scripts/Lindent. When you run it you have to give the file you want indented as parameter. This is what Linus has to say about it in Documentation/CodingStyle:

Now, again, GNU indent has the same brain-dead settings that GNU emacs
has, which is why you need to give it a few command line options.
However, that's not too bad, because even the makers of GNU indent
recognize the authority of K&R (the GNU people aren't evil, they are
just severely misguided in this matter), so you just give indent the
options "-kr -i8" (stands for "K&R, 8 character indents"), or use
"scripts/Lindent", which indents in the latest style.

6. Generate the patches using git format-patch

The patches sent to LKML need to have a specific format which is described in Documentation/SubmittingPatches. Fortunately, git format-patch generates patches with this format so you can do this automatically.
There are several options for git-format which I use:

  • -n will generate numbered patches in [PATCH n/m] format, even with a single patch.
  • -s makes sure you do not forget the Signed-off-by: line.
  • –cover-letter will generate a cover letter file so that you can fill in the description for your entire set of patches. If you only have one patch, you should skip this.
  • –subject-prefix=”PATCH” can be use to change the default “[PATCH]” prefix to something else. This is useful when sending later versions to LKML and you need to use prefixes like “[PATCH v2]”.

This would be a standard git format patch command:

git format-patch -n -s --cover-letter --subject-prefix="PATCH" SHA-1

where SHA-1 is the change right before yours.

If you generated a cover letter, you will have a file 0000-cover-letter.patch. You need to edit it with a subject and description for your changes. It is a good practice to mention the git tree you based your work on in the cover letter.

7. Check coding style using checkpatch

Even if you followed all coding style rules you might still have missed something. Once you have generated your patch you can verify that there are no violation using checkpatch. You can find it in the Linux Kernel in scripts/checkpatch.pl and run it with the patch you generated before. If you find any issues, you should address them in the code, regenerate the patches and run checkpatchagain.

8. Send the patches to yourself using git send-email

Now that you have the patches ready, how do you send them to LKML?

As you may have read in Documentation/SubmittingPatches, there are some strict rules about not having HTML code in the emails you send to the list. Do not manually copy the content of the patch and paste it in your email! Even if you set your email client to use plain text email, this is not enough. The best way is to use git send-email.

The first step is to send the patches to yourself and see if they apply properly. The easiest way to work with git send-email is to create different identities that you can use. I am using my GMail account with git send-email, so this is how my configuration looks like in ~/.gitconfig:

[sendemail "me"]
	smtpserver = smtp.gmail.com
	smtpuser = myuser@gmail.com
	smtpencryption = tls
	smtpserverport = 587
	to = My Name 
	chainreplyto = false
	suppresscc = all

It is a good practice to send all your patches in the same email thread as replies to the cover letter. For this, you need to set chainreplyto = false.
By default, git send-email will send an email to all people who signed-off on the patch. When you are just sending the patches to yourself for testing you do not want that, so you should add supresscc = all in the config.

Send the patches using the above configuration with:

git send-email --identity=me *.patch

Once you received the patches by email you need to check if they apply cleanly. You need to be able to export the emails to your machine. The GMail interface does not offer this feature, so you have to use an email client for this. I use Thunderbird and I can easily export and save the emails. Check out the upstream branch you were working on and apply the saved emails using git am. git am knows to apply entire emails so you do not need to do any changes to the saved messages. If they apply cleanly then you are ready to send your patches upstream. If not, go back and see what you did wrong.

9. Find out who the maintainers are

You are ready to send your patches upstream. But whom do you send them to?
You need to include the LKML (linux-kernel@vger.kernel.org), but you also need to add the maintainers. The maintainers for each subsystem are described in MAINTAINERS, but you can also use the get_maintainer script. You can find this script in the Linux Kernel sources in scripts/get_maintainer.pl. You need to run from the Linux Kernel tree on you patches:

scripts/get_maintainer.pl *.patch

This will give you the list of maintainers and main contributors that you should consider adding to your patch.

10. Send your patches to LKML and the maintainers

For sending the patches to LKML you need do use git send-email and set up a similar configuration as described in step 8. If you do this on a regular basis, you can add the maintainers to .gitconfig. Otherwise, you can just specify them in the git send-email command with --to and --cc.

This is an example for the git send-email configuration from ~/.gitconfig :

[sendemail "lkml"]
	smtpserver = smtp.gmail.com
	smtpuser = myuser@gmail.com
	smtpencryption = tls
	smtpserverport = 587
	chainreplyto = false
	cc = LKML <linux-kernel@vger.kernel.org>
        to = mantainer_1

Then you can actually send the patches using:

git send-email --identity=lkml *.patch

You should verify that all your patches made it to the list. You can check the last 100 messages from the archives. You may need to wait a while, since this is an archive and it will take a little time for your patches to appear. It may take around 10-15 min for your first patch to appear in the archive. If you have more patches you should wait longer.

That’s it! Now you have your first patch on LKML!