What Is a BPF Token?

The BPF Token prevents a process from gaining full root-level access to execute BPF code. The kernel hands out a BPF Token as a special permission slip to run BPF code. Only the privileged process can provide these specific BPF permissions, like loading, creating maps, etc to an unprivileged one to execute code or create maps, etc. Kernel version 6.9 introduced the BPF Token.

Why Do I Want To Hand-Roll a BPF Token Delegation?

Anywhere I searched for “BPF tokens”, I kept getting something like, “BPF tokens let unprivileged containers load eBPF" and I wanted to use it, but all Google searches either led me to kernel commit messages or nearly 100% AI generated blog posts. When I read these blog posts, they didn’t make sense, and I couldn’t find examples of failures someone ran into, or an explanation of why doing it a certain way led to failure, which IMO is critical to understanding the internals.

So I asked myself, can I build the entire token delegation handshake myself in one file and either create a token that I can pass to a process, or learn exactly why I can’t?

What Didn’t Work

Here are three ways I failed when I tried to create a BPF token. I am not providing commands to create these tokens, since an LLM can easily generate them.

  1. I tried to create a BPF token as a normal user (not root) and got EPERM from the kernel. A normal user doesn’t have CAP_BPF/CAP_SYS_ADMIN in their namespace, which is why it fails. To create a BPF token, the user needs either CAP_BPF or CAP_SYS_ADMIN. Most of us think in terms of root versus nonroot, but the kernel checks only capabilities and not the user. To test this, I stayed root, dropped both CAP_BPF and CAP_SYS_ADMIN, and tried to load a BPF object, and I got EPERM, which was what I expected.

  2. Next, I tried to create a token as a privileged user (root), and I got ENOTSUP (not supported) from the kernel. I got ENOTSUP because the BPF token is only for a namespace that does not have CAP_BPF/CAP_SYS_ADMIN. If the current namespace already has that capability, then it doesn’t make sense to request one. The comment in the kernel code literally says, “Creating BPF token in init_user_ns doesn’t make much sense.” https://github.com/torvalds/linux/blob/a38297e3fb012ddfa7ce0321a7e5a8daeb1872b6/kernel/bpf/token.c#L159

  3. The final thing I tried was sudo unshare --user … /mnt/bpffs-host, and I got EPERM again. sudo unshare --user executes the command as “root” on the host and tries to create a new user namespace. After creating it, it runs the command within the new namespace. This newly spun-up namespace will have the user with UID “0”. This UID inside the new namespace is “0,” which is not the same UID on the host. The command it is trying to run is trying to mount the host bpffs at /mnt/bpffs-host. I got EPERM because the guest is trying to mount the host namespace bpffs.

The Mental Model

Before diving into the code, I want to discuss the mental model, as it took me a while to understand, and once I did, there was an aha moment. I am going to use a consulate visa stamp analogy to explain this process.

Imagine you (guest) want to visit a country, and to do that, you have to get a visa (permission slip) from the consulate (host) of the country you are trying to visit.

  1. It is visa interview day, and you (guest) bring a visa application (request permission) to the consulate, because you cannot grant yourself entry into the country without permission from the consulate.
  2. The visa officer (the host) will check your visa application and, if all the necessary documents are there, will approve your visa, which will include how long you can stay and what you can do (work/study/travel), with specific permissions similar to delegate_cmds/maps/progs.
  3. Imagine a scenario where the visa officer (host) tries to issue a visa for themselves when they are already in their own country (init_user_ns), and it wouldn’t make sense, right? Why would they want to do that? That is what the kernel calls ENOTSUP, not EPERM.

TBH, this analogy isn’t perfect, but I am trying to use something relatable.

How I Made It Work

We’ll use one binary (178 lines of code) with 2 roles to demonstrate a process (guest) without CAP_BPF/CAP_SYS_ADMIN that can request permissions from another process (host) to execute BPF code.

For brevity, we won’t execute any actual BPF code in this example, but we’ll demonstrate how we were able to delegate a token.

➜ main ✗ % sudo go run ./cmd/token
[parent] running as host root (uid 0 )
[child]  running as uid 0 inside a new user namespace
[child] gets the socket from parent which is fd 3 and opens an empty bpffs (fd 6 ), sends it to parent for delegation and waits for ack
[parent] gets child's bpffs over socket (fd 5 ), sets delegation (cmds/maps/progs/attachs = any), creates superblock and sends ack
[child] mounts delegated bpffs at /tmp/bpffs-ok (mntfd 7 ) and creates a BPF token (fd 9 )
[child] gets the token fdinfo and the delegated powers are:
pos:    0
flags:  02000002
mnt_id: 2258
ino:    90166278
allowed_cmds:   any
allowed_maps:   any
allowed_progs:  any
allowed_attachs:        any

The output from running the binary.

I am going to break it down into 5 steps.

  1. As I mentioned, the binary has two modes: “parent/host” and “child/guest”. By default, the binary starts in the “parent” (default) mode, then forks itself as a “-child” within a new user and mount namespace. The parent process also hands the child a socket connection to itself, which the child uses to request permissions.

    The forked child will have these caps: CLONE_NEWUSER | CLONE_NEWNS. Initially, I only had CLONE_NEWUSER, so the UID was “0”, but when the code tried to fsopen/mount, I got EPERM because root inside the namespace didn’t have the capability to mount file systems. I needed to include the CLONE_NEWNS flag when cloning the child.

  2. After the parent process forks itself as a child, the first thing the child does is open the bpffs filesystem with unix.Fsopen("bpf", 0). bpffs (BPF filesystem) is a virtual filesystem, similar to tmpfs. It doesn’t correspond to any physical disk and is usually mounted at /sys/fs/bpf. The child gives the file descriptor to the parent, which can write the policy and the superblock. The child shares the file descriptor it created with the parent by encoding the FD and sending it via the socket connection unix.Sendmsg(3, []byte{1}, rights, nil, 0).

    1. The child uses file descriptor 3 to share the bpffs fd with the parent because that is the fd the parent provided for communication. The parent process uses file descriptor 3 because 0, 1, and 2 are used for stdin, stdout, and stderr.

  3. Now that the parent has the message from the child, it delegates permissions. To do that, it first decodes the file descriptor the child shared with it and parses the socket message. Then it converts it into Unix rights and creates its own file descriptor. It needs its own file descriptor for it to assign permissions. If you are wondering why the parent has to create its own file descriptor, it is because each process maintains its own file descriptor table. This means the file descriptor number in the child process is not the same as in the parent process, though they share the same kernel object under the hood.

Parent fd 5 and child fd 6 both refer to the same bpffs struct file

  1. The parent uses its newly created file desciptor from the previous step, assings permission FsconfigSetString(parentFsFd, “delegate_*”, “any”) which is a privildeged operation. Then the parent finalizes the filesystem with FsconfigCreate(parentFsFd) and turns the pending options into a real superblock. The superblock in bpffs has two critical fields:

    • s_user_n, which is the usernamespace that owns this and, in turn, is allowed to create a token here.
    • s_fs_info which has the bpffs specific data like delegate_*

  2. Then the parent sends an ACK via unix.Write(fds[0], []byte{1}), waking the child up so that the child can take over. Now that permission has been given, the child wakes up and mints the BPF token.

I think Incus, under the hood, probably has something like this for delegating permissions, since it’s currently the only container runtime that supports BPF tokens natively: https://linuxcontainers.org/incus/docs/main/explanation/bpf-tokens/

This has been an eye-opening experience for me compared to reading an LLM-generated blog post or copy-pasting LLM-generated code. Now I have somewhat of a better understanding and hope to eventually be able to start comprehending kernel commit emails :)

The source code for the above example is available at https://github.com/naveensrinivasan/tools/blob/c165a7760323b4b4c01dd7d42ca609372076691d/bpftoken/main.go#L22-L178.