Welcome to this issue of Activation Function. Every other week, I introduce you to a new and interesting open-source backend technology (that you’ve probably only kind of heard about… ) and explain it to you in 5 minutes or less so you can make better technical decisions moving forward. In this issue, we’ll explore eBPF (Extended Berkeley Packet Filter), an exciting new technology that makes the kernel flexible, safe, and accessible to developers. programming TLDR eBPF is a mechanism that makes the kernel dynamically programmable without modifying the source code. eBPF is safe, fast, incredibly flexible, and extensible. eBPF has been running in production for over half a decade at an internet scale on millions of servers. eBPF use cases range from observability, networking, security, tracing, and profiling. eBPF is now a standalone term that doesn’t stand for anything. You'll see the term BPF in Linux source code, and you'll see BPF and eBPF used interchangeably in tooling and documentation. The original BPF is sometimes called cBPF (classic BPF) to distinguish it from eBPF. [IMPORTANT NOTE] Why program the Kernel? The kernel can oversee and control the entire system, which, on the one hand, makes it the ideal place to implement networking, security, and observability capabilities and, on the other hand, makes it very risky to fiddle with. As a result, innovation at the kernel level has been super slow! After all, as the saying goes, With great power comes great responsibility. Up until recently, if you wanted to add functionality to the kernel, you had two options: Try to change the kernel's source code and convince the community that the change is required, which, as you can imagine, took ages. can be risky on many levels… (security, performance, stability, compatibility, and the list goes on.) and costly to maintain as every kernel version upgrade can break them. Load kernel modules (LKMs) Enters eBPF ‍ eBPF is a mechanism that makes the kernel dynamically programmable, kind of like how JavaScript lets you dynamically change the behavior of a webpage. Here is a great high-level explanation : by Brendan Gregg eBPF does to Linux what JavaScript does to HTML. (Sort of.) So, instead of a static HTML website, JavaScript lets you define mini-programs that run on events like mouse clicks, which are run in a safe virtual machine in the browser. And with eBPF, instead of a fixed kernel, you can now write mini-programs that run on events like disk I/O, which are run in a safe virtual machine in the kernel. In reality, eBPF is more like the v8 virtual machine that runs JavaScript rather than JavaScript itself. eBPF is part of the Linux kernel. Although eBPF is far from completely replacing LKMs, it sets itself apart by bringing great flexibility while mitigating risk by putting solid safety and controls in place. How we got here? 1992 – wanted to troubleshoot network issues, but existing network filters were too slow. He and his team developed BPF (Berkeley Packet Filter) to be fast, efficient, and easily verifiable to run in the kernel safely. Van Jacobson BPF was a great technology, but it had a few limitations that became apparent over the years as networking technology evolved. Among other things: It wasn't adapted to modern processors and multi-processor systems. It was stateless, which made it a bad fit for complex packet operations. It took a lot of work to extend for developers. 2014 – introduced the extended BPF (eBPF) design that took things to a whole new level by: Alexei Starovoitov Overhauling the BPF instruction set to take advantage of modern hardware. Introducing functions that eBPF programs can call to interact with the system. Helper Introducing the so that user space programs can interact with eBPF programs in the kernel. bpf() system call Introducing the eBPF which ensures that an eBPF program is loaded only if it’s safe to run. verifier Moving beyond packet filtering and opening the door for many use cases around networking, observability, security, tracing, and profiling. Today, eBPF is a general-purpose compute engine within the Linux kernel that allows you to hook into, observe, and act upon anything happening in the kernel. Check out by Alexei Starovoitov for an in-depth history of eBPF. this talk How does eBPF work? Before we delve into this, it’s essential to understand the difference between the kernel and user space in Linux. Here is a quick rundown: The Linux kernel is the software layer that sits between your applications and the hardware they run on in a layer called the user space. The user space is therefore, it can’t access the hardware directly. unprivileged; When an application requires something from the hardware, it will need to request the kernel, which is to do it on its behalf using the system call (syscall) interface. privileged, The kernel then relays the request to the hardware, coordinating concurrent requests and ensuring everything runs smoothly. Alright, back to eBPF. Without going the rabbit hole, here is how it works on a high level: Step #1 | Program Development You can write your own eBPF program using a tool like that provides an easy-to-learn high-level language or the . The program is then compiled into bytecode. bpftrace BPF Compiler Collection (BCC) Python framework As a beginner, you don’t need to write eBPF code from scratch, as BCC comes with over 70 tools you can use out of the box. Here is a glimpse of what you have at your disposal: [Note] Source — https://www.brendangregg.com/ ​ Step #2 | Program Verification The bytecode runs through the eBPF verifier inside a VM to ensure it will not harm the system before being loaded into the kernel. The verification process is quite complex. You can read more about it . Although much work has gone into improving and simplifying it, you can still run into strange errors when developing your program. If you need help, check out the . [Note] here eBPF Slack community channel Step #3 | Program Attachment The verified program is loaded into the kernel and attached to predefined hook points before being further JIT compiled at runtime into native machine instructions to ensure maximum performance. Step #4 | Program Execution The program is triggered on predefined events, and helper functions are called. Maps are then used to pass data between the kernel and user space or other eBPF functions and to maintain the state. eBPF program becomes active when loaded into the kernel. You don’t need to reboot the machine, restart existing processes, or change anything about other applications. [Note] eBPF in production Since its inception in 2014, eBPF capabilities have continued to grow, supported by 300+ kernel developers and major tech players, including Netflix, Meta, Google, Cloudflare, DoorDash, and many others, running eBPF-based tools in production for over half a decade 24/7 at internet scale on millions of servers. Let’s look at some examples: for Infrastructure Observability. LinkedIn uses eBPF sidecar called Flow Exporter, which uses eBPF tracepoints to capture TCP flows in near real-time. Netflix has developed a network observability , performance monitoring, and network observability. Cloudflare uses eBPF for network security for kernel security monitoring. Apple uses eBPF every packet coming into their data centers. Meta uses eBPF to process and load balance for application monitoring. DoorDash uses eBPF and use eBPF for GPU performance monitoring. Digital Ocean Cruise And the list . goes on eBPf limitations No technology is perfect, and eBPF isn’t an exception. Let’s discuss a few current limitations you should be aware of: eBPF was initially released in a limited capacity in 2014 with Linux 3.18. You need at least Linux 4.4 or above to use eBPF fully. between kernel versions and distributions is still not 100% there. Despite much work, eBPF portability eBPF is still a pretty complex technology that isn’t easy to grasp for the average developer. Anyone working with eBPF will need a solid knowledge of networking and kernel inner workings. eBPF is still in the early stages of expanding to other OS ecosystems, with Windows leading the charge . Despite significant efforts by the community and large companies like Google to secure eBPF, it’s still vulnerable . to cyber-attacks eBPF-based projects Despite a lot of effort put in by the community to make eBPF more accessible, the reality is that it’s still a pretty complex technology to work with for the majority of developers. The good news is that if you want to leverage the power of eBPF, there are a growing number of projects that can help you do that without writing eBPF programs: ​​ is a behavioral activity monitor designed to detect anomalous activity in applications. Falco provides eBPF-based transparent security observability combined with real-time runtime enforcement. Tetragon helps you track memory, CPU, and I/O bottlenecks broken down by method name, class name, and line number over time. Parca is an open-source project that provides eBPF-powered networking, security, and observability. Cilium Open Source is designed to simplify, scale, and secure container and Kubernetes networks. Calico is a fully distributed networking and security observability platform for cloud-native workloads. Hubble is an eBPF-based tool for tracing network packets in the Linux kernel with advanced filtering capabilities. pwru is an open-source observability tool for Kubernetes applications. Pixie And the . list goes on Where to next? If eBPF sounds like your cup of tea, and you’re interested in exploring further, you’re in luck, as many great free resources are available. Here are a few: Documentation BPF Documentation BPF & XDP Reference Guide BPF Design Q&A Articles & Blogs A thorough introduction to eBPF Brendan Gregg's blog eBPF’s official community website eBPF - From a Programmer’s Perspective The Beginner's Guide to eBPF Tutorials . This tutorial is about developing tools and programs using the Python interface bcc Python Developer Tutorial bcc : Tutorial and Examples. Learn eBPF Tracing - Learn bpftrace for Linux in 12 easy lessons, where each lesson is a one-liner you can try running The bpftrace One-Liner Tutorial Books Learning eBPF What is eBPF? Linux Observability with BPF Security Observability with eBPF Videos & talks eBPF - Rethinking the Linux Kernel Beginner's Guide to eBPF Programming for Networking eBPF - The Future Of Isolated/Malware Analysis Communities eBPF Slack community channel eBPF Stack Overflow eBPF reddit community People to follow Brendan Gregg Liz Rice Alexei Starovoitov David S. Miller That’s it for today. I hope this gave you a good idea of what eBPF is and how you can use it for your next project. There is still SO MUCH to unpack when it comes to eBPF, so I encourage you to go out and explore! Until next time! Also published . here

Too Long; Didn't Read

Making security fun one episode at a time

A 5-min Intro to Programming the Kernel with eBPF

Too Long; Didn't Read

Karim Traiaia

About Author

TOPICS

THIS ARTICLE WAS FEATURED IN...

A 5-min Intro to Programming the Kernel with eBPF

Too Long; Didn't Read

Karim Traiaia

About Author

TOPICS

THIS ARTICLE WAS FEATURED IN...

RELATED STORIES