Sysdig: What It Is and How to Use It

Sysdig is a universal system visibility tool with support for containers. What makes Sysdig special, is that it hooks itself into the machine's kernel and segregates the information on a per-container basis. For the scope of this tutorial, we will focus on the open-source version of Sysdig. In the next sections, you will: Install Sysdig Spin up a Wordpress installation using docker-compose Use Sysdig to collect events and analyze them at a later time Use Sysdig to analyze data in real-time Prerequisites Docker is installed on your system. For details about installing Docker, refer to the page. Install Docker Docker Compose is installed on your system. Refer to the page for instructions about how to install Docker Compose. Install Docker Compose The kernel headers are installed on the host system. Install Sysdig Follow these steps to install Sysdig inside a Docker container: 1. In a terminal window, execute the following command to pull the Sysdig Docker image: docker pull sysdig/sysdig Using default tag: latest latest: Pulling from sysdig/sysdig 2967486b0658: Pull complete 78101b780c72: Pull complete 7e78b657334d: Pull complete 650327159ca8: Pull complete 47ebf73ab754: Pull complete bf51ac76a6d9: Pull complete 0cd11104dbf6: Pull complete e6dcf17d00d8: Pull complete 230d60083576: Pull complete fd5ea9faf384: Pull complete 6de86c8ed6e9: Pull complete 8d1825f8be4b: Pull complete Digest: sha256:bbfe6953fd2b3221a8974eb13024dd33c7e78aebef8fee3d7a0d9ecdeed84ce0 Status: Downloaded newer image sysdig/sysdig:latest for 2. Run Sysdig in a container by entering: docker run -i -t --name sysdig --privileged -v /var/run/docker.sock:/host/var/run/docker.sock -v /dev:/host/dev -v /proc:/host/proc:ro -v /boot:/host/boot:ro -v /lib/modules:/host/lib/modules:ro -v /usr:/host/usr:ro sysdig/sysdig * Setting up /usr/src links from host * Unloading sysdig-probe, present * Running dkms install sysdig Error! Your kernel headers kernel 3.10.0-957.12.2.el7.x86_64 cannot be found at /lib/modules/3.10.0-957.12.2.el7.x86_64/build or /lib/modules/3.10.0-957.12.2.el7.x86_64/ . * Running dkms build failed, couldn if for echo for source 't find /var/lib/dkms/sysdig/0.26.4/build/make.log * Trying to load a system sysdig-probe, if present * Trying to find precompiled sysdig-probe for 3.10.0-957.12.2.el7.x86_64 Found kernel config at /host/boot/config-3.10.0-957.12.2.el7.x86_64 * Trying to download precompiled module from https://s3.amazonaws.com/download.draios.com/stable/sysdig-probe-binaries/sysdig-probe-0.26.4-x86_64-3.10.0-957.12.2.el7.x86_64-82e2ae1fb159132636f7b50a762f20ef.ko Download succeeded, loading module root@7b14a23f22eb:/# A few things to note about the above command: The flag keeps STDIN open. -i The parameter provides Also it sets SELinux to allow the processes running inside of the container the as a process running on the host. --privileged access to all devices on the host. same access to the host The flag specifies the list of files ( ) that Sysdig can access. -v on the host Spin Up a Wordpress Installation In this section, you will install Wordpress using the command. docker-compose 1. In a , move into your projects directory and type the following commands: new terminal window mkdir wordpress-sysdig && wordpress-sysdig cd 2. Create a file called with the following content: docker-compose version: services: db: image: mysql:5.7 volumes: - db_data:/var/lib/mysql restart: always environment: MYSQL_ROOT_PASSWORD: somewordpress MYSQL_DATABASE: wordpress MYSQL_USER: wordpress MYSQL_PASSWORD: wordpress wordpress: depends_on: - db image: wordpress:latest ports: - restart: always environment: WORDPRESS_DB_HOST: db:3306 WORDPRESS_DB_USER: wordpress WORDPRESS_DB_PASSWORD: wordpress WORDPRESS_DB_NAME: wordpress volumes: db_data: {} '3.3' "8000:80" 3. Run the command in detached mode with: docker-compose up docker-compose up -d Creating network with the default driver Creating volume with default driver Pulling wordpress (wordpress:latest)... latest: Pulling from library/wordpress 8ec398bc0356: Pull complete 85cf4fc86478: Pull complete 970dadf4ccb6: Pull complete 8c04561117a4: Pull complete d6b7434b63a2: Pull complete 83d8859e9744: Pull complete 9c3d824d0ad5: Pull complete 9e316fd5b3b3: Pull complete 578b40496c37: Pull complete 814ae7711d3c: Pull complete 4896fed78b6b: Pull complete e74d71e9611d: Pull complete 46017765526c: Pull complete 280386098458: Pull complete f32eb0d8c540: Pull complete 5c47b9ea747a: Pull complete ecda5b7aad12: Pull complete 84256a6b6b44: Pull complete 35d4f385efb7: Pull complete bf697c2ae701: Pull complete d054b015f084: Pull complete Digest: sha256:73e8d8adf491c7a358ff94c74c8ebe35cb5f8857e249eb8ce6062b8576a01465 Status: Downloaded newer image wordpress:latest Creating wordpress-sysdig_db_1 ... Creating wordpress-sysdig_wordpress_1 ... "wordpress-sysdig_default" "wordpress-sysdig_db_data" for done done 4. You can verify the status of your containers with: docker ps If all is going well, you should see something similar to the following output: CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES f390eec29f52 wordpress:latest About a minute ago Up About a minute 0.0.0.0:8000->80/tcp wordpress-sysdig_wordpress_1 a844840626d8 mysql:5.7 About a minute ago Up About a minute 3306/tcp, 33060/tcp wordpress-sysdig_db_1 7b14a23f22eb sysdig/sysdig 13 minutes ago Up 13 minutes sysdig "docker-entrypoint.s…" "docker-entrypoint.s…" "/docker-entrypoint.…" 5. Now Wordpress is up and running. Point your browser to to start the installation wizard: http://localhost:8000 6. Once the installation wizard is finished, let us go ahead and create a sample post: Collecting Data to a File In this section, we'll show how you can use Sysdig to collect events and analyze them at a later time. 1. To dump all captured events to a file, move to the Sysdig container, and enter the following command: sysdig -w monitoring-wordpress.scap 2. In a new terminal window, use to make 10000 requests with a maximum of 100 requests running concurrently: ab ab -n 1000 -c 100 http://localhost:8000/?p=7 This is ApacheBench, Version 2.3 Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/ Licensed to The Apache Software Foundation, http://www.apache.org/ Benchmarking localhost (be patient) Completed 100 requests Completed 200 requests Completed 300 requests Completed 400 requests Completed 500 requests Completed 600 requests Completed 700 requests Completed 800 requests Completed 900 requests Completed 1000 requests Finished 1000 requests $Revision Note that the above output was truncated for brevity. 3. Move back to tour Sysdig container and stop capturing data by entering "CTRL+C". Analyzing data Now, if you look at the size of the file, you'll notice that Sysdig captured no less than 80M of data: monitoring-wordpress.scap ls -lh monitoring-wordpress.scap -rw-r--r--. 1 root root 80M Jan 7 16:28 monitoring-wordpress.scap To find your way through this mountain of data, you'll use something called a . chisel A chisel is basically a Lua script that analyzes the event stream and performs useful actions. You can run the following command to display the list of chisels: sysdig -cl Category: Application --------------------- httplog HTTP requests httptop Top HTTP requests memcachelog memcached requests Category: CPU Usage ------------------- spectrogram Visualize OS latency real time. subsecoffset Visualize subsecond offset execution time. topcontainers_cpu Top containers by CPU usage topprocs_cpu Top processes by CPU usage Category: Errors ---------------- topcontainers_error Top containers by number of errors topfiles_errors Top files by number of errors topprocs_errors top processes by number of errors log log in Note that the above output was truncated for brevity. To retrieve detailed information about a chisel, run the command followed by the flag and the name of the chisel, as in the following example: sysdig -i sysdig -i httptop Category: Application --------------------- httptop Top HTTP requests Show top HTTP requests by: ncalls, time or bytes Args: [string] by - Show top HTTP transactions by: ncalls, time or by tes, default is ncalls Continuing our example, here's how you can use the chisel to display the top HTTP requests: httptop sysdig -r monitoring-wordpress.scap -c httptop ncalls method url -------------------------------------------------------------------------------- 2001 GET localhost:8000/?p=7 14 OPTIONS * 2 GET localhost:8000/favicon.ico 1 GET /wp-content/themes/twentytwenty/assets/fonts/inter/Inter-upright-var.woff2 1 GET localhost/v1.24/containers/6bd8418eb03f/json 1 GET localhost/v1.24/containers/06def7875617/json 1 GET /v1.24/images/1b1624b63467ec61fab209b6be6e79707ae786df86607b9474b246acd31600 1 GET /v1.24/images/db39680b63ac47a1d989da7b742f7b382af34d85a68214f8977bad59c05901 1 GET localhost:8000/ You can see the same information in a format with the flag: container-friendly -pcontainer sysdig -r monitoring-wordpress.scap -c httptop -pcontainer ncalls container method url -------------------------------------------------------------------------------- 1000 wordpress-sysdig_wo GET localhost:8000/?p=7 1000 host GET localhost:8000/?p=7 43 wordpress-sysdig_wo OPTIONS * 1 sysdig GET /v1.24/images/1b1624b63467ec61fab209b6be6e79707ae786df86607b9474b246acd31600 1 sysdig GET localhost/v1.24/containers/06def7875617/json 1 sysdig GET localhost/v1.24/containers/cd06093b141b/json 1 sysdig GET /v1.24/images/00e230fe24da9067f9b6e65cfbe9935a5affac1ae8e44edb6a5b0ccc26374d 1 sysdig GET /v1.24/images/db39680b63ac47a1d989da7b742f7b382af34d85a68214f8977bad59c05901 Digging Deeper Sysdig captures content-rich information that lets you get detailed insights into the inner-workings of your containers. Let's suppose you're running a few containers and want to know which process consumes the most CPU. 1. List the containers that were active during the period in which you captured events: sysdig -r monitoring-wordpress.scap -c lscontainers 2. You can identify the container that consumed the most CPU with: sysdig -r monitoring-wordpress.scap -c topcontainers_cpu CPU% container.name -------------------------------------------------------------------------------- 5.37% wordpress-sysdig_wordpress_1 1.35% wordpress-sysdig_db_1 0.84% host 0.51% sysdig 3. You can dig even deeper and identify the most CPU intensive process with the chisel: topprocs_cpu sysdig -r monitoring-wordpress.scap -c topprocs_cpu container.name contains wordpress_1 CPU% Process PID -------------------------------------------------------------------------------- 0.12% apache2 8383 0.11% apache2 9413 0.11% apache2 9300 0.11% apache2 9242 0.11% apache2 8897 0.11% apache2 8422 0.10% apache2 9372 0.10% apache2 9241 0.10% apache2 8424 0.09% apache2 9429 If you want to see more details, the chisel provides a more verbose alternative: ps sysdig -r monitoring-wordpress.scap -c ps container.name=wordpress-sysdig_wordpress_1 TID PID USER VIRT RES FDLIMIT CMD 5896 5896 root 232.82M 22.32M 429496729 apache2 8383 8383 www-data 307.44M 25.46M 429496729 apache2 8422 8422 www-data 235.44M 22.90M 429496729 apache2 8424 8424 www-data 307.44M 25.46M 429496729 apache2 8897 8897 www-data 235.44M 22.89M 429496729 apache2 9154 9154 www-data 235.44M 22.91M 429496729 apache2 9241 9241 www-data 307.44M 25.66M 429496729 apache2 9242 9242 www-data 307.44M 25.67M 429496729 apache2 9300 9300 www-data 235.44M 22.89M 429496729 apache2 9372 9372 www-data 235.44M 22.89M 429496729 apache2 9413 9413 www-data 233.44M 20.77M 429496729 apache2 Tips Useful If you run Sysdig to capture events as in the above example ( ), the event file will grow continuously until it consumes all the available space. There are a few methods that can help prevent this from happening: sysdig -w monitoring-wordpress.scap Specify the number of events Sysdig should capture by passing it the flag. Once Sysdig captures the specified number of events, it'll automatically exit: -n sysdig -n 5000 -w monitoring-wordpress.scap Use the flag to configure Sysdig so that it breaks the capture into smaller files of a specified size. The following example continuously saves events to files < 10MB: -C sysdig -C 10 -w monitoring-wordpress.scap This will create a bunch of files no larger than 10 MB: ls -lh monitoring-wordpress* -rw-r--r--. 1 root root 9.6M Jan 7 17:13 monitoring-wordpress.scap0 -rw-r--r--. 1 root root 9.6M Jan 7 17:14 monitoring-wordpress.scap1 -rw-r--r--. 1 root root 9.6M Jan 7 17:14 monitoring-wordpress.scap2 -rw-r--r--. 1 root root 9.6M Jan 7 17:14 monitoring-wordpress.scap3 -rw-r--r--. 1 root root 9.6M Jan 7 17:14 monitoring-wordpress.scap4 -rw-r--r--. 1 root root 9.6M Jan 7 17:14 monitoring-wordpress.scap5 -rw-r--r--. 1 root root 9.6M Jan 7 17:14 monitoring-wordpress.scap6 -rw-r--r--. 1 root root 9.6M Jan 7 17:14 monitoring-wordpress.scap7 -rw-r--r--. 1 root root 6.4M Jan 7 17:14 monitoring-wordpress.scap8 Specify the maximum number of files Sysdig should keep with the flag. For example, you can combine the and flags like so: -W -C -W sysdig -C 10 -W 4 -w monitoring-wordpress.scap The above command will only keep the last four capture files: ls -lh monitoring-wordpress* -rw-r--r--. 1 root root 7.2M Jan 7 17:21 monitoring-wordpress.scap0 -rw-r--r--. 1 root root 9.6M Jan 7 17:21 monitoring-wordpress.scap1 -rw-r--r--. 1 root root 9.6M Jan 7 17:21 monitoring-wordpress.scap2 -rw-r--r--. 1 root root 9.6M Jan 7 17:21 monitoring-wordpress.scap3 root@cd06093b141b:/ # sysdig -C 10 -W 4 -w monitoring-wordpress.scap Real-Time Monitoring With Sysdig, you can also analyze data real-time. At first glance, this can seem like a daunting task because, by default, all events are continuously printed out to the console. Fortunately, chisels are here to help. Let's take an example. Analyze Processes on a Per Container Basis 1. Run the following command to list your containers: docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 5b253e74e8e7 sysdig/sysdig 9 minutes ago Up 9 minutes sysdig 06def7875617 wordpress:latest 3 hours ago Up 3 hours 0.0.0.0:8000->80/tcp wordpress-sysdig_wordpress_1 6bd8418eb03f mysql:5.7 3 hours ago Up 3 hours 3306/tcp, 33060/tcp wordpress-sysdig_db_1 "/docker-entrypoint.…" "docker-entrypoint.s…" "docker-entrypoint.s…" 2. You can analyze the processes running in the WordPress container with: sysdig -pc -c topprocs_cpu container.name=wordpress-sysdig_wordpress_1 3. Similarly, you can analyze the processes running in the MySQL container: sysdig -pc -c topprocs_cpu container.name=wordpress-sysdig_db_1 Note that, not much different from this example, Sysdig can monitor network traffic, disk usage, and so on. In this tutorial, you have gone over the fundamentals of using Sysdig to get a clear understanding of the activity generated by your containers. The examples in this blog post helped you get started and, in future tutorials, we'll show you how to use and . Csysdig Sysdig Inspect