--- myst: html_meta: "description lang=en": "How to use the Sysbox Runtime with Kasm Workspaces to provide containers with elevated privileges in an isolated runtime." "keywords": "Kasm, How to, How-to, Web, Sysbox, Nestybox, Systemd" "property=og:locale": "en_US" --- ```{title} Sysbox Runtime ``` # Sysbox Runtime The default docker container runtime is "runc". [Sysbox](https://github.com/nestybox/sysbox) is an open-source and free container runtime, originally developed by Nestybox, to function as a drop-in replacement container runtime. Sysbox provides improved container isolation and support of VM workloads. Kasm has the option of using Sysbox to allow Workspaces to run as root or with sudo privileges without comprimising the security of the Kasm Agent server. Using Sysbox with Kasm also provides the ability for Workspaces to run system-level software that is not normally possible in a containerized environment, such as systemd, buildx, and Docker. If a Kasm user needs the ability to install software in a Workspace or use sudo (or root access), Sysbox is a recommended method to keep the Kasm Agent secure against container breakout. Please note that [limitations](#limitations-of-kasm-with-sysbox) apply. The following Kasm Workspaces core images, and all variants built on top of those images, support a systemd init methodology: | Container | Source Code | | :----: |--- | | [kasmweb/core-almalinux-8:{{ release }}](https://hub.docker.com/r/kasmweb/core-almalinux-8) | [Dockerfile](https://github.com/kasmtech/workspaces-core-images/blob/develop/dockerfile-kasm-core-oracle) | | [kasmweb/core-almalinux-9:{{ release }}](https://hub.docker.com/r/kasmweb/core-almalinux-9) | [Dockerfile](https://github.com/kasmtech/workspaces-core-images/blob/develop/dockerfile-kasm-core-oracle) | | [kasmweb/core-debian-bullseye:{{ release }}](https://hub.docker.com/r/kasmweb/core-debian-bullseye) | [Dockerfile](https://github.com/kasmtech/workspaces-core-images/blob/develop/dockerfile-kasm-core) | | [kasmweb/core-debian-bookworm:{{ release }}](https://hub.docker.com/r/kasmweb/core-debian-bookworm) | [Dockerfile](https://github.com/kasmtech/workspaces-core-images/blob/develop/dockerfile-kasm-core) | | [kasmweb/core-kali-rolling:{{ release }}](https://hub.docker.com/r/kasmweb/core-kali-rolling) | [Dockerfile](https://github.com/kasmtech/workspaces-core-images/blob/develop/dockerfile-kasm-core) | | [kasmweb/core-opensuse-15:{{ release }}](https://hub.docker.com/r/kasmweb/core-opensuse-15) | [Dockerfile](https://github.com/kasmtech/workspaces-core-images/blob/develop/dockerfile-kasm-core-suse) | | [kasmweb/core-oracle-8:{{ release }}](https://hub.docker.com/r/kasmweb/core-oracle-8) | [Dockerfile](https://github.com/kasmtech/workspaces-core-images/blob/develop/dockerfile-kasm-core-oracle) | | [kasmweb/core-oracle-9:{{ release }}](https://hub.docker.com/r/kasmweb/core-oracle-9) | [Dockerfile](https://github.com/kasmtech/workspaces-core-images/blob/develop/dockerfile-kasm-core-oracle) | | [kasmweb/core-parrotos-6:{{ release }}](https://hub.docker.com/r/kasmweb/core-parrotos-6) | [Dockerfile](https://github.com/kasmtech/workspaces-core-images/blob/develop/dockerfile-kasm-core) | | [kasmweb/core-rockylinux-8:{{ release }}](https://hub.docker.com/r/kasmweb/core-rockylinux-8) | [Dockerfile](https://github.com/kasmtech/workspaces-core-images/blob/develop/dockerfile-kasm-core-oracle) | | [kasmweb/core-rockylinux-9:{{ release }}](https://hub.docker.com/r/kasmweb/core-rockylinux-9) | [Dockerfile](https://github.com/kasmtech/workspaces-core-images/blob/develop/dockerfile-kasm-core-oracle) | | [kasmweb/core-ubuntu-focal:{{ release }}](https://hub.docker.com/r/kasmweb/core-ubuntu-focal) | [Dockerfile](https://github.com/kasmtech/workspaces-core-images/blob/develop/dockerfile-kasm-core) | | [kasmweb/core-ubuntu-jammy:{{ release }}](https://hub.docker.com/r/kasmweb/core-ubuntu-jammy) | [Dockerfile](https://github.com/kasmtech/workspaces-core-images/blob/develop/dockerfile-kasm-core) | This allows any container based off of these core images to be run with the [Sysbox Runtime](https://github.com/nestybox/sysbox) and init with [systemd](https://www.freedesktop.org/wiki/Software/systemd/). When using Sysbox, containers behave much closer to Virtual Machines and allow users in a shared environment to run at root level in their Workspaces with a lower risk of escaping isolation. For additional security enhancements of using Sysbox with your Workspaces check out the [Sysbox User Guide: Security](https://github.com/nestybox/sysbox/blob/master/docs/user-guide/security.md). ```{note} When using the Sysbox Runtime for a Kasm Workspaces session it may take longer for the Workspaces to be ready as compared to launching with native Docker init. This is due to the general overhead of using shiftfs along with a full systemd init chain needing to be run before the KasmVNC service is finally started. ``` ### Limitations of Kasm with Sysbox These features are incompatible with Sysbox: - Persistent profiles - NVIDIA GPU support within Kasm Workspaces - Limited Storage mapping or Volume mapping support. No support between Workspaces using Sysbox and Workspaces not using Sysbox (see [Using Storage Mapping with Sysbox](#using-storage-mapping-with-sysbox) for more information on this) ## Installing Sysbox Before getting started see the list of supported host operating systems [here](https://github.com/nestybox/sysbox/blob/master/docs/distro-compat.md#supported-linux-distros), and the official installation guide [here](https://github.com/nestybox/sysbox/blob/master/docs/user-guide/install-package.md). The purpose of this guide is not to replace the existing Sysbox methodology just review our recommended setup for using the `sysbox-runc` runtime. Currently we recommend using an [Ubuntu Jammy](https://releases.ubuntu.com/jammy/) host running the 6.5 LTS kernel as this will have `shiftfs` integrated with the kernel. If you have a Linux kernel version between 5.15 and 6.3, refer to the [Install shiftfs](#install-shiftfs) steps below. The *minimum* compatible Linux kernel for Sysbox is 5.15. To ensure the necessary Kasm dependencies are installed correctly it is recommended to download and extract the Kasm installer. Once done, navigate to the `kasm_release` directory and run the `install_dependencies.sh` script. This will install all Kasm prerequisits after which you can [Install Sysbox](#install-sysbox-runtime). ```{parsed-literal} cd /tmp curl -O {{ release_url }} tar -xf {{ release_url.split('/') | last }} sudo bash kasm_release/install_dependencies.sh ``` ### Install Sysbox Runtime As mentioned previously this step should be performed after Docker is installed but before Kasm Workspaces is installed. Sysbox can be installed on top of a machine with an active Kasm Workspaces deployment, but we cannot guarantee containers will not be removed or deleted. >```bash > sudo apt-get install jq wget > wget https://downloads.nestybox.com/sysbox/releases/v0.6.4/sysbox-ce_0.6.4-0.linux_amd64.deb > sudo apt-get install ./sysbox-ce_0.6.4-0.linux_amd64.deb >``` Post installation confirm the Docker daemon was configured correctly: >```bash > cat /etc/docker/daemon.json > > { > "runtimes": { > "sysbox-runc": { > "path": "/usr/bin/sysbox-runc" > } > }, > "bip": "172.20.0.1/16", > "default-address-pools": [ > { > "base": "172.25.0.0/16", > "size": 24 > } > ] > } >``` ## Configuring Workspaces to use Sysbox Runtime Any Workspace that you would like to leverage the Sysbox Runtime on will need their runtime set to use systemd for init. This can be achieved with the following `Docker Run Config Override (JSON)`: >```json > { > "runtime": "sysbox-runc", > "entrypoint": [ > "/sbin/init" > ], > "user": 0 > } >``` This setting can be Workspace specific under `Admin > Workspaces > Edit > Docker Run Config Override (JSON)` or set at a group level with `Access Management > Groups > Edit > Settings tab > Add Setting > run_config`. ```{note} When using the Sysbox Runtime the Workspace will lose the ability to leverage an NVIDIA GPU using the [NVIDIA Container Runtime](https://developer.nvidia.com/nvidia-container-runtime). DRI3 using open source drivers will still function in a Sysbox Workspace, for more information on that please see the [DRI3 documentation](../how_to/manual_intel_amd.md). ``` ### Enable sudo for users Now that Sysbox is installed and the Workspace is using it as the designated runtime, it is time to allow sudo for the workspace. Within the Kasm Workspace configuration page, scroll down to the {ref}`docker-exec-config` block and add the below JSON block. This code will run as root when the Workspace first launches, then it will install sudo, and add the kasm-user to the sudoers group. To read more about configuring a Kasm Workspace for sudo using Docker Exec, check the [Installing and Configuring sudo via Docker Exec](../how_to/running_as_root.md#installing-and-configuring-sudo-via-docker-exec) docs. >```json > { > "first_launch":{ > "user":"root", > "cmd":"bash -c '/usr/bin/desktop_ready && apt-get update && apt-get install -y sudo && echo \"kasm-user ALL=(ALL) NOPASSWD: ALL\" >> /etc/sudoers'" > } > } > ``` ## Using Storage Mapping with Sysbox ```{important} When using a Storage Provider or Volume mapping configuration with Sysbox, attaching the same Storage Provider or Volume mapping to a non-Sysbox container **will** cause permission problems between Workspaces. The owner of the files and folders in a Sysbox container will be different than the owner of a non-Sysbox container. Read more below to understand why this happens and why it is important. ``` Using Kasm's Storage Provider and/or Volume mapping with Sysbox introduces some differences in the way this feature normally works (for additional information on why, check out the [Sysbox Documentation](https://github.com/nestybox/sysbox?tab=readme-ov-file#how-it-works) as well as their documentation on [User Namespace ID Mapping](https://github.com/nestybox/sysbox/blob/master/docs/user-guide/security.md#user-namespace-id-mapping)). When Kasm starts a Workspace container normally it runs the container as a user called `kasm-user` inside all of it's Workspaces (refer to our [Building Custom Images](../how_to/building_images.md#basic-build) to see how we do this). This user has a UID (User ID) and GID (Group ID) of `1000`. When Kasm maps Storage or Volume drives to a Workspace the UID/GID of `1000` is used to grant permission to the files and folders in the shared drive. Because of this fact, most of Kasm's [Storage Mapping](../guide/storage_mappings.md#volume-config) documentation uses a config similar to the below JSON: >```json > { > "driver" : "rclone", > "driver_opts" : { > "type" : "s3", > "s3-provider" : "AWS", > "s3-env-auth" : "false", > "s3-region" : "us-east-1", > "uid" : "1000", > "gid" : "1000", > "allow_other" : "true" > } > } > >``` In the above JSON field, the `"uid" : "1000"` and `"gid" : "1000"` values are used to set the permissions on all of the files and folders in the share to correspond with the `kasm-user` UID and GID so the Workspace user has permission to view and modify files and folders in the mapped volume. To understand how Sysbox changes things it is important to get a good handle on how it is remapping UID and GID values and what Kasm does with those for file and folder permissions. Sysbox changes permissions because it remaps the UID and GID values from the host to random, high values with no or very limited permissions on the host. Under the hood, the `sysbox-runc` runtime uses Docker's [User Namespaces](https://docs.docker.com/engine/security/userns-remap/) to handle this remapping. This allows apparent `root` level access inside the container, while associating all container processes to a random high UID/GID value on the host. To see what this looks like on a host you can `cat` the `/etc/passwd` file to see what users exist and to see their associated UID and GID values. >```{code} bash > $ cat /etc/passwd > > ... > opc:x:1000:1000::/home/opc:/bin/sh > ubuntu:x:1001:1001:Ubuntu:/home/ubuntu:/bin/bash > kasm:x:1002:1002::/home/kasm:/bin/sh > sysbox:x:1003:1003::/home/sysbox:/bin/false > ... > >``` On this host, there are 4 (four) users with login permssions and they have the UID values from 1000-1003 and GID values from 1000-1003. To see the Sysbox remapped UID/GID values you can `cat` the `/etc/subuid` and `/etc/subgid` files on your host after installing Sysbox. >```{code} bash > $ cat /etc/subuid > > opc:100000:65536 > ubuntu:165536:65536 > kasm:231072:65536 > sysbox:296608:65536 > >``` >```{code} bash > $ cat /etc/subgid > > opc:100000:65536 > ubuntu:165536:65536 > kasm:231072:65536 > sysbox:296608:65536 > >``` ```{note} When using Sysbox CE, the only user responsible for running containers is the `sysbox` user. Thus, the only UID/GID values of importance in the `/etc/subuid` or `/etc/subgid` files are those associated with the `sysbox` user. ``` In these files it is possible to see that Sysbox has remapped the user `sysbox` with a **host** UID of 1003 and **host** GID of 1003 to the random high UID of `296608` and random high GID of `296608` and has given it a total of `65536` additional UID and GID values for any other required users inside its containers. Thus, **from the host perspective**, when a container runs using the `sysbox-runc` runtime, all processes and file or folder permissions will be associated with the UID and GID of `296608` on this host (*this will likely be different on yours*). From the **container** perspective, however, the user is `root` with a UID and GID of `0`. Below we show running a container with the Sysbox runtime and get some information from it and from the host to demonstrate this process. >```{code} bash > ## Run a container using the `sysbox-runc` runtime > ubuntu@kasm-agent:~$ docker run --runtime=sysbox-runc --name=sysbox-test --rm -d alpine tail -f /dev/null > 2a70c22a9c3af66e3f4f76df3de9cf46dafed97ac7831953b0ebc3561eeeda91 > > ## Container view using uid_map > ubuntu@kasm-agent:~$ docker exec sysbox-test cat /proc/self/uid_map > 0 296608 65536 > > ## User info from within the container > ubuntu@kasm-agent:~$ docker exec sysbox-test id > uid=0(root) gid=0(root) groups=0(root),1(bin),2(daemon),3(sys),4(adm),6(disk),10(wheel),11(floppy),20(dialout),26(tape),27(video) > > ## Host process info of `tail` command running inside the container > ubuntu@kasm-agent:~$ ps -fC "tail -f /dev/null" > UID PID PPID C STIME TTY TIME CMD > 296608 171532 171503 0 21:35 ? 00:00:00 tail -f /dev/null > >``` Now that we know the UID and GID of the container `root` user is `296608` we must figure out what the UID and GID is of the `kasm-user` used inside of every Kasm Workspace container. Previously we said that the UID and GID of the `kasm-user` inside a Kasm Workspace is `1000`. So, to figure out how we need to modify the Storage or Volume mapping permissions values, we must take the Sysbox `root` UID of `296608` and add `1000` to it to get the **host** UID of the `kasm-user`.

Get kasm-user UID for Storage/Volume Mapping:

```{math} 296608 + 1000 = 297608 ``` ```{attention} Aside from the kasm-user UID (1000), the values above are the ones the Sysbox service used for my deployment. **Yours will likely be different!** ``` Now that we have the `kasm-user` UID and GID values we can update our Storage or Volume mapping values so that our file shares work appropriately with these users. >```json > { > "driver" : "rclone", > "driver_opts" : { > "type" : "s3", > "s3-provider" : "AWS", > "s3-env-auth" : "false", > "s3-region" : "us-east-1", > "uid" : "297608", > "gid" : "297608", > "allow_other" : "true" > } > } > >``` ## Install shiftfs Application "[shiftfs](https://github.com/toby63/shiftfs-dkms)" is a kernel filesystem for the Linux kernel. It provides easier uid/gid-shifting for containers. Shiftfs is a dependency of Sysbox. Linux kernels 5.15.x and later have shiftfs preinstalled. ### Installing shiftfs While Kernels 5.15-6.3 in Ubuntu Jammy do support shiftfs it is best to install an up to date version for performance reasons. ```bash sudo apt-get update sudo apt-get install -y make dkms git wget git clone -b k5.16 https://github.com/toby63/shiftfs-dkms.git shiftfs-k516 cd shiftfs-k516/ ./update1 sudo make -f Makefile.dkms modinfo shiftfs ``` You should see output similar to: ``` filename: /lib/modules/5.15.0-1031-aws/updates/dkms/shiftfs.ko license: GPL v2 description: id shifting filesystem author: Christian Brauner author: Seth Forshee author: James Bottomley alias: fs-shiftfs ``` At this point shiftfs is ready, but best to reboot the machine before continuing `sudo reboot`.