---
myst:
  html_meta:
    "description lang=en": "Manual and automated upgrade process documentation for Kasm Workspaces multi server deployments."
    "keywords": "Kasm, Workspace, Multi-server, Multi, Server, Upgrade"
    "property=og:locale": "en_US"
---
```{title} Multi Server Upgrade
```

# Multi Server Upgrade

```{warning}
When running any upgrade please ensure there are no active Kasm Workspaces sessions running. On large deployments this will likely require a maintenance window.
```

## Manually

Please read through the entire process before getting started.

Issues can be reported on the [Kasm Workspaces Issues Page](https://github.com/kasmtech/workspaces-issues/issues)

```{include} image_warning.md
```

- Stop all Kasm Services on all Servers

```Bash
sudo /opt/kasm/bin/stop
```

### Upgrade Database Server Role

Create a Database Backup

- Ensure Kasm services are stopped on all hosts. Database corruption can occur if Kasm services are still connected to the database when it goes down.

```{note}
  If your database is a standalone remote database refer here for backup instructions. [](../how_to/remote_database.md#backing-up-the-postgresql-server).
```

```Bash
sudo /opt/kasm/bin/stop
```

- Execute the database backup utility

```{parsed-literal}
sudo mkdir -p /opt/kasm/backups/
sudo bash /opt/kasm/{{ previous_release }}/bin/utils/db_backup -f /opt/kasm/backups/kasm_db_backup.tar -p /opt/kasm/{{ previous_release }}/
```

- Verify the presence and location of the database backup

```Bash
sudo ls -al /opt/kasm/backups/kasm_db_backup.tar
```

- Download and extract the new installation media

```{parsed-literal}
cd /tmp/
curl -O {{ release_url }}
tar -xf kasm_release_*.tar.gz
```

```{note}
If you are performing an offline upgrade please ensure that you have downloaded and installed versions of docker and docker compose that meet the updated system requirements see {doc}`System Requirements <../install/system_requirements>` for details.
```

- Get the existing database password for use in the subsequent commands.

```{parsed-literal}
sudo grep " password" /opt/kasm/{{ previous_release }}/conf/app/api.app.config.yaml
```

- Get the existing redis password for use in the subsequent commands.

```{parsed-literal}
sudo grep "redis_password" /opt/kasm/{{ previous_release }}/conf/app/api.app.config.yaml
```

**Perform a clean install**

Perform a clean install of the Database Role Services

```{note}
If your database is a standalone remote database refer here for clean install instructions. [](../how_to/remote_database.md#initializing-postgresql-server).
```

- Install kasm from the release media downloaded in the prior steps.

  - When performing an offline update add these flags `--offline-workspaces <KASM_WORKSPACE_IMAGES_TAR> --offline-service <KASM_SERVICE_IMAGES_TAR>`

```Bash
sudo bash kasm_release/install.sh -S db -D -Q <DATABASE_PASSWORD> -R <REDIS_PASSWORD>
```

### Restore and upgrade the database

Restore and update the database from the prior version

- Ensure all Kasm services are stopped on all hosts

```{note}
If your database is a standalone remote database refer here for restoration instructions. [](../how_to/remote_database.md#restoring-the-postgresql-server-from-a-backup).
```

```Bash
sudo /opt/kasm/bin/stop
```

- Execute the database restore command

```{parsed-literal}
sudo /opt/kasm/{{ release }}/bin/utils/db_restore -f /opt/kasm/backups/kasm_db_backup.tar -p  /opt/kasm/{{ release }}/
```

- Perform an upgrade of the database schema

```{note}
If your database is a standalone remote database refer here for upgrade instructions. [](../how_to/remote_database.md).
```

```{parsed-literal}
sudo /opt/kasm/{{ release }}/bin/utils/db_upgrade -p /opt/kasm/{{ release }}
```

- **Advanced**: Review the contents of the existing configs to ensure any custom docker settings are migrated to the new configuration.

```{parsed-literal}
diff /opt/kasm/{{ previous_release }}/docker/docker-compose.yaml  /opt/kasm/{{ release }}/docker/docker-compose.yaml
```

- Install The Kasm {{ release }} Default Images for your platform

  - If doing an offline upgrade first extract the default image seed data from the workspace images tar

    For x86 (amd64) platforms:

    ```{parsed-literal}
    tar xf <KASM_WORKSPACE_IMAGES_TAR> --strip-components=1 -C /opt/kasm/{{ release }}/conf/database/seed_data/ workspace_images/default_images_amd64.yaml
    ```

    For ARM (arm64) platforms:

    ```{parsed-literal}
    tar xf <KASM_WORKSPACE_IMAGES_TAR> --strip-components=1 -C /opt/kasm/{{ release }}/conf/database/seed_data/ workspace_images/default_images_arm64.yaml
    ```

For x86 (amd64) platforms:

```{parsed-literal}
sudo /opt/kasm/{{ release }}/bin/utils/db_init -s /opt/kasm/{{ release }}/conf/database/seed_data/default_images_amd64.yaml
```

For ARM (arm64) platforms:

```{parsed-literal}
sudo /opt/kasm/{{ release }}/bin/utils/db_init -s /opt/kasm/{{ release }}/conf/database/seed_data/default_images_arm64.yaml
```

- Start the Kasm services

```Bash
sudo /opt/kasm/bin/start
```

### Upgrade Web App Server Role

- Stop existing Kasm Services

```Bash
sudo /opt/kasm/bin/stop
```

- Download and extract the new installation media

```Bash
cd /tmp/
curl -O |release_url|
tar -xf kasm_release_*.tar.gz
```

```{note}
If you are performing an offline upgrade please ensure that you have downloaded and installed versions of docker and docker compose that meet the updated system requirements see {doc}`System Requirements <../install/system_requirements>` for details.
```

- Install kasm from the release media downloaded in the prior steps.

  - When performing an offline update add this flag `--offline-service <KASM_SERVICE_IMAGES_TAR>`

```Bash
sudo bash kasm_release/install.sh -S app -D -q <DATABASE_HOSTNAME> -Q <DATABASE_PASSWORD> -R [REDIS_PASSWORD]
```

- Copy **manager_id** and **server_hostname** from the old configuration into the new

```{parsed-literal}
grep manager_id /opt/kasm/{{ previous_release }}/conf/app/api.app.config.yaml
grep server_hostname /opt/kasm/{{ previous_release }}/conf/app/api.app.config.yaml
sudo vi /opt/kasm/{{ release }}/conf/app/api.app.config.yaml
```

- **Advanced**: Review the contents of the existing configs to ensure any custom docker settings are migrated to the new configuration.

```{parsed-literal}
diff /opt/kasm/{{ previous_release }}/docker/docker-compose.yaml  /opt/kasm/{{ release }}/docker/docker-compose.yaml
```

- Start the Kasm services

```Bash
sudo /opt/kasm/bin/start
```

### Upgrade Agent Server Role

- Ensure all Kasm services are stopped

```Bash
sudo /opt/kasm/bin/stop
```

- Download and extract the new installation media

```{parsed-literal}
cd /tmp/
curl -O {{ release_url }}
tar -xf kasm_release_*.tar.gz
```

- Get the existing manager token for use in the subsequent commands.

```{parsed-literal}
sudo grep "token" /opt/kasm/{{ previous_release }}/conf/app/agent.app.config.yaml
```

```{note}
If you are performing an offline upgrade please ensure that you have downloaded and installed versions of docker and docker compose that meet the updated system requirements see {doc}`System Requirements <../install/system_requirements>` for details.
```

- Install kasm from the release media downloaded in the prior steps.

  - When performing an offline update add these flags `--offline-workspaces <KASM_WORKSPACE_IMAGES_TAR> --offline-service <KASM_SERVICE_IMAGES_TAR>`

```Bash
sudo bash kasm_release/install.sh -S agent -D -p <AGENT_HOSTNAME> -m <MANAGER_HOSTNAME> -M <MANAGER_TOKEN>
```

- Copy the server_id and the public_hostname properties from the old agent to the new

```{parsed-literal}
grep server_id /opt/kasm/{{ previous_release }}/conf/app/agent.app.config.yaml
grep public_hostname /opt/kasm/{{ previous_release }}/conf/app/agent.app.config.yaml
sudo vi /opt/kasm/{{ release }}/conf/app/agent.app.config.yaml
```

- Copy the auto-generated nginx configs for any sessions that may exists on the Agent

```{parsed-literal}
sudo cp /opt/kasm/{{ previous_release }}/conf/nginx/containers.d/* /opt/kasm/{{ release }}/conf/nginx/containers.d/
```

- **Advanced**: Review the contents of the existing configs to ensure any custom docker settings are migrated to the new configuration.

```{parsed-literal}
diff /opt/kasm/{{ previous_release }}/docker/docker-compose.yaml  /opt/kasm/{{ release }}/docker/docker-compose.yaml
diff /opt/kasm/{{ previous_release }}/conf/app/agent.app.config.yaml  /opt/kasm/{{ release }}/conf/app/agent.app.config.yaml
```

- Start the Kasm services

```Bash
sudo /opt/kasm/bin/start
```

- Verify the Agent Service is properly checking into the Manager Service. Log into the UI as an Administrator.
  Select **Agents** and verify the Agent is listed with a **Last Reported** time of less than 1 minute.
  If the agent has not checked in, it likely can’t resolve or connect to \[AGENT_HOSTNAME\]:443 . Inspect the logs for details.

```bash
sudo tail -f /opt/kasm/current/log/agent.log
```

### (Optional) Install the Connection Proxy (Guac) Role

The Connection Proxy role is new in Workspaces 1.12.0. This service is used to connect to VM/Hardaward running RDP. 
If these capabilities are not needed, this role does not need to be installed.

Login to the Workspaces UI as an administrator. Retrieve the value of  **Component Registration Token** from the Global
Settings.


- Download and extract the new installation media

```{parsed-literal}
cd /tmp/
curl -O {{ release_url }}
tar -xf kasm_release_*.tar.gz
```

- Install the Connection Proxy role from the release media downloaded in the prior steps.

  - When performing an offline update add these flags `--offline-workspaces <KASM_WORKSPACE_IMAGES_TAR> --offline-service <KASM_SERVICE_IMAGES_TAR>`

```Bash
sudo bash kasm_release/install.sh --role guac --api-hostname [MANAGER_HOSTNAME] --public-hostname [CONNECTION_PROXY_HOSTNAME] --registration-token [SERVICE_REGISTRATION_TOKEN]
```


```{include} upgrade_images.md
```

## Upgrade Script

Since Kasm Workspaces version 1.11.0 an upgrade script is included with the installation package under kasm_release/upgrade.sh.

````{note}
Before starting the upgrade process please ensure that the Web App servers are stopped. If left running there is a chance for database corruption on your Webb App servers
```Bash 
sudo /opt/kasm/bin/stop
```
````

The upgrade script can be found in the installation package under kasm_release/upgrade.sh.

These instructions assume there are 3 servers with the app, db, and agent roles.

Login to your db server and execute:

`````{tabs}

````{tab} Standard Upgrade

```{parsed-literal}
cd /tmp
curl -O {{ release_url }}
tar -xf {{ release_url.split('/') | last }}
sudo bash kasm_release/upgrade.sh --role db
```

````

````{tab} Offline Upgrade x86-64

```{parsed-literal}
cd /tmp
curl -O {{ release_url }}
curl -O {{ amd_service_images_url }}
curl -O {{ amd_workspace_images_url }}
tar -xf {{ release_url.split('/') | last }}
sudo bash kasm_release/upgrade.sh --role db --offline-workspaces /tmp/{{ amd_workspace_images_url.split('/') | last }} --offline-service /tmp/{{ amd_service_images_url.split('/') | last }}
```

````

````{tab} Offline Upgrade arm64

```{parsed-literal}
cd /tmp
curl -O {{ release_url }}
curl -O {{ arm_service_images_url }}
curl -O {{ arm_workspace_images_url }}
tar -xf {{ release_url.split('/') | last }}
sudo bash kasm_release/upgrade.sh --role db --offline-workspaces /tmp/{{ arm_workspace_images_url.split('/') | last }} --offline-service /tmp/{{ arm_service_images_url.split('/') | last }}
```

````

`````

Login to your agent server and execute:

`````{tabs}

````{tab} Standard Upgrade

```{parsed-literal}
cd /tmp
curl -O {{ release_url }}
tar -xf {{ release_url.split('/') | last }}
sudo bash kasm_release/upgrade.sh --role agent
```

````

````{tab} Offline Upgrade x86-64

```{parsed-literal}
cd /tmp
curl -O {{ release_url }}
curl -O {{ amd_service_images_url }}
curl -O {{ amd_workspace_images_url }}
tar -xf {{ release_url.split('/') | last }}
sudo bash kasm_release/upgrade.sh --role agent --offline-workspaces /tmp/{{ amd_workspace_images_url.split('/') | last }} --offline-service /tmp/{{ amd_service_images_url.split('/') | last }}
```

````

````{tab} Offline Upgrade arm64

```{parsed-literal}
cd /tmp
curl -O {{ release_url }}
curl -O {{ arm_service_images_url }}
curl -O {{ arm_workspace_images_url }}
tar -xf {{ release_url.split('/') | last }}
sudo bash kasm_release/upgrade.sh --role agent --offline-workspaces /tmp/{{ arm_workspace_images_url.split('/') | last }} --offline-service /tmp/{{ arm_service_images_url.split('/') | last }}
```

````

`````

Login to your Web App server and execute:

`````{tabs}

````{tab} Standard Upgrade

```{parsed-literal}
cd /tmp
curl -O {{ release_url }}
tar -xf {{ release_url.split('/') | last }}
sudo bash kasm_release/upgrade.sh --role app
```

```` 

````{tab} Offline Upgrade x86-64

```{parsed-literal}
cd /tmp
curl -O {{ release_url }}
curl -O {{ amd_service_images_url }}
tar -xf {{ release_url.split('/') | last }}
sudo bash kasm_release/upgrade.sh --role app --offline-service /tmp/{{ amd_service_images_url.split('/') | last }}
```

````

````{tab} Offline Upgrade arm64

```{parsed-literal}
cd /tmp
curl -O {{ release_url }}
curl -O {{ arm_service_images_url }}
tar -xf {{ release_url.split('/') | last }}
sudo bash kasm_release/upgrade.sh --role app --offline-service /tmp/{{ arm_service_images_url.split('/') | last }}
```

````

`````

You can also handle basic image management functions using the helper flags during the upgrade process:

```text
Flag                       Description
------------------------------------------------------------------------------------
| -h|--help                | Display this help menu                                |
| -L|--proxy-port          | Default Proxy Listening Port                          |
| -I|--no-images           | Don't seed or pull default Workspaces Images          |
| -U|--upgrade-images      | Upgrade Images to current set purging previous images |
| -K|--add-images          | Ingest the latest images keeping old images in place  |
| -w|--offline-workspaces  | Path to the tar.gz workspace images offline installer |
| -s|--offline-service     | Path to the tar.gz service images offline installer   |
| -S|--role                | Role to Upgrade: [app|db|agent|remote_db]             |
------------------------------------------------------------------------------------
```