Skip to main content
Version: 1.22

Verification of model integrity

Models deployed in Privatemode are stored on dm-verity protected disks. This allows you to verify the integrity and content of a model before using it to process your data.

This guide describes how to reproduce the model disks deployed in Privatemode and obtain the expected root hashes. See the source code verification guide to learn how to verify that the hashes are enforced by the Privatemode deployment.

info

This is an optional workflow to build trust in Privatemode. You can securely use Privatemode without performing these steps.

info

Some OS-specific settings, such as SELinux, can interfere with the disk setup. To ensure reproducibility, we recommend running the verification on a fresh Ubuntu 24.04 system with Docker engine installed.

Step 1: Build the disk image generator​

We provide a tool to create the model disk images that you can build from source:

  1. Ensure your system meets the prerequisites:

  2. Clone the source code repository:

    git clone https://github.com/edgelesssys/privatemode-public
    cd privatemode-public

  3. Build the container image:

    nix build .#verity-disk-generator
    docker load < result

Step 2: Inspect the Kubernetes deployment configuration​

The repository contains several storage-class-*.yaml files defining Kubernetes Storage Classes used to manage the storage of the model disks. The following annotations are attached to each Storage Class and are needed to replicate the disk:

  • privatemode.edgeless.systems/disk_size_gb: The size of the disk in gigabytes.
  • privatemode.edgeless.systems/model_source: The URL of the model repository.
  • privatemode.edgeless.systems/commit_hash: The commit hash of the model repository to download.
  • privatemode.edgeless.systems/root_hash: The expected dm-verity root hash of the model disk.
  • privatemode.edgeless.systems/excluded_files: A list of files or wildcards matching files in the model repository that are excluded from the final disk image to reduce size.

Step 3: Create a model disk image​

Using the container you built in Step 1 and the disk information from Step 2, you can now create a dm-verity protected replica of the model disk.

info

Depending on the repository, you may require a valid access token to download the model. Follow the Hugging Face documentation to generate your token.

Assuming you want to reproduce a model disk for facebook/opt-125m at commit 27dcfa74d334bc871f3234de431e71c6eeba5dd6, with a disk size of 1 GB, and excluding files matching example-file-*.txt, run the following commands:

model_source="https://huggingface.co/facebook/opt-125m"
commit_hash="27dcfa74d334bc871f3234de431e71c6eeba5dd6"
disk_size_gb="1"
excluded_files="example-file-*.txt"
git_pat=<your_git_pat> # leave empty if not required
disk_image=model.disk

truncate -s ${disk_size_gb}G ${disk_image}
touch repart.json

docker run --rm -it \
    --privileged \
    -v ${PWD}/${disk_image}:/${disk_image} \
    -v ${PWD}/repart.json:/repart.json \
    -e GIT_PAT=${git_pat} \
    -e EXCLUDE_GIT_FILES="${excluded_files}" \
    verity-disk-generator \
    "${disk_image}" "${model_source}" "${commit_hash}"

Retrieve the dm-verity root hash from the repart.json file:

jq -r '.[0].roothash' repart.json

This root hash should match the privatemode.edgeless.systems/root_hash annotation of the Storage Class you inspected in Step 2.

tip

Take a look at the source code of Privatemode's disk-mounter to learn how Privatemode verifies the integrity of a model at runtime. Follow the source code verification guide to learn how to verify and reproduce the binary of the disk-mounter.