Skip to main content
Version: 1.38

Verification of model integrity

Models deployed in Privatemode are stored on dm-verity-protected disks. This allows you to verify the integrity and content of a model before using it.

This guide describes how to reproduce the model disks deployed in Privatemode and obtain the expected root hashes. See the source code verification guide to learn how to verify that the hashes are enforced by the Privatemode deployment.

info

This is an optional workflow to build trust in Privatemode. You can securely use Privatemode without performing these steps.

Prerequisites​

Ensure your system meets the following requirements:

info

Some OS-specific settings can interfere with the disk setup. To ensure reproducibility, we recommend running the verification on a fresh Ubuntu 24.04 system with Docker engine installed.

Get the public source code required for the verification:

# Replace vX.Y.Z with the version you want to verify, e.g., v1.37.0
git clone --branch vX.Y.Z https://github.com/edgelesssys/privatemode-public
cd privatemode-public

Step 1: Inspect the Kubernetes deployment configuration​

The repository contains a models.yaml file. This file contains a list of all the models currently available in Privatemode, along with all the information required to rebuild the exact same disk from its sources.

Step 2: Build the disk image generator​

We provide a tool to create the model disk images that you can build from source.

In the privatemode-public repository, build the disk generation container image:

nix build .#verity-disk-generator
docker load < result

The image is now available locally as verity-disk-generator:vX.Y.Z.

Step 3: Create a model disk image​

Using the disk information from Step 1 and the container you built in Step 2, you can now create a byte-by-byte identical replica of the dm-verity-protected model disk.

info

Depending on the repository, you may require a valid access token to download the model. Follow the Hugging Face documentation to generate your token.

From the models.yaml file, find the ID of the model you want to verify:

echo "Model IDs:"
yq '.data | keys()' charts/models.yaml

Assuming you want to reproduce a model disk for facebook-opt-125m, run the following commands:

#!/usr/bin/env bash

model_id="facebook-opt-125m"
build_version="vX.Y.Z" # replace with actual build version
git_pat="your_git_pat" # leave empty if not required

model_data="$(yq ".data.${model_id}" models.yaml)"

excluded_files="$(echo "${model_data}" | yq -r '.excluded_files')"
model_source="$(echo "${model_data}" | yq -r '.model_source')"
commit_hash="$(echo "${model_data}" | yq -r '.commit_hash')"

echo "Model ID: ${model_id}"
echo "Model Source: ${model_source}"
echo "Commit Hash: ${commit_hash}"
echo "Excluded files: ${excluded_files}"

docker run -it --rm -v "${PWD}:/models" \
  -e "STAGING_PATH=/models" \
  -e "EXCLUDE_GIT_FILES=${excluded_files}" \
  -e "GIT_PAT=${git_pat}" \
  "verity-disk-generator:${build_version}" \
  "${model_source}" "${commit_hash}"

Retrieve the dm-verity root hash from the output:

VERITY header information for /models/facebook-opt-125m.tmp.
...
Root hash:              08da1ef4dfc95438a13d964d565a341c0e5de32a5d65c28dcb1414a3a7c675dc

This root hash should match the root_hash field of the model definition you inspected in Step 1.

tip

Take a look at the source code of Privatemode's disk-mounter to learn how Privatemode verifies the integrity of a model at runtime. Follow the source code verification guide to learn how to verify and reproduce the disk-mounter component that makes the disk available for the workload.

Last updated on