Verification of model integrity
Models deployed in Privatemode are stored on dm-verity protected disks. This allows you to verify the integrity and content of a model before using it to process your data.
This guide describes how to reproduce the model disks deployed in Privatemode and obtain the expected root hashes. See the source code verification guide to learn how to verify that the hashes are enforced by the Privatemode deployment.
This is an optional workflow to build trust in Privatemode. You can securely use Privatemode without performing these steps.
Some OS-specific settings, such as SELinux, can interfere with the disk setup. To ensure reproducibility, we recommend running the verification on a fresh Ubuntu 24.04 system with Docker engine installed.
Step 1: Build the disk image generator
We provide a tool to create the model disk images that you can build from source:
-
Ensure your system meets the prerequisites:
- Linux operating system (x86-64 architecture)
- Nix
- To install Nix, we recommend the Determinate Systems Nix installer.
- Docker
- jq
-
Clone the source code repository:
git clone https://github.com/edgelesssys/privatemode-public
cd privatemode-public -
Build the container image:
nix build .#verity-disk-generator
docker load < result
Step 2: Inspect the Kubernetes deployment configuration
The repository contains several storage-class-*.yaml
files defining Kubernetes Storage Classes used to manage the storage of the model disks.
The following annotations are attached to each Storage Class and are needed to replicate the disk:
privatemode.edgeless.systems/disk_size_gb
: The size of the disk in gigabytes.privatemode.edgeless.systems/model_source
: The URL of the model repository.privatemode.edgeless.systems/commit_hash
: The commit hash of the model repository to download.privatemode.edgeless.systems/root_hash
: The expected dm-verity root hash of the model disk.privatemode.edgeless.systems/excluded_files
: A list of files or wildcards matching files in the model repository that are excluded from the final disk image to reduce size.
Step 3: Create a model disk image
Using the container you built in Step 1 and the disk information from Step 2, you can now create a dm-verity protected replica of the model disk.
Depending on the repository, you may require a valid access token to download the model. Follow the Hugging Face documentation to generate your token.
Assuming you want to reproduce a model disk for facebook/opt-125m
at commit 27dcfa74d334bc871f3234de431e71c6eeba5dd6
, with a disk size of 1 GB, and excluding files matching example-file-*.txt
, run the following commands:
model_source="https://huggingface.co/facebook/opt-125m"
commit_hash="27dcfa74d334bc871f3234de431e71c6eeba5dd6"
disk_size_gb="1"
excluded_files="example-file-*.txt"
git_pat=<your_git_pat> # leave empty if not required
disk_image=model.disk
truncate -s ${disk_size_gb}G ${disk_image}
touch repart.json
docker run --rm -it \
--privileged \
-v ${PWD}/${disk_image}:/${disk_image} \
-v ${PWD}/repart.json:/repart.json \
-e GIT_PAT=${git_pat} \
-e EXCLUDE_GIT_FILES="${excluded_files}" \
verity-disk-generator \
"${disk_image}" "${model_source}" "${commit_hash}"
Retrieve the dm-verity root hash from the repart.json
file:
jq -r '.[0].roothash' repart.json
This root hash should match the privatemode.edgeless.systems/root_hash
annotation of the Storage Class you inspected in Step 2.
Take a look at the source code of Privatemode's disk-mounter to learn how Privatemode verifies the integrity of a model at runtime. Follow the source code verification guide to learn how to verify and reproduce the binary of the disk-mounter.