Security

Prompt cache security

Privatemode uses the prefix cache in vLLM to reuse work when requests begin with the same prompt. This page explains the security considerations introduced by this cache and how Privatemode isolates cached data. To enable and configure the feature, see Prompt caching.

Cached data is sensitive

During inference, the model computes key and value vectors for the tokens in a prompt. Keeping these vectors in a prompt cache avoids recomputing them when a later request has the same prefix. A prompt cache doesn't contain the prompt as plaintext, but it represents the prompt and requires the same confidentiality protection.

Privatemode keeps the cache in GPU memory and encrypted CPU memory within the confidential AI worker. The existing confidential computing, encryption, and remote attestation mechanisms protect it from direct access by the infrastructure and service providers.

Timing side channel

A cache hit skips part of the prompt processing and is therefore faster than a cache miss. Without cache isolation, an attacker with access to the same inference service could submit guessed prompt prefixes and use response times to determine whether a guess matches cached data. Repeating this process can reveal another user's prompt block by block. This attack doesn't require direct access to worker memory and therefore needs a separate protection beyond confidential computing.

The vLLM security advisory describes the timing difference and attack.

Cache salting

Privatemode separates cache entries with a secret cache_salt. The client sends the salt within the end-to-end encrypted request, so the server side exposes it only inside the confidential worker. vLLM divides a prompt into token blocks and computes a cache lookup key for each full block with the SHA-256 cryptographic hash function. The first block's key includes the salt, and every subsequent key includes the previous block's key. This propagates the salt through the complete cached prefix.

flowchart LR
  S[Secret cache salt] --> H1[SHA-256]
  B1[Token block 1] --> H1
  H1 --> K1[Cache key 1]
  K1 --> H2[SHA-256]
  B2[Token block 2] --> H2
  H2 --> K2[Cache key 2]
  K2 --> H3[SHA-256]
  B3[Token block 3] --> H3
  H3 --> K3[Cache key 3]

Consequently, a cache hit requires both an identical prompt prefix and the same salt. A request with a different salt can't produce matching cache keys, so an attacker outside the cache scope observes only cache misses for their guesses.

The Privatemode proxy always includes a salt in inference requests. By default, it generates a new random 256-bit salt for every request, preventing cache reuse between requests. Configuring caching causes the proxy or client to reuse a salt for the desired cache scope.

Cache scope and limitations

All clients that use the same salt belong to the same cache scope. A salt can isolate the cache for one user or provide sharing within a trusted team or app. Clients within a shared scope can reuse and probe entries created by the other clients in that scope. Choose the narrowest scope that provides the required performance benefit.

When managing cache salts:

  • Keep salts private and handle them like credentials.
  • Generate salts from at least 256 bits of cryptographically secure randomness.
  • Don't derive salts from predictable values such as user IDs or organization names.

Cache entries are transient, and the worker evicts them as it needs memory. Rotating or losing a salt prevents access to entries created with the old salt, but the old blocks can remain in worker memory until eviction.

See Prompt caching for the supported cache scopes and configuration options.

Further reading