Prompt caching

Privatemode supports prompt caching to reduce response latency when the first part of a prompt can be reused across requests. This is especially relevant for requests with long shared context or long conversation history. Caching is available for chat completion, completion, and Anthropic messages requests.

Enable caching in the proxy

Prompt caching is off by default when you run your own proxy: the proxy sends a new random salt with each request. Start the proxy with --disable-prompt-cache=false to share one cache across all requests sent through that proxy; no changes to the requests are required. The proxy's prompt caching flags control the shared salt.

Set a salt per request

Alternatively, configure caching per request via the request field cache_salt, encoded as a string. All requests that use the same salt share a cache. For example, using the same salt in all requests of a user creates an isolated cache for that user.

#!/usr/bin/env bash

curl localhost:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "gpt-oss-120b",
    "messages": [{"role": "user", "content": "Tell me a joke!"}],
    "cache_salt": "637fb2de72d87f76b40af4fb56db88d16eb9e985f297445dbddf011c8f5edab5"
  }'

import openai
import os

client = openai.OpenAI(
    # A placeholder suffices when the proxy runs with --api-key.
    api_key=os.environ.get("PRIVATEMODE_API_KEY", "placeholder"),
    base_url="http://localhost:8080/v1",
)

client.chat.completions.create(
    model="gpt-oss-120b",
    messages=[{"role": "user", "content": "Tell me a joke!"}],
    extra_body={
        "cache_salt": "637fb2de72d87f76b40af4fb56db88d16eb9e985f297445dbddf011c8f5edab5",
    },
)

When using the OpenAI Python client, provide cache_salt as part of an extra_body argument. If caching is configured in both, the Privatemode proxy and in a request, the value from the request is used, allowing for more granular control by clients.

Cache scopes and salts

All requests that use the same salt belong to the same cache scope. Clients within a scope can reuse, and probe, the entries created by the other clients in that scope. A salt can isolate the cache for one user or provide sharing within a trusted team or app. Choose the narrowest scope that provides the required performance benefit.

When managing cache salts:

• Keep salts private and handle them like credentials.
• Generate salts from at least 256 bits of cryptographically secure randomness, for example with openssl rand -hex 32.
• Don't derive salts from predictable values such as user IDs or organization names.
• Salts must be at most 128 characters long and must not contain @, /, \, or NUL.

Rotating a salt prevents access to entries created with the old salt. See Prompt cache security for the isolation design and the limits of cache scopes.