Prompt caching

Privatemode supports prompt caching to reduce response latency when the first part of a prompt can be reused across requests. This is especially relevant for requests with long shared context or long conversation history. Caching is available for chat completion and completion requests.

Prompt caching is inactive by default. You can enable it in the Privatemode proxy. All requests sent via the same proxy share a cache and no further changes are required when making requests.

Alternatively, you can configure it per request via the request field cache_salt, encoded as a string. All requests that use the same salt share a cache. For example, using the same salt in all requests of a user will create an isolated cache for that user.

#!/usr/bin/env bash

curl localhost:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "gpt-oss-120b",
    "messages": [{"role": "user", "content": "Tell me a joke!"}],
    "cache_salt": "Y3+y3nLYf3a0CvT7VtuI0W656YXyl0Rdvd8BHI9e2rU="
  }'

import openai
import os

client = openai.OpenAI(
    api_key=os.environ.get("PRIVATE_MODE_API_KEY"), base_url="http://localhost:8080/v1"
)

client.chat.completions.create(
    model="gpt-oss-120b",
    messages=[{"role": "user", "content": "Tell me a joke!"}],
    extra_body={
        "cache_salt": "Y3+y3nLYf3a0CvT7VtuI0W656YXyl0Rdvd8BHI9e2rU=",
    },
)

When using the OpenAI Python client, provide cache_salt as part of an extra_body argument. If caching is configured in both, the Privatemode proxy and in a request, the value from the request is used, allowing for more granular control by clients.

Cache salts should be kept private and have an entropy of at least 256 bits. You can generate a secure salt with openssl rand -base64 32.