Quickstart

The Privatemode API provides a GenAI inference service designed with end-to-end encryption and privacy preservation at its core. Setting it up is a breeze. It shouldn't take more than 10 minutes of your time.

1. Install Docker

Follow these instructions to install Docker.

2. Run the proxy

The Privatemode API comes with its own proxy. The privatemode-proxy takes care of client-side encryption and verifies the integrity and identity of the entire service using remote attestation. Use the following command to run the proxy:

docker run -p 8080:8080 ghcr.io/edgelesssys/privatemode/privatemode-proxy:latest --apiKey <your-api-token>

Alternatively to Docker, you may also run the native binary on Linux amd64/ arm64 as described here.

This opens an endpoint on your localhost on port 8080. Afterward, you can simply send prompts to this endpoint. Of course you can run the proxy wherever you want and configure TLS.

3. Send prompts

Now you're all set to use the API. The proxy handles all the security (and confidential computing) intricacies for you. Let's start by sending our first prompt:

Bash

Example request

curl localhost:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "ibnzterrell/Meta-Llama-3.3-70B-Instruct-AWQ-INT4",
    "messages": [
      {
        "role": "system",
        "content": "Hello Privatemode!"
      }
    ]
  }'

Example response

    "id": "chat-c87bdd75d1394dcc886556de3db5d0c9",
    "object": "chat.completion",
    "created": 1727429032,
    "model": "latest",
    "choices": [
        {
            "index": 0,
            "message": {
                "role": "assistant",
                "content": "Hello. I'm here to help you in any way I can.",
                "tool_calls": []
            },
            "logprobs": null,
            "finish_reason": "stop",
            "stop_reason": null
        }
    ],
    "usage": {
        "prompt_tokens": 34,
        "total_tokens": 49,
        "completion_tokens": 15
    },
    "prompt_logprobs": null
}

Python

Example request

import requests
import json

# A wrapper class for the API for convenience
class Privatemode:
    def __init__(self, url, port):
        self.endpoint = f"http://{url}:{port}/v1/chat/completions"  # Adjust path if necessary
        self.model = "latest"

    def run(self, prompt):
        # JSON payload with the necessary parameters
        payload = {
            "model": self.model,
            "messages":  [{
                "role": "system",
                "content": prompt
            }],
        }

        headers = {"Content-Type": "application/json"}

        # Sending the request to your proxy
        response = requests.post(self.endpoint, headers=headers, data=json.dumps(payload))

        # Error handling in case of a bad response
        if response.status_code != 200:
            raise Exception(f"Error {response.status_code}: {response.text}")

        # Return the response JSON
        return response.json()

# Example usage
if __name__ == "__main__":

    # Initialize the wrapper with the URL and port of proxy
    model = Privatemode(url="localhost", port=8080)

    # Run a prompt through the API
    try:
        response = model.run("Hello Privatemode")
        print(response.get('choices')[0].get('message').get('content'))  # Print the API response
    except Exception as e:
        print(f"An error occurred: {e}")

Example response

It's nice to meet you. Is there something I can help you with or would you like to chat?

Javascript

Example request

import fetch from "node-fetch";

// A wrapper class for the API for convenience
class Privatemode {
  constructor(url, port) {
    this.endpoint = `http://${url}:${port}/v1/chat/completions`; // Adjust path if necessary
    this.model = "latest";
  }

  async run(prompt) {
    // JSON payload with the necessary parameters
    const payload = {
      model: this.model,
      messages: [
        {
          role: "system",
          content: prompt,
        },
      ],
    };

    const headers = {
      "Content-Type": "application/json",
    };

    try {
      // Sending the request to your proxy
      const response = await fetch(this.endpoint, {
        method: "POST",
        headers: headers,
        body: JSON.stringify(payload),
      });

      // Error handling in case of a bad response
      if (!response.ok) {
        throw new Error(`Error ${response.status}: ${response.statusText}`);
      }

      // Return the response JSON
      const data = await response.json();
      return data;
    } catch (error) {
      // Handle errors
      throw new Error(`Request failed: ${error.message}`);
    }
  }
}

// Example usage
(async () => {
  // Initialize the wrapper with the URL and port of the proxy
  const model = new Privatemode("localhost", 8080);

  // Run a prompt through the API
  try {
    const response = await model.run("Hello Privatemode");
    console.log(response.choices[0].message.content); // Print the API response
  } catch (error) {
    console.log(`An error occurred: ${error.message}`);
  }
})();

Example response

It's nice to meet you. Is there something I can help you with or would you like to chat?

The code performs the following three simple steps:

1. Construct a prompt request following the OpenAI Chat API specifications.
2. Send the prompt request to the proxy. The proxy handles end-to-end encryption and verifies the integrity of the Privatemode backend that serves the endpoint.
3. Receive and print the response.

info

We don't use any OpenAI services. We only adhere to the same interface definitions to provide a great development experience and ensure easy code portability.