litellm/prompt_compression

Prompt Compression Plugins

Microservices that plug into the LiteLLM gateway as pre_call guardrails to compress incoming prompts before they reach the LLM provider. Each runs independently and speaks the LiteLLM generic guardrail API.

How it works

LiteLLM sends the full message array to the microservice before forwarding the request to the LLM provider. The microservice compresses it and returns the compressed messages. LiteLLM then sends the smaller payload to the provider — reducing cost and latency transparently to the caller.

Available plugins

Plugin	Compression approach
headroom	ML-based; deduplicates JSON arrays, compresses tool outputs and prose via a trained model

Adding a plugin to your LiteLLM config

guardrails:
  - guardrail_name: headroom-compression
    litellm_params:
      guardrail: generic_guardrail_api
      mode: pre_call
      api_base: http://localhost:8100
      # api_key: your-secret-key

The mode: pre_call ensures compression runs before the request reaches the provider. The plugin returns GUARDRAIL_INTERVENED with compressed structured_messages when savings are found, or NONE to pass through unchanged.