mirror of
https://github.com/BerriAI/litellm.git
synced 2026-09-11 22:51:28 +00:00
cookbook: add LiteLLM Proxy + SpendGuard runtime budget guardrails
Adds a cookbook notebook (cookbook/logging_observability/ LiteLLM_Proxy_SpendGuard.ipynb) demonstrating how to wire SpendGuard's budget-gating CustomLogger into the LiteLLM proxy. What SpendGuard adds on top of LiteLLM's built-in spend tracking: - Pre-call reserve: contract-DSL evaluator decides ALLOW / DENY / DEGRADE / REQUIRE_APPROVAL BEFORE the upstream provider is hit. Budget exhausted → HTTP 403; provider invoice clock never starts. - Post-call commit with reconciliation: end-of-stream reconciler reads response.usage.completion_tokens and commits the real cost. - Signed append-only audit chain: every decision lands as a CloudEvent in canonical_events with the LiteLLM-specific 12-field enrichment (litellm_call_id, model, team_id, pricing_version, ...). - Fail-closed default; SPENDGUARD_LITELLM_FAIL_OPEN=1 dev override. Pure CustomLogger callback — no LiteLLM source changes. Companion to the existing Langfuse/Arize/Lunary notebooks. SpendGuard sits in the same observability slot but as a fail-closed budget control gate, not just a tracer. PyPI: https://pypi.org/project/spendguard-sdk/0.3.0/ Repo: https://github.com/m24927605/agentic-spendguard Signed-off-by: Michael Chen <m24927605@gmail.com>
This commit is contained in:
Michael Chen
parent
e59e34bed3
commit
49bb01125b
1 changed files with 234 additions and 0 deletions
234
cookbook/logging_observability/LiteLLM_Proxy_SpendGuard.ipynb
vendored
Normal file
234
cookbook/logging_observability/LiteLLM_Proxy_SpendGuard.ipynb
vendored
Normal file
|
|
@ -0,0 +1,234 @@
|
|||
{
|
||||
"cells": [
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Budget Guardrails — LiteLLM Proxy + SpendGuard\n",
|
||||
"\n",
|
||||
"This notebook demonstrates how to use the LiteLLM Proxy with [SpendGuard](https://github.com/m24927605/agentic-spendguard) to gate every `/v1/chat/completions` call against a budget BEFORE the upstream provider is hit.\n",
|
||||
"\n",
|
||||
"**What SpendGuard adds on top of LiteLLM's own spend tracking:**\n",
|
||||
"- **Pre-call reserve** — SpendGuard's contract-DSL evaluator decides ALLOW / DENY / DEGRADE / REQUIRE_APPROVAL before LiteLLM forwards to the provider. Budget exhausted? HTTP 403, provider invoice clock never starts.\n",
|
||||
"- **Post-call commit with reconciliation** — End-of-stream reconciler reads `response.usage.completion_tokens` and commits the real cost (not the worst-case estimator).\n",
|
||||
"- **Signed append-only audit chain** — Every decision lands as a CloudEvent in `canonical_events`, with the 12-field LiteLLM-specific enrichment (`litellm_call_id`, `model`, `team_id`, `pricing_version`, etc) for forensics.\n",
|
||||
"- **Fail-closed default** — Sidecar unreachable → HTTP 503; the request is denied, not forwarded.\n",
|
||||
"\n",
|
||||
"SpendGuard wires in as a standard LiteLLM `CustomLogger` callback. No LiteLLM source change required."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## 1. Setup LiteLLM Proxy + SpendGuard\n",
|
||||
"\n",
|
||||
"### 1.1 Install the SpendGuard SDK\n",
|
||||
"\n",
|
||||
"```bash\n",
|
||||
"pip install 'spendguard-sdk[litellm]==0.3.0'\n",
|
||||
"```\n",
|
||||
"\n",
|
||||
"The `litellm` extra pulls `litellm[proxy]` transitively so the proxy CLI works out of the box.\n",
|
||||
"\n",
|
||||
"### 1.2 Run the SpendGuard sidecar\n",
|
||||
"\n",
|
||||
"The SpendGuard sidecar is a Rust binary that runs alongside your LiteLLM proxy and exposes a Unix domain socket for the callback to talk to. The fastest path is the runnable example:\n",
|
||||
"\n",
|
||||
"```bash\n",
|
||||
"git clone https://github.com/m24927605/agentic-spendguard\n",
|
||||
"cd agentic-spendguard/examples/litellm-proxy-composite\n",
|
||||
"docker compose up --build\n",
|
||||
"```\n",
|
||||
"\n",
|
||||
"This brings up postgres + sidecar + your LiteLLM proxy in 3 containers."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### 1.3 Write your operator callback module\n",
|
||||
"\n",
|
||||
"Save as `spendguard_callback.py` next to your `proxy_config.yaml`:\n",
|
||||
"\n",
|
||||
"```python\n",
|
||||
"import os\n",
|
||||
"from spendguard._proto.spendguard.common.v1 import common_pb2\n",
|
||||
"from spendguard.integrations.litellm import (\n",
|
||||
" BudgetBinding, ResolverContext, _LoopBoundCallback,\n",
|
||||
")\n",
|
||||
"\n",
|
||||
"_UNIT = common_pb2.UnitRef(unit_id=os.environ['SPENDGUARD_UNIT_ID'],\n",
|
||||
" token_kind='output_token', model_family='gpt-4')\n",
|
||||
"_PRICING = common_pb2.PricingFreeze(\n",
|
||||
" pricing_version=os.environ['SPENDGUARD_PRICING_VERSION'],\n",
|
||||
" price_snapshot_hash=bytes.fromhex(os.environ['SPENDGUARD_PRICE_SNAPSHOT_HASH_HEX']),\n",
|
||||
" fx_rate_version=os.environ['SPENDGUARD_FX_RATE_VERSION'],\n",
|
||||
" unit_conversion_version=os.environ['SPENDGUARD_UNIT_CONVERSION_VERSION'],\n",
|
||||
")\n",
|
||||
"_BINDING = BudgetBinding(\n",
|
||||
" budget_id=os.environ['SPENDGUARD_BUDGET_ID'],\n",
|
||||
" window_instance_id=os.environ['SPENDGUARD_WINDOW_INSTANCE_ID'],\n",
|
||||
" unit=_UNIT, pricing=_PRICING,\n",
|
||||
")\n",
|
||||
"\n",
|
||||
"def _resolve(ctx: ResolverContext) -> BudgetBinding | None:\n",
|
||||
" # In production: dispatch by ctx.user_api_key_dict.team_id\n",
|
||||
" return _BINDING\n",
|
||||
"\n",
|
||||
"def _estimate(ctx):\n",
|
||||
" return [common_pb2.BudgetClaim(\n",
|
||||
" budget_id=_BINDING.budget_id, unit=_UNIT, amount_atomic='50',\n",
|
||||
" direction=common_pb2.BudgetClaim.DEBIT,\n",
|
||||
" window_instance_id=_BINDING.window_instance_id,\n",
|
||||
" )]\n",
|
||||
"\n",
|
||||
"def _reconcile(ctx, response):\n",
|
||||
" tokens = int(getattr(getattr(response, 'usage', None), 'completion_tokens', 0) or 0)\n",
|
||||
" return [common_pb2.BudgetClaim(\n",
|
||||
" budget_id=_BINDING.budget_id, unit=_UNIT, amount_atomic=str(max(tokens, 1)),\n",
|
||||
" direction=common_pb2.BudgetClaim.DEBIT,\n",
|
||||
" window_instance_id=_BINDING.window_instance_id,\n",
|
||||
" )]\n",
|
||||
"\n",
|
||||
"handler_instance = _LoopBoundCallback(\n",
|
||||
" socket_path=os.environ['SPENDGUARD_SIDECAR_UDS'],\n",
|
||||
" tenant_id=os.environ['SPENDGUARD_TENANT_ID'],\n",
|
||||
" budget_resolver=_resolve,\n",
|
||||
" claim_estimator=_estimate,\n",
|
||||
" claim_reconciler=_reconcile,\n",
|
||||
")\n",
|
||||
"```"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### 1.4 Update proxy_config.yaml\n",
|
||||
"\n",
|
||||
"```yaml\n",
|
||||
"model_list:\n",
|
||||
" - model_name: gpt-4o-mini\n",
|
||||
" litellm_params:\n",
|
||||
" model: openai/gpt-4o-mini\n",
|
||||
" api_key: os.environ/OPENAI_API_KEY\n",
|
||||
"\n",
|
||||
"litellm_settings:\n",
|
||||
" callbacks: [\"spendguard_callback.handler_instance\"]\n",
|
||||
" drop_params: true\n",
|
||||
"\n",
|
||||
"general_settings:\n",
|
||||
" master_key: os.environ/LITELLM_MASTER_KEY\n",
|
||||
"```\n",
|
||||
"\n",
|
||||
"Set `PYTHONPATH` to include the directory of `spendguard_callback.py` before launching the proxy."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### 1.5 Launch the proxy\n",
|
||||
"\n",
|
||||
"```bash\n",
|
||||
"export SPENDGUARD_SIDECAR_UDS=/var/run/spendguard/adapter.sock\n",
|
||||
"export SPENDGUARD_TENANT_ID=...\n",
|
||||
"export SPENDGUARD_BUDGET_ID=...\n",
|
||||
"# ... (full env-var list at github.com/m24927605/agentic-spendguard/blob/main/docs/specs/litellm-integration/PROXY_RECIPE.md)\n",
|
||||
"\n",
|
||||
"python -m litellm.proxy.proxy_cli --config proxy_config.yaml --port 4000\n",
|
||||
"```"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## 2. Make LLM Requests to the Gated Proxy\n",
|
||||
"\n",
|
||||
"Standard OpenAI client — no SpendGuard SDK in your app code. The gating is invisible until the budget hits a limit."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"import openai\n",
|
||||
"\n",
|
||||
"client = openai.OpenAI(\n",
|
||||
" base_url=\"http://localhost:4000\",\n",
|
||||
" api_key=\"sk-demo-key\", # LITELLM_MASTER_KEY\n",
|
||||
")\n",
|
||||
"\n",
|
||||
"# ALLOW path — SpendGuard reserves + commits normally.\n",
|
||||
"response = client.chat.completions.create(\n",
|
||||
" model=\"gpt-4o-mini\",\n",
|
||||
" messages=[{\"role\": \"user\", \"content\": \"hello\"}],\n",
|
||||
")\n",
|
||||
"print(response.choices[0].message.content)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## 3. Verify the Audit Chain\n",
|
||||
"\n",
|
||||
"Every call writes a signed CloudEvent to `canonical_events`. Cross-table join with LiteLLM's own `LiteLLM_SpendLogs` (if you have a SpendLogs database configured):\n",
|
||||
"\n",
|
||||
"```sql\n",
|
||||
"SELECT\n",
|
||||
" ce.event_type,\n",
|
||||
" ce.tenant_id,\n",
|
||||
" cost_advisor_safe_decode_payload(ce.payload_json)->'spendguard'->>'litellm_call_id'\n",
|
||||
" AS litellm_call_id,\n",
|
||||
" cost_advisor_safe_decode_payload(ce.payload_json)->'spendguard'->>'model' AS model,\n",
|
||||
" cost_advisor_safe_decode_payload(ce.payload_json)->'spendguard'->>'team_id' AS team_id,\n",
|
||||
" cost_advisor_safe_decode_payload(ce.payload_json)->>'final_decision' AS decision\n",
|
||||
"FROM canonical_events ce\n",
|
||||
"WHERE ce.event_type IN ('spendguard.audit.decision', 'spendguard.audit.outcome')\n",
|
||||
" AND ce.event_time > NOW() - INTERVAL '1 hour'\n",
|
||||
"ORDER BY ce.event_time DESC;\n",
|
||||
"```\n",
|
||||
"\n",
|
||||
"The `spendguard` JSONB sub-object on each row carries the 12-field LiteLLM-specific enrichment per [DESIGN §8.2a](https://github.com/m24927605/agentic-spendguard/blob/main/docs/specs/litellm-integration/DESIGN.md)."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## 4. Going Further\n",
|
||||
"\n",
|
||||
"- **Multi-tenant via virtual keys:** the operator template above is single-team. To dispatch per LiteLLM team, inspect `ctx.user_api_key_dict.team_id` in `_resolve` and look up the binding in your control plane. Full template: [PROXY_RECIPE.md §2](https://github.com/m24927605/agentic-spendguard/blob/main/docs/specs/litellm-integration/PROXY_RECIPE.md).\n",
|
||||
"- **Streaming:** SpendGuard's `_async_log_success_streaming` reconciles end-of-stream `usage.completion_tokens` automatically. No app code change.\n",
|
||||
"- **Direct (non-proxy) async callers:** `from spendguard.integrations.litellm import SpendGuardDirectAcompletion` wraps `litellm.acompletion()`. Sync `litellm.completion()` is not supported; route via the SpendGuard egress proxy for that.\n",
|
||||
"- **Fail-open dev override:** `SPENDGUARD_LITELLM_FAIL_OPEN=1` lets calls through when the sidecar is unreachable (development only).\n",
|
||||
"\n",
|
||||
"Runnable end-to-end demo:\n",
|
||||
"```bash\n",
|
||||
"git clone https://github.com/m24927605/agentic-spendguard\n",
|
||||
"cd agentic-spendguard\n",
|
||||
"make demo-up DEMO_MODE=litellm_real # 4-step ALLOW + DENY + STREAM + MULTI-TEAM\n",
|
||||
"make demo-up DEMO_MODE=litellm_deny # 3 fail-closed sub-steps\n",
|
||||
"```"
|
||||
]
|
||||
}
|
||||
],
|
||||
"metadata": {
|
||||
"kernelspec": {
|
||||
"display_name": "Python 3",
|
||||
"language": "python",
|
||||
"name": "python3"
|
||||
},
|
||||
"language_info": {
|
||||
"name": "python"
|
||||
}
|
||||
},
|
||||
"nbformat": 4,
|
||||
"nbformat_minor": 2
|
||||
}
|
||||
Loading…
Add table
Reference in a new issue