* feat(proxy): add admin-only /debug/report sharing the bug report environment fields
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(proxy): add verbose=true to /debug/report listing every config key with typed values
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): inject auth into /debug/report through Annotated to keep the B008 budget flat
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): bound the verbose config walk, drop nested-list recursion from the safe renderer, regenerate schema.d.ts
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): count pass-through and mcp header maps plus operator-named budget maps in verbose /debug/report, single-exit scalar renderers
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* chore(ui): regenerate schema.d.ts after dropping the verbose query from /debug/report
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: ryan <ryan@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(router): time-windowed team reservation of deployments via model_info.access_windows
Deployments can declare model_info.access_windows, a list of local wall-clock windows (IANA timezone, cross-midnight allowed) that reserve the deployment for the listed team_ids. While a window is active the router drops the deployment for every other request, including no-team and admin requests, on every candidate path (model name, model id, specific_deployment, early-resolve, wildcard, litellm_params.model lookup, fallbacks). If every candidate is reserved the request fails with a 400 naming the window end instead of falling back. Outside a window routing is unchanged and reserved deployments stay visible in /model/info and /v1/models. Malformed windows (bad time, unknown timezone, empty team_ids, start equal to end, offset-aware times) fail proxy startup with a clear error since the proxy router runs with ignore_invalid_deployments=True
Resolves LIT-8308
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(router): cover _filter_reserved_deployments directly for coverage gate
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(router): keep reservation filtering immutable
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(router): drop strategy markers before reservation filtering
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: yassin <yassin@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(ui): surface the owner's user budget on keys without their own budget
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(ui): apply the owner's budget hint to team keys when apply_user_budget_to_team_keys is on
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(ui): read only the apply_user_budget_to_team_keys flag from general_settings
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): mark the general_settings cast as cast-ok
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(ui): load the owner's budget for keys opened outside the current page
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: jesus <jesus@berri.ai>
* fix(cost): bill batch prompts above 272K at OpenAI's long-context batch tier
* fix(cost): mirror batch long-context keys on custom pricing params
Register the two *_above_272k_tokens_batches keys on CustomPricingLiteLLMParams so a per-deployment override stays out of the shared backend key, add them to the inline model-info schema and alias-count tests, and build LiteLLM_Params and GenericLiteLLMParams through model_validate at the two dict-splat call sites so basedpyright's reportArgumentType budget ratchets down instead of blocking the new fields.
* fix(cost): add the gpt-5.5-pro batch long-context tier and ignore malformed batch tier keys
* fix(cost): bill cached batch tokens at OpenAI's cached batch rate
Adds cache_read_input_token_cost_batches and
cache_read_input_token_cost_above_272k_tokens_batches for the tiered
OpenAI entries at half the standard cached rate, bills cached batch
tokens at that rate per output line, and parses string-valued batch
rates in deployment-level model_info.
* fix(cost): bill batch cache writes at the batch cache-write rate and carry published batch rates for one-sided deployments
OpenAI's Batch table prices cache writes for gpt-6-astra, gpt-5.6, gpt-5.6-sol, gpt-5.6-terra and gpt-5.6-luna at half the standard cache-write rate, so the cost map gains cache_creation_input_token_cost_batches and its above_272k tier for those entries and batch cost pulls written tokens out of the input bucket at that rate; models without the key keep billing writes at the batch input rate.
A deployment declaring only one side of its batch pricing now carries every published batch rate of the other side (tier, cached, cache write), its own keys win, and a lone tier, cached or cache-write batch key counts as declared pricing instead of being ignored.
* fix(cost): select the batch long-context tier from any batch tier key
A deployment that declares its own flat standard input rate keeps every
published batch rate of the output direction, including the 272K output
tier, but the tier was only ever selected when an input tier key was also
present. Detect the crossed tier from any of the four batch tier keys so
the carried output, cache-read, and cache-write tiers bill at their tier
rate above 272K tokens.
* chore(proxy): keep the OpenAPI snapshot as CI generates it
* fix(cost): pick each batch price component's tier from its own keys
The batch rate picker crossed one threshold for every component, so a
deployment declaring only an output tier also moved its input, cached, and
cache-write rates to that cutoff. Each component now crosses its own
*_above_<N>k_tokens_batches keys and falls back to its flat key.
The JSON schema is regenerated with the generator as it is on main:
cost-map-guard renders the PR's cost map with the base branch's generator,
so the descriptions for the new batch cache keys move to a follow-up.
* chore(proxy): restore the lazy OpenAPI snapshot to what CI's Python 3.12 generates
The merge commit carried a snapshot regenerated on a Python 3.14 venv, which dedents
docstrings at compile time, so one description line differed from the file CI regenerates
on 3.12 and the schema.d.ts sync check went red. The snapshot is byte-identical to main again
* feat(arize): per-team success and error sampling rates for the Arize AX callback
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(arize): fail open on invalid sampling rates and type the new sampling code
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(arize): type the sampling test helpers and parametrized fixture
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: yassin <yassin@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(proxy): add TinyFish Agent API passthrough with per-step billing
* chore(ui): regenerate dashboard API types for /tinyfish passthrough
* fix(proxy): satisfy strict lint budget for tinyfish passthrough
* style: ruff format tinyfish passthrough handler
* test(proxy): exercise tinyfish route through the app with a faked upstream
* refactor(proxy): make cross-module tinyfish billing hooks public
* fix(proxy): tolerate transient tinyfish poll failures instead of dropping the charge
* fix(proxy): defer billing for disconnected tinyfish SSE runs to the background poller
* Revert "fix(proxy): defer billing for disconnected tinyfish SSE runs to the background poller"
This reverts commit ef0bcfb4a0.
* fix(proxy): bill tinyfish SSE runs via detached poller and only COMPLETED runs
Disconnected run-sse clients previously left completed runs unbilled: the stream-end
handler saw a still-RUNNING run and logged $0. The poller now spawns from the streaming
path on the first run_id frame, outlives the disconnect, and writes the one spend row when
the run turns terminal; the stream-end path only logs the $0 fallback for run_id-less
streams. Costs now apply only to COMPLETED runs ($0 for FAILED/CANCELLED, matching the
upstream invoice), spend rows carry the request's litellm_call_id (previously NULL
request_ids collided and were silently dropped), and the GET /v1/runs listing is blocked
so callers behind the shared key cannot discover each other's runs.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Pxz11TQKYvCXvKawNUa2g4
* fix(proxy): drop GET /v1/runs from the tinyfish allowlist error message
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Pxz11TQKYvCXvKawNUa2g4
* chore(proxy): sync openapi artifacts for tinyfish docstring, suppress LIT011 on flag write
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Pxz11TQKYvCXvKawNUa2g4
* style(proxy): ruff-format the sse poller flag write
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Pxz11TQKYvCXvKawNUa2g4
* style(proxy): keep the rebind-ok suppression on the flag write's own line
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Pxz11TQKYvCXvKawNUa2g4
* fix(proxy): harden tinyfish billing paths from review findings
Skip failure dispatch when the SSE poller owns billing (a failure row collided with the
poller's billed row on request_id and dropped the charge), late-spawn the poller for
run_ids that arrive in unterminated frames instead of mispricing RUNNING runs at $0,
thread litellm_params into poller-billed standard logging objects so SLO consumers see
attribution, untype the run error field so upstream error-shape drift cannot void a
billable run, normalize a schemeless TINYFISH_AGENT_API_BASE, extend the poll budget to
cover queue wait (3600s) with ~60s outage tolerance, and log poller cancellation.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Pxz11TQKYvCXvKawNUa2g4
* chore(proxy): satisfy ratcheted BLE001/LIT002 budgets from main in tinyfish handler
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Pxz11TQKYvCXvKawNUa2g4
* style(proxy): drop stray blank line from merge resolution
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Pxz11TQKYvCXvKawNUa2g4
* fix(proxy): reject passthrough envelope controls on tinyfish route, raise blocking-run timeout
The generic passthrough unwraps a caller-supplied custom_body as the forwarded request and
honors a caller stream flag, so custom_body.use_vault bypassed the credentialed-run 403 and
stream: true flipped a blocking run into the streaming pipeline. The route now 400s the
envelope fields (custom_body, stream, query_params); streaming comes from the endpoint.
Blocking runs also get a 1500s default timeout covering the upstream 1200s run cap, unless
the operator configured pass_through_request_timeout.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Pxz11TQKYvCXvKawNUa2g4
* style(proxy): resolve operator timeout without a dict-literal default
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Pxz11TQKYvCXvKawNUa2g4
* test(passthrough): list the TinyFish route among protocol-constrained pass-through routes
* chore(proxy): regenerate the lazy OpenAPI snapshot after merging main
* chore(proxy): keep the lazy OpenAPI snapshot as CI's Python 3.12 renders it
* fix(proxy): reject TinyFish POST bodies that are not a JSON object
A form-encoded or text body carried stream and use_vault past both field
gates, because the gates only saw fields the body parsed to as JSON. The
route now checks the content type before reading the body and answers 400
for anything that is not a JSON object.
* fix(tinyfish): reject submit paths with extra slashes so run-async always bills
The allowlist dropped empty path segments, so POST /v1/automation/run-async/
was forwarded upstream while the billing dispatch only recognises the exact
path and would have logged the submit at $0 without starting the poller.
Any path with a trailing or doubled slash now returns 403 before forwarding.
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* feat(proxy): opt-in litellm_call_id in JSON error bodies
Add general_settings.include_call_id_in_error_body. When true, the value
already on the x-litellm-call-id response header is copied into JSON error
bodies: as error.litellm_call_id on the OpenAI-shaped routes, /v1/messages,
and streaming first-chunk errors, and as a top-level litellm_call_id on
pass-through routes. Off by default, so error bodies stay byte-identical
unless an admin opts in
* chore(proxy): drop helper docstring and restore lazy OpenAPI snapshot
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(guardrails): scan video prompts for key-attached guardrails on /v1/videos
/v1/videos dispatches call_type avideo_generation, which CallTypes did not
know and no guardrail translation handler covered, so the unified guardrail
hook returned the request unscanned. Add the video call types and an OpenAI
video guardrail translation package that scans the prompt for create, remix,
edit and extension requests
Resolves LIT-6685
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* chore(ui): regenerate api types for video call types
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test: skip avideo_generation in azure sdk client exhaustive check
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(e2e): retry a leaked video job until the guardrail sync deadline
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(guardrails): satisfy the type-discipline gate in the video handler
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(guardrails): gate the video e2e on a chat probe so a miss starts at most one paid job
Addresses Greptile review: typed RewritingGuardrail override, dropped routine docstrings, and the e2e waits for the key guardrail to sync via /chat/completions before its single /v1/videos call
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: yucheng <yucheng@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Password login now stamps login_method=username_password into the UI session key
metadata, and change_password rejects any caller that is not a litellm-dashboard
key carrying that marker with 403 before the user row is read. SSO sessions and
user-associated virtual keys can no longer use the endpoint as a current_password
guessing oracle. The forced-reset session is still minted by the password login
path, so it keeps access to the endpoint
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Register s3_access_key_id, s3_secret_access_key and s3_encryption_key_id as
LiteLLM-owned batch params so they are no longer forwarded to Bedrock as
additionalModelRequestFields (which 400s ordinary chat on a batch-configured
deployment), keep them on CredentialLiteLLMParams so the batch/file paths
still receive them, and redact the S3 credential key names in debug logs.
Resolves LIT-8290
Co-authored-by: yucheng <yucheng@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Register s3_access_key_id, s3_secret_access_key and s3_encryption_key_id as
LiteLLM-owned batch params so they are no longer forwarded to Bedrock as
additionalModelRequestFields (which 400s ordinary chat on a batch-configured
deployment), keep them on CredentialLiteLLMParams so the batch/file paths
still receive them, and redact the S3 credential key names in debug logs.
Resolves LIT-8290
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
/config/field/info and /config/list already report per-key source on main,
so this drops the branch's versions of those and keeps /alerting/settings,
/get/ui_settings and /router/settings.
Read endpoints no longer write the freshly read database row back into the
shared settings store; the reload path already keeps it current, and a GET
that mutates global state leaks across callers.
Regenerates the lazy OpenAPI snapshot on Python 3.12, matching CI, and the
dashboard API types for the two new response fields.
* fix(bedrock): send s3BucketOwner on batch input and output data config
Resolve s3_bucket_owner from litellm_params, then optional_params, then
AWS_S3_BUCKET_OWNER and emit it on both S3 data configs so cross-account
batch buckets pass Bedrock ownership validation. Omitted when unset
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(bedrock): build batch output config with explicit returns
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: yucheng <yucheng@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
