* chore(cost-map): remove models past their deprecation date
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(cost-calc): drop the empty parametrize left behind by the gemini web search removal
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(cost-map): drop merge base block left by conflict resolution
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(cost-calc): drop gemini image cost tests pinned on removed model
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: kerry <kerry@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(fal_ai): align /fal_ai queue gate with the pricer and normalise resolution type
The /fal_ai gate admitted catalog keys the Fal pricer cannot price, so
those jobs were forwarded and logged at 0.0 spend. The gate now reuses
the pricer as its eligibility predicate. Resolution is normalised to a
string before the keyed price lookup so int and str spellings bill the
same.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(fal_ai): assert passthrough pricing invariants on synthetic catalog entries
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(fal_ai): price the submitted body in the /fal_ai queue gate so keyed-only entries are admitted
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(fal_ai): move the queue gate repro to tests/integration
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: kerry <kerry@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(proxy): add TinyFish Agent API passthrough with per-step billing
* chore(ui): regenerate dashboard API types for /tinyfish passthrough
* fix(proxy): satisfy strict lint budget for tinyfish passthrough
* style: ruff format tinyfish passthrough handler
* test(proxy): exercise tinyfish route through the app with a faked upstream
* refactor(proxy): make cross-module tinyfish billing hooks public
* fix(proxy): tolerate transient tinyfish poll failures instead of dropping the charge
* fix(proxy): defer billing for disconnected tinyfish SSE runs to the background poller
* Revert "fix(proxy): defer billing for disconnected tinyfish SSE runs to the background poller"
This reverts commit ef0bcfb4a0.
* fix(proxy): bill tinyfish SSE runs via detached poller and only COMPLETED runs
Disconnected run-sse clients previously left completed runs unbilled: the stream-end
handler saw a still-RUNNING run and logged $0. The poller now spawns from the streaming
path on the first run_id frame, outlives the disconnect, and writes the one spend row when
the run turns terminal; the stream-end path only logs the $0 fallback for run_id-less
streams. Costs now apply only to COMPLETED runs ($0 for FAILED/CANCELLED, matching the
upstream invoice), spend rows carry the request's litellm_call_id (previously NULL
request_ids collided and were silently dropped), and the GET /v1/runs listing is blocked
so callers behind the shared key cannot discover each other's runs.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Pxz11TQKYvCXvKawNUa2g4
* fix(proxy): drop GET /v1/runs from the tinyfish allowlist error message
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Pxz11TQKYvCXvKawNUa2g4
* chore(proxy): sync openapi artifacts for tinyfish docstring, suppress LIT011 on flag write
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Pxz11TQKYvCXvKawNUa2g4
* style(proxy): ruff-format the sse poller flag write
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Pxz11TQKYvCXvKawNUa2g4
* style(proxy): keep the rebind-ok suppression on the flag write's own line
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Pxz11TQKYvCXvKawNUa2g4
* fix(proxy): harden tinyfish billing paths from review findings
Skip failure dispatch when the SSE poller owns billing (a failure row collided with the
poller's billed row on request_id and dropped the charge), late-spawn the poller for
run_ids that arrive in unterminated frames instead of mispricing RUNNING runs at $0,
thread litellm_params into poller-billed standard logging objects so SLO consumers see
attribution, untype the run error field so upstream error-shape drift cannot void a
billable run, normalize a schemeless TINYFISH_AGENT_API_BASE, extend the poll budget to
cover queue wait (3600s) with ~60s outage tolerance, and log poller cancellation.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Pxz11TQKYvCXvKawNUa2g4
* chore(proxy): satisfy ratcheted BLE001/LIT002 budgets from main in tinyfish handler
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Pxz11TQKYvCXvKawNUa2g4
* style(proxy): drop stray blank line from merge resolution
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Pxz11TQKYvCXvKawNUa2g4
* fix(proxy): reject passthrough envelope controls on tinyfish route, raise blocking-run timeout
The generic passthrough unwraps a caller-supplied custom_body as the forwarded request and
honors a caller stream flag, so custom_body.use_vault bypassed the credentialed-run 403 and
stream: true flipped a blocking run into the streaming pipeline. The route now 400s the
envelope fields (custom_body, stream, query_params); streaming comes from the endpoint.
Blocking runs also get a 1500s default timeout covering the upstream 1200s run cap, unless
the operator configured pass_through_request_timeout.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Pxz11TQKYvCXvKawNUa2g4
* style(proxy): resolve operator timeout without a dict-literal default
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Pxz11TQKYvCXvKawNUa2g4
* test(passthrough): list the TinyFish route among protocol-constrained pass-through routes
* chore(proxy): regenerate the lazy OpenAPI snapshot after merging main
* chore(proxy): keep the lazy OpenAPI snapshot as CI's Python 3.12 renders it
* fix(proxy): reject TinyFish POST bodies that are not a JSON object
A form-encoded or text body carried stream and use_vault past both field
gates, because the gates only saw fields the body parsed to as JSON. The
route now checks the content type before reading the body and answers 400
for anything that is not a JSON object.
* fix(tinyfish): reject submit paths with extra slashes so run-async always bills
The allowlist dropped empty path segments, so POST /v1/automation/run-async/
was forwarded upstream while the billing dispatch only recognises the exact
path and would have logged the submit at $0 without starting the poller.
Any path with a trailing or doubled slash now returns 403 before forwarding.
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(proxy): release unclaimed budget reservations at request end
* fix(proxy): release unclaimed budget reservations of websocket sessions too
* test(proxy): drop the structural middleware inheritance check
* fix(proxy): claim the budget reservation on streaming pass-through before its cost callback
The SSE chunk processor hands its success handler to the logging worker
after the response, so the request-end release freed the reservation
first and left the key unguarded until the worker drained. Claim it at
both end-of-stream hand-offs, the immediate enqueue and the coroutine
parked for deferred dispatch.
Give the xai realtime test double the litellm_params attribute every
real Logging object carries, since the wrapper now reads it.
* test(pass-through): give the vertex streaming test doubles a litellm_params dict
The spec'd Logging mocks in test_vertex_ai_anthropic_streaming_cost_injection.py
lacked the instance attribute the chunk processor now reads to claim the budget
reservation. Also restores main's _lazy_openapi_snapshot.json: the branch's copy
had been regenerated under Python 3.14, which dedents one docstring description
that the CI regeneration on Python 3.12 keeps indented, and the PR adds no lazily
loaded route, so main's file is the correct one.
* fix(pass-through): claim the budget reservation only after its cost callback is enqueued
Every pass-through success hand-off stamped callback_bound before handing the
coroutine to the logging worker. When that enqueue raised, the reservation stayed
claimed with no callback left to reconcile it, so the request-end release skipped it
and the reserved cost stayed pinned on the key's counter. Enqueue first, then claim,
so a failed hand-off leaves the reservation for the request-end release.
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* feat(proxy): opt-in litellm_call_id in JSON error bodies
Add general_settings.include_call_id_in_error_body. When true, the value
already on the x-litellm-call-id response header is copied into JSON error
bodies: as error.litellm_call_id on the OpenAI-shaped routes, /v1/messages,
and streaming first-chunk errors, and as a top-level litellm_call_id on
pass-through routes. Off by default, so error bodies stay byte-identical
unless an admin opts in
* chore(proxy): drop helper docstring and restore lazy OpenAPI snapshot
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
With store_prompts_in_spend_logs on, the persisted request body kept the client's model string even when the row's model, model_group, and error text had been replaced by the unknown-model placeholder. The body's model now takes the same placeholder on those rows. Also annotates the new test locals with Final and wraps the four test lines that ran past 120 characters.
Pass-through requests inject the proxy span into upstream headers since #40669, which
replaced an explicit x-pass-traceparent with an unrelated trace and dropped its
x-pass-tracestate. Keep the caller's context when the carrier already names a
different trace, and keep the proxy child span for same-trace or missing headers.
Co-authored-by: yucheng <yucheng@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Authorization and pricing read the first model and language query value, but the raw query was forwarded, so Deepgram (which honours the last repeated value) could be sent a model the key was never allowed. Later duplicates of those two keys are now dropped before the upstream URL is built
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
A caller could pick a model with only a pre-recorded registry row, or no row at all, and the session would be billed at the pre-recorded rate or logged at zero cost, so budgets did not apply. The route now closes the WebSocket with 1008 before dialing Deepgram unless deepgram/streaming/<model> (or the -multilingual row for language=multi) is an exact registry hit, and the logging handler applies the same check so a registry change under a live session records the duration with no cost instead of a substitute rate
Regression tests cover the route refusal, an operator-supplied streaming row for another model being accepted, and the handler never substituting the pre-recorded rate
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Resolve dot segments in the /azure_speech endpoint path before the endpoint family and the admin-only batch guard are decided, so the guard and the forwarded upstream path agree. Bill short-audio requests for the longer of the uploaded audio duration and the recognized duration, so a NoMatch or silence response still charges for the audio Azure processed
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Pass-through requests inject the proxy span into upstream headers since #40669, which
replaced an explicit x-pass-traceparent with an unrelated trace and dropped its
x-pass-tracestate. Keep the caller's context when the carrier already names a
different trace, and keep the proxy child span for same-trace or missing headers.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
The route test only resolved route names. It now sends every method through
the proxy with a virtual key and asserts the upstream receives that method,
the proxy's TypeSafe key and the caller's body
Non-admin keys may only start transcription jobs whose media and transcript output live in the S3 buckets listed in general_settings.transcribe_media_buckets, and may not supply DataAccessRoleArn or JobExecutionSettings. The setting is editable from the Admin UI general settings table (new List editor) and DB values load into the running proxy when config.yaml does not set it
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
#41607 registered the typesafe pass-through with a route that only accepted
GET and POST, so a PUT, DELETE or PATCH to /typesafe/... came back 405
before reaching the upstream. CircleCI's pass-through method test caught it,
but that lane does not run on the PR gate, so the mapped unit test now
covers the same invariant for typesafe
The same CircleCI run also failed test_models_by_provider because typesafe
is not a key of models_by_provider. Registering it there would satisfy the
assertion without changing behaviour: typesafe has no LlmProviders member,
so a typesafe/* deployment never loads and get_valid_models returns nothing,
and its spend is priced straight from model_cost. The test already skips
search-mode providers for that reason, so it now skips evaluation mode too
Restrict signed media fetches to https URLs, treat a job AWS no longer knows as
priceable from the media named in its StartTranscriptionJob response instead of
polling to the eight hour maximum, and read the media length with libsndfile
headers instead of decoding the whole file into memory
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Standard jobs are tagged litellm-owner on StartTranscriptionJob so GetTranscriptionJob
and DeleteTranscriptionJob only work for the owner or a proxy admin, and account-wide
operations need a proxy admin. Media rewritten after job creation is charged the eight
hour maximum, and the success handler takes an injected log dispatch instead of tests
patching its private method
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Virtual-hosted URLs for bucket names containing dots fail TLS verification, so the
media duration fetch failed and completed jobs were charged the eight hour maximum
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Under xdist the per-test litellm reload is skipped, so a leaked max_budget from another proxy test sent the real
key auth path into the global spend lookup, which the MagicMock prisma client cannot await
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Amazon Transcribe bills every second of the media file, silence included, while the
transcript's last end_time stops at the last word, so pricing from the transcript
undercharged. After a job completes, download Media.MediaFileUri from S3 with the
proxy's credentials and read its length with libsndfile. Formats libsndfile cannot
read (mp4, m4a, webm, amr) and custom language models under LanguageIdSettings are
refused before signing. The S3 signature is only sent to hosts in the AWS partition
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Key auth on the Deepgram WebSocket route now sees the same model the upstream target will carry, so a key restricted to other models can no longer reach nova-3 by leaving model out of the query. user_api_key_auth_websocket keeps its signature and delegates to user_api_key_auth_websocket_for_model, which the Deepgram route calls with deepgram_listen_requested_model
Sessions are priced from new deepgram/streaming/* registry rows (nova-3, nova-3-multilingual for language=multi) plus per-minute add-on rows for redact, keyterm, detect_entities and diarize, all read from Deepgram's pricing page on 2026-09-17. Models without a streaming row fall back to their pre-recorded row as before
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Ordinary keys could read, patch and delete batch transcription jobs that other keys created with the proxy's shared Azure subscription, so every /speechtotext/v3.2 method is now admin only while fast transcription stays open. Also clears SERVER_ROOT_PATH in the real-auth test helper because test_custom_proxy leaves it set at import time and the shared app then 404s pass-through routes
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
StartTranscriptionJob was logged with response_cost 0.0, so key, team and proxy
budgets never stopped repeated jobs on the proxy's AWS credentials. The success
handler now polls GetTranscriptionJob to completion, reads the audio duration
from the transcript artifact and charges whole seconds at the cost map rate,
charging the longest media AWS accepts when the duration cannot be read. The
route refuses job classes and surcharge features the cost map does not price
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Deepgram charges for the total processed audio across channels, so a stereo /listen session with multichannel=true costs twice its duration. The handler now multiplies the session duration by a validated channel count taken from Metadata.channels, then the widest Results channel_index, then the channels query parameter, defaulting to one. Booleans, floats, strings, zero and negative values are ignored
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
With callback or callback_method in the query, Deepgram sends every Results and Metadata frame to the caller's URL and only a request id down this socket, so the proxy would meter zero seconds of audio while its own Deepgram credential paid for the transcription. The route now closes such connections with 1008 before contacting Deepgram, naming the offending parameters in the close reason. Adds helper and route tests for both parameters and a nine mutation sweep, all killed
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
