1144 commits

Author	SHA1	Message	Date
Devin AI	53d3f13297	Merge remote-tracking branch 'origin/main' into litellm_test_model_endpoint	2026-11-05 02:20:30 +00:00
ryan-crabbe-berri	42b4cfc5bf	fix(nvidia_riva): send the api key exactly once by calling riva over grpc directly (#50140) ... * fix(nvidia_riva): send the api key once per call by talking to Riva over grpc directly Drops nvidia-riva-client from the stt-nvidia-riva extra. Depending on its version it sent the bearer token zero times on plaintext or twice on TLS, so token-checking Riva deployments rejected every request (#50059). The provider now declares the few Riva messages it uses, opens its own grpc channel and puts authorization and function-id on each call exactly once * test(nvidia_riva): run the provider against a real grpc server built from nvidia's own definitions * fix(nvidia_riva): validate endpointing settings with pydantic before building the riva message * test(nvidia_riva): check the key is sent once over tls, and drop a comment that restated the code	2026-11-04 18:07:31 -08:00
joshua-berri	906fa7ed28	feat(accounting): add opt-in SpendLogs ledger (#49856) ... * feat(accounting): add opt-in SpendLogs ledger delivery * fix(accounting): preserve missing organization daily spend parity * fix(accounting): preserve ledger receipts when replaying migration * fix(accounting): preserve callback windows and tune ledger batches * fix(accounting): preserve window charges and bound ledger statements * test(accounting): verify budget callback registration order * test(accounting): verify captured charges after repeated startup * feat(accounting): store durable payloads independently of spend logs * fix(accounting): preserve first delivery and trace ledger writes * fix(accounting): keep SDK imports independent of proxy tracing * fix(accounting): honor log retention during batch takeover * test(accounting): cover retained batch alerts and invalid settings * test(accounting): verify downgrade command against pending work * fix(accounting): drain retries without starving fresh deliveries * test(accounting): cover delivery projection, metrics and window key reads Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test(accounting): read delivery metrics without the shared registry Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(accounting): charge a leftover $0 batch row when its claim fails A failed batch claim buffers the retrieve's row and reports it charged. At flush the row conflicted with the pre-upgrade $0 SpendLogs row, so its delivery was rolled back and the charge never reached the spend tables. The flush now takes that row over in the same transaction and keeps the delivery. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(accounting): trace the batch takeover at flush as a SpendLogs update Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(accounting): keep local spend logs when SPEND_LOGS_URL has no export client The export client is never created, so main always writes SpendLogs to the proxy's own database even with SPEND_LOGS_URL set. The ledger skipped local rows whenever the URL was set, which dropped those request logs. It now treats a flush as an export only when the client exists, and batch claims write local rows as before. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(accounting): keep every request's charge when its log row is shared, evicted or rejected - Key non-batch deliveries on a per-request delivery_id so a reused client call id is still charged once per request, while buffer replays stay deduplicated - Persist the redacted api_key instead of an opaque bearer token - Keep the accounting payload when the spend-log queue evicts a row or Postgres rejects it - Strip NUL from payloads and skip non-text tags so projection never parks a paid request Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(accounting): isolate a raw-SQL spend log row Postgres rejects instead of requeueing the batch A rejected row on the ledger insert path raises RawQueryError, which the exact DataError check missed, so the whole batch was requeued every flush. Data and integrity SQLSTATE classes now bisect like before, while missing tables and deadlocks keep retrying Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(accounting): keep charging a request whose tag is too long for the daily tag index A tag past Postgres's btree entry limit failed the DailyTagSpend upsert, and on the ledger path that rolled back the request's key, team and user charges with it. Such tags are now left out of the daily tag rollup only Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(accounting): store only guardrail billing fields in the accounting payload The ledger payload copied whole guardrail records, including responses and match details, into LiteLLM_SpendLogDelivery even when spend logs were disabled. It now keeps the id, name, usage and cost fields the projection reads. NUL stripping reuses safe_json_structure instead of a recursive walk Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * refactor(proxy): move spend payload stamping out of update_database Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test(proxy): restore the modules the import-layering guard evicts The guard popped litellm.proxy.utils from sys.modules for good, so later tests on the same xdist worker patched a fresh module while calling functions bound to the old one, and spend queue tests failed depending on scheduling Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * refactor(proxy): keep update_database's payload stamping on its own local Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * refactor(accounting): type guardrail entries and use a public key redaction helper Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(accounting): keep charging requests whose end-user id is too long for an index An oversized end-user id failed the SpendLogs insert with SQLSTATE 54000, which was requeued forever, and failed the end-user upsert in the delivery, which parked the key and team charge with it. Class 54 rows are now isolated like other rejected rows, and such ids are left out of the end-user rollups only Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * refactor(proxy): type the spend log queue snapshot Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(accounting): leave out any projected row Postgres cannot index or store Caller-supplied ids and counters, such as a long tag or end-user id or an out-of-range compression count, failed the delivery transaction and rolled back the request's key, team and user charges. One check in project() now drops only the row that cannot be stored, replacing the per-field cutoffs Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(accounting): keep the guardrail fields compression savings are counted from Slimming the guardrail records dropped the provider and the tokens_saved figure that Headroom compression savings are read from, and dropped a bare-dict record entirely. The payload now keeps both while still leaving out guardrail content Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(accounting): cap projected counters so daily totals cannot overflow Counters such as a caller-supplied compression count fit one by one but overflowed the bigint daily total once summed across requests, rolling back the key, team and user charges in the same transaction. Each counter is now capped at 2^40, far below any total that could overflow Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(accounting): let a buffered batch charge take over its zero-cost receipt A batch charge queued after a failed lookup hit the $0 receipt an earlier poll left behind, and ON CONFLICT dropped it. The buffered insert now runs the same receipt takeover as a live retrieve Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(accounting): read a rescued batch row's spend from its accounting payload Rows rescued from an overflowing queue keep only the accounting payload, so the buffered takeover never saw their positive spend and left the $0 receipt uncharged Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(accounting): read budget windows through the delivery transaction Window seed and existing-row reads used a separate connection while the delivery transaction held one, so with a one-connection pool they waited on their own transaction until it timed out. Inside an enclosing transaction they now share its connection Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(accounting): retry a transient failure while keeping a dropped row's charge The rescue swallowed every error, so a database blip right after a row was rejected lost the unpaid charge. Only a rejected row is still logged and skipped, anything else propagates so the flush retries Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test(accounting): give delivery projection payloads their charged entities project() now reads entities only from accounting_entities, so the hand-built delivery payloads carry them like real stored payloads do. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * refactor(accounting): own ledger rows by their payload and key batches on the batch id Drop the accounting_path stamp and its SpendLogs column. A buffered row that carries an accounting payload is ledger owned, one without it is legacy, and spend counters get a plain windows_from_ledger flag. Window seeding tells applied ledger rows apart by their delivery row. A ledger batch is plain transaction A keyed by the batch id, so a repeated completed retrieve conflicts and charges nothing. A $0 retrieve records no delivery, a batch a legacy retrieve already charged is not charged again, and the legacy batch path is unchanged from main apart from refusing a batch the ledger already owns. Batch takeover and its delta bookkeeping are removed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test(accounting): drop a projection test duplicated while rebasing Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * refactor(accounting): drop previous_payload now that deliveries are never replaced Batch takeover was the only writer of previous_payload, so the worker applies each delivery's payload as stored and the column leaves the migration. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * refactor(accounting): drop the ledger downgrade checklist script The use_spend_logs_ledger flag is the rollback path, so a separate downgrade gate is no longer needed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(accounting): name the window seed reads so their Prisma span resolves Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * refactor(accounting): remove the unused accounting event types and protocols types.py, protocols.py and measurements.py were left from the old event scaffolding. Nothing outside the package imports them, so they go with their tests. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(accounting): park only deliveries whose own data fails A delivery that keeps failing on a database outage, lock timeout or deadlock stays pending and retries with capped backoff, so a recovered database applies it. Only a data error or an invalid payload parks the isolated row. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(accounting): isolate a spend log row the insert statement rejects The raw SpendLogs insert validates columns and JSON before Postgres sees the row, so a row that fails there is now bisected out and dropped alone while its charge is kept, instead of failing and replaying the whole flush. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(accounting): carry a job's model window on every completed poll in ledger mode The ledger keeps the poll whose delivery claims the job, which need not be the poll that won the limiter's in-memory charge. Every completed poll now carries the job's window, so the delivery that lands always has it, while the live model budget is still charged once. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * refactor(accounting): leave model-budget windows on the existing window writer Model-budget windows are written by the model budget limiter rather than derived from a request's spend, so they stay on main's window queue like live budget counters do. This drops window capture from the accounting payload, the deferred limiter registration and the job-window capture, and the ledger keeps owning key and team budget windows. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(accounting): keep a delivery pending when the database is unreachable Prisma reports a can't reach database server outage as a DataError, so the worker now checks for an unavailable database before treating a data error as the delivery's own fault. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(accounting): isolate only rows the raw insert cannot encode Catching every ValueError also swallowed unexpected insert errors that main re-raises. The raw insert now raises RejectedSpendLogRow for an unknown column or non-JSON value, and only that is isolated as a rejected row. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(accounting): store zero savings when caller data makes them uncomputable A huge caller-supplied tokens_saved overflowed the savings math inside build_accounting_payload, which dropped the request's charge in ledger mode. Savings that cannot be computed are now recorded as zero and the charge lands. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Joshua Valluru <326636767+joshua-berri@users.noreply.github.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-04 17:49:15 -08:00
Mateo Wang	7a641e5172	fix(proxy): answer an upstream overload with 503 server_is_overloaded on OpenAI routes (#49993) ... * fix(anthropic): answer a 503 as 529 overloaded_error on /v1/messages Anthropic's error contract has no 503, so a client built against it (Claude Code) treats a 503 from /v1/messages as a plain 5xx and retries it ten times instead of taking its overload path. A Bedrock 503 now leaves /v1/messages and /v1/messages/count_tokens as 529 overloaded_error, for both a JSON answer and a stream that fails before its first byte. An Anthropic 529 overloaded_error now maps to litellm.ServiceUnavailableError (503) instead of InternalServerError (500), so OpenAI-shaped routes answer it as 503, the status OpenAI documents for an overloaded engine, and router retry and allowed-fails policies key it as ServiceUnavailableError. * test(proxy): expect overloaded_error on a 503 stream error after a keepalive ping * fix(proxy): record the 529 the Anthropic routes answer and type the chat stream error frame The messages route stamps the answered status on the exception before post_call_failure_hook runs, so the spend log row, the OTel server span, the failed-request metric, and alerting record the 529 the client saw instead of the deployment's 503, while the deployment failure metric and cooldown keep the deployment's own status. A chat completions stream error frame names the type its status stands for instead of the null .type ServiceUnavailableError carries. An Anthropic-shaped 503 envelope passes through as 529 overloaded_error through a validated TypedDict, and count_tokens answers every ProxyException as the Anthropic envelope. Five integration tests that pinned 503 on the messages route now expect 529 * test(proxy): expect the top-level Anthropic envelope from count_tokens errors count_tokens now answers a ProxyException as a JSONResponse carrying the Anthropic error envelope instead of raising HTTPException, which FastAPI wrapped in {"detail": ...}. The three legacy error-format tests now read the response body and check that no detail wrap is left. * test(proxy): cover the 529 overloaded_error answer across routes, cooldown, and retry budgets * fix(anthropic): keep Anthropic overloads as InternalServerError and answer 529 from the provider status An upstream Anthropic 529 stays litellm.InternalServerError (status 500) so InternalServerErrorRetries, InternalServerErrorAllowedFails, SDK handlers, and the OpenAI-shaped routes behave as on main. The error now carries provider_status_code, and the /v1/messages routes answer 529 overloaded_error from it. * refactor(proxy): unwrap a mid-stream fallback error once instead of recursing for the provider status * fix(anthropic): keep count_tokens' detail body, end keepalive streams with an Anthropic error event, and log any 529 as an overload * test(integration): cover the audit round 3 cells and expect 529 on the count_tokens 503 cases * fix(proxy): re-raise an unchanged stream HTTPException as itself and walk nested mid-stream fallbacks for the provider status * fix(proxy): walk nested mid-stream fallbacks without recursion * fix(anthropic): preserve 503s and answer upstream 529s on messages routes Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(anthropic): answer an upstream Anthropic 529 as 529 on count_tokens Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(anthropic): use integration coverage for keepalive errors Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(proxy): answer an upstream overload with 503 server_is_overloaded on OpenAI routes Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(proxy): pass answered overload status to failure hooks Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(proxy): preserve Anthropic route exception handling Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * refactor(proxy): bind the transformed failure without rebinding the parameter Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(proxy): observe the overload failure hook through a real ProxyLogging Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-11-04 16:37:55 -08:00
Oliver Yu	733b776403	fix(least-busy): reserve the picked deployment so a burst of requests spreads out (#47484) ... * fix(least-busy): reserve the picked deployment so a burst of requests spreads out The in-flight count only went up in log_pre_api_call, which runs after selection and the pre-call hooks. Requests selected in that window all saw the same counts and landed on the same deployment (16 simultaneous requests over 4 deployments went as lopsided as 12/4/0/0 in the proxy). When the router selects for a request it is about to send, least-busy now records a reservation for the chosen deployment in the same synchronous step that reads the reservations, and log_pre_api_call turns it into the normal in-flight count. Selections without request kwargs (token counting, probes) do not reserve. A reservation whose request never reaches the API call expires after 30 s. The shared and per-worker counts and their success/failure decrements are unchanged. * fix(least-busy): prune expired and empty selection reservations The reservation map kept a key per model group forever, so a wildcard route with caller-chosen model names grew it without bound. Drop expired and empty entries on every write. Signed-off-by: Oliver Yu <tc3oliver@gmail.com> * fix(least-busy): keep pick reservations in the shared router cache Selectors that share a router cache dedupe to one registered callback, so only that instance saw log_pre_api_call. Reservations held on the other instances were never released and skewed picks for the full TTL. Storing them in the router cache's in-memory layer lets any selector release them, and the per-key TTL replaces the full-map rebuild on every write. Tests now hold the burst with a barrier instead of a sleep and use a manual clock object. * fix(least-busy): restart the reservation TTL on every write InMemoryCache keeps the TTL a live key was first given, so a reservation key that kept being rewritten still expired 30 s after its first pick and took the newer reservations with it. Drop the key before each write so the TTL counts from the latest one, and leave it deleted once it is empty. Signed-off-by: Oliver Yu <tc3oliver@gmail.com> * fix(router): count least-busy decisions calls and release reservations on cache hits Least-busy now reads the model group from litellm_metadata before metadata, so /v1/decisions calls that carry caller metadata are counted in flight and release their reservation at pre-call. A response cache hit releases its reservation instead of decrementing another request's in-flight count, and Responses WebSocket turn admission selects without reserving. The reservation store is reached through a typed protocol. * fix(decisions): count proxy /v1/decisions calls in least-busy's in-flight totals * fix(least-busy): keep reservations out of the shared router cache and count requests that start during a Redis read Least-busy now reads a call's deployment from the routing metadata the router sets, so /v1/decisions logging params stay as on main and failed decisions calls no longer start cooldowns. * style(router): wrap the least-busy reservation store line * fix(least-busy): lock the reservation read-and-update so threaded sync calls keep each other's picks Router.completion from a thread pool and sync success callbacks can read and rewrite a deployment's reservations at the same time, so one thread's pick or release could be lost. * fix(least-busy): release a pick whose call ends before reaching the provider A call that fails or returns before the provider handoff (a 400 for an unsupported param, token counting, a proxy-gate rejection without a start log, batch create) now drops its reservation instead of decrementing a count it never added. Router.select picks and batch cancel no longer reserve, since neither makes a call that logs. * test(least-busy): give callback fixtures a fixed call-start time The routing-group tests fed least-busy a finished call without the start time litellm stamps before counting it, so the count stayed up after the fix. Fixtures now use a fixed timestamp instead of the clock. * fix(least-busy): free tier-group picks, stop lookups reserving, and count only started attempts Reservations are keyed by deployment id, so a complexity, auto, adaptive or quality router frees its tier pick when the call starts. Each started attempt is recorded under its call id, and only a recorded start lowers the count when the attempt ends, so a fallback hop rejected before the provider frees its pick. Whether a pick reserves now comes from a context variable set by the routed call sites, so get_available_deployment and the deprecated async_get_available_deployment no longer hold a deployment, and custom routing strategies and Router subclasses keep their method signatures. * fix(least-busy): free a request's unstarted picks when it ends Each pick now carries its call id, so a start frees only its own call's pick, and the router frees every pick a request never started once that request ends. A fallback hop rejected before the provider no longer leaves its pick reserved for 30 seconds. * fix(router): free a least-busy pick when a vector store create ends * fix(router): free least-busy picks of calls made without a call id * fix(router): free the least-busy pick of a sync text completion * test(router): use fixed start times in least-busy fixtures * fix(router): free the least-busy pick of a sync generic call that fails before its provider * fix(router): read in-flight counts and start attempts under one lock so a busy deployment never looks free * fix(router): keep pass-through lookups from reserving least-busy picks * test(least-busy): wait for the helper thread to start before the timed join * fix(least-busy): keep realtime sessions out of in-flight counts --------- Signed-off-by: Oliver Yu <tc3oliver@gmail.com> Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-11-05 00:30:12 +00:00
Devin AI	fd91ab9b82	fix(health): run every /health/test_model turn through the proxy pre-call hooks so key rate limits apply	2026-11-04 23:29:56 +00:00
Devin AI	2cbd949083	fix(health): read multipart chat replies, require turn 3 for prompt_caching, drop the turn 3 cache retry, enforce budgets on /health/test_model	2026-11-04 23:10:30 +00:00
devin-ai-integration[bot]	81e8d652b0	test(observability): reserve sink port throughout outage (#50100) ... Co-authored-by: mateo <277851410+mateo-berri@users.noreply.github.com> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-11-04 21:07:21 +00:00
mrinal-berri	6d60bc4929	test(integration): poll for the cached JWT rejection sentinel instead of a single Redis read (#50014) ... Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-11-04 11:46:38 -08:00
Kerry Lu	b662f3f7a4	feat(decisions): add EmpirioLabs AI as a decisions provider (#49977) ... * feat(decisions): add EmpirioLabs AI as a decisions provider empiriolabs/aplomb-1 now works with litellm.decisions and litellm.systemone, the proxy /v1/decisions and /v1/systemone routes, the decision model guardrail and the Admin UI. Requests go to POST /v1/decisions on the EmpirioLabs AI API with the System One body, keyed by EMPIRIOLABS_API_KEY with an optional EMPIRIOLABS_API_BASE Chat models keep routing through the existing OpenAI-compatible JSON provider, and a test pins that * ci: run the EmpirioLabs AI decisions tests in the llms unit shard * fix: align EmpirioLabs registration with CI gates --------- Co-authored-by: Adam Dalloul <47503782+Adam-Dalloul@users.noreply.github.com> Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-11-04 10:41:16 -08:00
joshua-berri	02b8da2f16	refactor: replace accounting event scaffolding with SpendLogs projection (#49843) ... * refactor(accounting): replace event scaffolding with SpendLogs projection * test: preserve spend timestamps without removed capture callback * fix: guard accounting event removal across schema update paths * refactor(accounting): retain schema and prepare durable payloads * fix(accounting): preserve dates in serialized payloads * fix(accounting): retain schema push compatibility * fix(accounting): retain span registration for the legacy schema * test(accounting): remove obsolete capture acknowledgement test * fix(accounting): date every daily projection row the way the model rollup does A space-separated startTime kept the whole timestamp as the daily key, while the model usage rollup took the calendar date. Every daily row now uses the same date helper Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * chore(ui): regenerate dashboard API types after rebasing on main Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * refactor(accounting): project only from the stored accounting payload project() now reads entities only from the payload's accounting_entities, so the raw SpendLogs fallback branches are gone. build_accounting_payload() builds the metadata once instead of dumping it to JSON and parsing it back. Guardrail usage dates use a tolerant UTC parse, so a malformed startTime drops only the guardrail rows instead of failing the whole projection. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Joshua Valluru <326636767+joshua-berri@users.noreply.github.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-04 10:37:07 -08:00
yuneng-jiang	b18188dee2	test(e2e): retire the quiet_stack lock and batch the claude_code model registration (#49984) ... * test(e2e): retire the quiet_stack lock quiet_stack held an exclusive host-wide lock for six tests, draining every other xdist worker while they ran. On the full e2e lane that left 35-38% of worker time idle. Each of those tests now lives where it can run without pausing the suite - the two memory tests move to perf/ behind the perf marker, so the perf lane runs them on its own fresh stack. Their refusing deployments use a literal key because the perf stack carries no provider keys - the stored-request size check from the memory test becomes an integration test on an owned proxy that stores prompts - the discount and margin spend-row tests are dropped from e2e. Their breakdown assertions join the existing integration pricing test, which already owns the discount and margin repricing flow - the Bedrock guardrail cost test stays in e2e without the pricing reset fixture, since nothing else on the stack writes pricing config now - the MCP semantic tool filter test turns on a proxy-wide setting, so it gets a new proxy_wide_config opt-in marker and runs after the main run * test(e2e): register the claude_code deployments as one batch The claude_code session fixture registered its 24 deployments one at a time on every xdist worker, and each one waited out the 15 s propagation window on its own, about 6 minutes per worker. ProxyClient.register_models posts the whole batch, polls each model until every replica serves it, then waits out propagation once from the last write. A model that fails to post or never becomes servable comes back as ModelNotRegistered, and an unservable one is deleted, so the fixture keeps reporting per-deployment failures the way it did	2026-11-04 09:55:53 -08:00
Mateo Wang	80b18d165b	chore(proxy): use neutral example names for baggage tag mappings (#50011)	2026-11-04 14:49:55 +00:00
devin-ai-integration[bot]	527d882508	test(responses): isolate cancel wire checks from background polling (#50002) ... Co-authored-by: mateo <277851410+mateo-berri@users.noreply.github.com> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-11-04 05:12:01 +00:00
devin-ai-integration[bot]	0150896a50	test(spend): run explained reads within real read transactions (#49996) ... Co-authored-by: mateo <277851410+mateo-berri@users.noreply.github.com> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-11-04 05:11:23 +00:00
tavren27	b9ad61d4f6	fix(agentic)!: surface a failed agentic follow-up call instead of the pre-loop tool call (#49227) ... * fix(agentic): surface a failed agentic follow-up call instead of the pre-loop tool call * fix(agentic): surface failed follow-ups from the OpenAI chat dispatcher and update pinned integration tests * fix(agentic): end a capped agentic turn on chat and responses instead of a retried 500 * fix(openai): run agentic hooks after the first call's error handling so a follow-up error keeps its own type and message * fix(agentic loop): end capped turns on every chat path and bill the first call when a follow-up fails A turn refused at max_agentic_loops now drops the internal tool calls of every hook that claims calls in it, while a call the client declared survives. The hosted_vllm and openai chat paths now carry the loop depth, fingerprints and max_agentic_loops into the follow-up, so they stop at the cap instead of looping forever. A failed follow-up call now records the first call's usage and cost on the request's failure log before the error reaches the client. A code interpreter sandbox that is missing or fails becomes the tool call's output for the model to handle, so the turn completes without a retry. On native streaming /v1/messages a hook or follow-up failure now reaches the client as an Anthropic error event with the mapped type, and compresr declares its retrieve tool so the stream holds it back. Comments and docstrings the branch added against the repo rule are removed * fix(agentic loop): bill retried attempts and clean up what failed or capped turns return The router's retry loop now carries each failed attempt's paid first call into the request's billing, so a request whose follow-up keeps failing bills every first call on its failure row, and one that recovers bills the failed attempt's first call on its success row. A failure row no longer adds unpriced estimated tokens on top of attempts that were billed, so its tokens match its spend A sandbox failure now answers the model with a fixed sentence and marks the code_interpreter_call as failed, without the sandbox's error text. Code interpreter on /v1/responses with Claude now runs its loop, because the bridged inner call no longer strips the hook's own markers Dropped internal tool calls leave the kept calls numbered from 0, also on a replayed stream. A capped native /v1/messages stream now says the loop hit its limit and that max_agentic_loops raises it, instead of asking the client to retry The agentic refusal and hook error logs no longer include the model or the exception text, and the agentic loop helpers take typed messages, params, and read-only fingerprints * fix(agentic loop): keep a recovered retry's cost breakdown on the response it returns A retried agentic request that recovers no longer adds the failed attempt's first call to its success row. The proxy keeps the follow-up's own row for that request id, so the added spend never showed up in the stored row, and that duplicate row is tracked separately. When every attempt fails, the failure row still bills each attempt's first call Pricing a first call for the failure row no longer replaces the request's cost breakdown. Before, a recovered retry answered with x-litellm-response-cost-original, -input, and -output from the failed attempt's first call. Now its headers and logged breakdown match a request that succeeds on its first try * fix(agentic): stop retrying a request whose agentic follow-up failed and bill every round it served An error raised by an agentic follow-up after the first call was served now carries a typed mark, so the router's retry loops, the mid-stream retry, and litellm's num_retries wrapper raise it as is instead of rerunning the whole request. A failure of the first call itself still retries, and fallbacks still run. The failure row bills every round the loop served before the failure, at any depth, and a later attempt that succeeds bills what the failed attempts paid on its success row, with the cost breakdown and the response cost headers adding up. The error response's x-litellm-response-cost reports the same amount its failure row bills. Failure rows outside agentic hooks keep the prompt-token estimate they had before. A refused tool call now matches by id among the refused ids, or by name among refused calls that carry no id, so a name-only refusal next to an id refusal is no longer kept. * fix(code_interpreter): delete a session sandbox whose code run failed so the next call starts a fresh one * fix(code_interpreter): delete only the session sandbox that failed, never its replacement * test(code_interpreter): type the session sandbox test helpers * fix(proxy): report what a failed agentic follow-up's failure row bills on /v1/messages errors * fix(agentic): keep the no-retry mark through error mapping and bill only failed follow-ups on success An agentic loop error that is not a provider error, such as the streaming chat cap or a crash in a hook's plan build, now keeps its no-retry mark when litellm maps it to an APIConnectionError, so the router no longer reruns every paid round. The mark is stored on the exception, so a built-in error can carry it too. A crash in a hook's plan build on /v1/messages and /v1/responses bills the first call on its failure row. A recovered attempt adds a failed attempt's usage to its success row only when that attempt was a failed agentic follow-up. Other failed attempts, such as a dropped stream, log their own row as on main. A code interpreter sandbox that fails while running the code now says the code may have run but its result is lost, instead of saying the code did not run. * test(responses): bill a retried MCP request each attempt on its own row again Round 4 stopped adding a non-agentic failed attempt to the success row, so this test goes back to main's assertions: the success row bills only its own rounds and the failed attempt's first round stays on its own failure row * fix(spend): name a failed follow-up attempt the success record already bills With the usage meter on, a failed agentic follow-up attempt gets its own charge-eligible record, and the request's success record also carries that attempt's usage and cost once the success row bills it. The success record now lists the attempt's record ids in included_attempt_ids, so a reader can tell the success record already covers it. Nothing changes when the meter is off * fix(spend): count a billed failed follow-up's tokens on the success row * fix(spend): bill a failed follow-up on a fallback served from cache * fix(spend): show a billed failed follow-up in a cached fallback's cost breakdown * fix(spend): keep a failed follow-up's billing state off the request-merged call details * fix(spend): record the cost breakdown of a successful agentic follow-up The chat agentic loop returned the follow-up response with the cost its own internal call had already stamped, so the request's logging reused that cost and never recorded cost_breakdown on the spend row. The loop now drops that stamped cost unless the follow-up was served from the cache, and the request prices the response itself --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-11-04 02:00:33 +00:00
Mateo Wang	d5ca88900c	fix(anthropic): answer an upstream Anthropic 529 as 529 overloaded_error on /v1/messages (#46063) ... * fix(anthropic): answer a 503 as 529 overloaded_error on /v1/messages Anthropic's error contract has no 503, so a client built against it (Claude Code) treats a 503 from /v1/messages as a plain 5xx and retries it ten times instead of taking its overload path. A Bedrock 503 now leaves /v1/messages and /v1/messages/count_tokens as 529 overloaded_error, for both a JSON answer and a stream that fails before its first byte. An Anthropic 529 overloaded_error now maps to litellm.ServiceUnavailableError (503) instead of InternalServerError (500), so OpenAI-shaped routes answer it as 503, the status OpenAI documents for an overloaded engine, and router retry and allowed-fails policies key it as ServiceUnavailableError. * test(proxy): expect overloaded_error on a 503 stream error after a keepalive ping * fix(proxy): record the 529 the Anthropic routes answer and type the chat stream error frame The messages route stamps the answered status on the exception before post_call_failure_hook runs, so the spend log row, the OTel server span, the failed-request metric, and alerting record the 529 the client saw instead of the deployment's 503, while the deployment failure metric and cooldown keep the deployment's own status. A chat completions stream error frame names the type its status stands for instead of the null .type ServiceUnavailableError carries. An Anthropic-shaped 503 envelope passes through as 529 overloaded_error through a validated TypedDict, and count_tokens answers every ProxyException as the Anthropic envelope. Five integration tests that pinned 503 on the messages route now expect 529 * test(proxy): expect the top-level Anthropic envelope from count_tokens errors count_tokens now answers a ProxyException as a JSONResponse carrying the Anthropic error envelope instead of raising HTTPException, which FastAPI wrapped in {"detail": ...}. The three legacy error-format tests now read the response body and check that no detail wrap is left. * test(proxy): cover the 529 overloaded_error answer across routes, cooldown, and retry budgets * fix(anthropic): keep Anthropic overloads as InternalServerError and answer 529 from the provider status An upstream Anthropic 529 stays litellm.InternalServerError (status 500) so InternalServerErrorRetries, InternalServerErrorAllowedFails, SDK handlers, and the OpenAI-shaped routes behave as on main. The error now carries provider_status_code, and the /v1/messages routes answer 529 overloaded_error from it. * refactor(proxy): unwrap a mid-stream fallback error once instead of recursing for the provider status * fix(anthropic): keep count_tokens' detail body, end keepalive streams with an Anthropic error event, and log any 529 as an overload * test(integration): cover the audit round 3 cells and expect 529 on the count_tokens 503 cases * fix(proxy): re-raise an unchanged stream HTTPException as itself and walk nested mid-stream fallbacks for the provider status * fix(proxy): walk nested mid-stream fallbacks without recursion * fix(anthropic): preserve 503s and answer upstream 529s on messages routes Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(anthropic): answer an upstream Anthropic 529 as 529 on count_tokens Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(anthropic): use integration coverage for keepalive errors Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-11-03 17:37:09 -08:00
devin-ai-integration[bot]	65fef5f172	fix(proxy): test_model skips the response cache, bounds the spend lookup, bills unsaved custom prices and checks spend rows against response_cost	2026-11-04 00:27:34 +00:00
mrinal-berri	7fa438df06	feat(auth): add link-only JWT mapping and remove route eligibility filtering (#49956) ... * feat(auth): allow skipping JWT key route eligibility check * feat(auth): support link-only JWT key mapping * fix(auth): model JWT mapping rejection as a failure value * feat(auth): select existing JWT keys without route filtering	2026-11-03 16:24:51 -08:00
Kerry Lu	ce73cf8b9d	chore(cost-map): remove the 52 rows whose provider retired them before 2026 (#49974) ... * chore(cost-map): remove the 52 rows whose provider retired them before 2026 * test(utils): pin the proxy function-calling parity case to azure/gpt-4.1 * test(batches): price the Azure batch-logging regression case against gpt-4.1 --------- Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-11-03 16:23:06 -08:00
Mateo Wang	de7bb21887	fix(credentials): preserve stored info when patching credentials (#49919) ... * fix(credentials): preserve stored info when patching credentials Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(credentials): type patch test parameters Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-11-03 16:15:09 -08:00
devin-ai-integration[bot]	d1e4b50d6a	fix(router): keep a deployment's token limits off the shared backend cost map key (#49952) ... * fix(router): keep a deployment's token limits off the shared backend cost map key Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(router): preserve catalog token limits on shared keys Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(router): use catalog for shared token limits Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(anthropic): drop the class attribute patch from the deployment max_tokens test Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(router): avoid provider auth during catalog key resolution Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: mateo <277851410+mateo-berri@users.noreply.github.com> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-11-03 16:14:44 -08:00
Devin AI	7447f8f492	Merge remote-tracking branch 'origin/main' into litellm_test_model_endpoint	2026-11-04 00:08:57 +00:00
joshua-berri	b5952f243a	test(mcp): hold the progress test's upstream notification acks until they are dispatched (#49967) ... The MCP SDK server answers a notification POST with 202 before it hands the notification to the session. When the gateway sends DELETE right after `notifications/initialized`, the session can close in that gap, and the SDK logs an ASGI error that fails the test at teardown. Wrap the upstream with the existing `_acknowledge_notifications_after_dispatch` helper the other SDK peers already use, so the 202 only goes out once the session has the notification Co-authored-by: Joshua Valluru <326636767+joshua-berri@users.noreply.github.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-03 15:26:48 -08:00
yucheng-berri	ffbbb6366b	feat(proxy): add general_settings.fips_mode startup gate (#48600) ... * feat(proxy): add LITELLM_FIPS_MODE startup gate With LITELLM_FIPS_MODE=true the proxy refuses to start when TLS verification is off, when Python's OpenSSL does not enforce FIPS, or when the startup password migration fails. A value that is not true/false also refuses. * feat(proxy): refuse FIPS mode when cryptography's OpenSSL is not FIPS The stock cryptography wheel bundles its own non-FIPS OpenSSL, which serves JWT, Fernet and AES-GCM even when Python's OpenSSL enforces FIPS. * fix(proxy): refuse FIPS mode when cryptography's FIPS provider cannot hash The stock cryptography wheel reads the host's FIPS config and reports FIPS on, but its bundled OpenSSL cannot load the host's FIPS provider, so every operation fails. Require a working SHA-256 alongside the FIPS flag. * fix(proxy): refuse TLS verification off anywhere in FIPS mode A model's own ssl_verify: false reached the HTTP client after the startup check passed. get_ssl_verify now raises in FIPS mode whenever verification resolves to off, which also covers the startup check. * refactor(proxy): return FIPS boot refusals as values and harden the FIPS tests * fix(proxy): stop refusing TLS verification off in FIPS mode Skipping certificate verification is a security choice, not a FIPS one: the TLS connection still only uses algorithms the FIPS provider allows. * feat(fips): refuse a database used without FIPS mode, drop the password migration change FIPS mode marks a new database with a fips_mode row in LiteLLM_Config and refuses one that already holds LiteLLM rows without it. Without FIPS mode, the proxy refuses a database carrying the marker. * test(fips): close the database test clients with aclose so the engine stops * feat(fips): read FIPS mode from general_settings.fips_mode instead of LITELLM_FIPS_MODE * refactor(fips): keep fips_mode out of the API-writable general settings, it is read from the config file only * fix(fips): retry the startup marker and state reads once after a dropped connection A dropped connection during the non-FIPS marker read used to count as an unmarked database, so a blip at boot let a non-FIPS proxy start on a FIPS database. Both reads now go through call_with_db_reconnect_retry, the reconnect-and-retry the other startup reads use. A read that still fails keeps its old outcome. * fix(fips): refuse a fips_mode key that has no value fips_mode: os.environ/X with X unset, or a bare fips_mode: / null, resolved to None and started without FIPS mode and without a message, while an empty string refused. A key that is present now has to be a boolean; only a config without the key means off. * refactor(fips): move the database check out of the boot gate The boot gate keeps the setting and the two OpenSSL checks. The check that a database matches the FIPS setting moves to its own PR, so a boot without the flag runs no extra database read. * test(fips): pin how a fips_mode string is read through enforce_fips_mode	2026-11-03 14:56:14 -08:00
joshua-berri	6c6594f1d0	fix(proxy): omit cost headers for unpriced responses (#49467) ... * fix(proxy): omit cost headers for unpriced responses * fix(proxy): preserve declared free response costs * refactor(proxy): share the cost map pricing lookup for the response cost header The header check now reuses the same raw entry, fill-missing and rule lookup that cost model selection uses, so the two can't drift apart. It also counts every declared rate field, so a free model priced only per request keeps its 0.0 header Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Joshua Valluru <326636767+joshua-berri@users.noreply.github.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-03 12:54:08 -08:00
shrey	0aa7079f04	fix(spend): read only the newest rows when paging sessions (LIT-8513) (#48886)	2026-11-03 10:51:09 -08:00
zoveki	ff182c194e	fix(logging): keep the caller's call_type when chat completions are bridged to the Responses API (#49710) ... * fix(logging): keep the caller's call_type when chat completions are bridged to the Responses API * fix(logging): log bridged chat completions with the chat response they returned * fix(logging): log bridged text completions and /v1/messages as served, with the request's options Bridged rows were translated without the request's options, so structured output and replay fields could differ from what the client got. Text completions and non-streamed /v1/messages through the chat bridge also kept a Responses API object or the encoded id. * fix(logging): redact reasoning summaries on bridged rows and log bridged text completions as text completions Redaction now drops reasoning_items on chat messages and deltas, so a bridged row keeps no reasoning summary when message logging is off. A non-streamed bridged text completion logs the text completion the client got, priced like the chat response it came from. The bridge handler's input TypedDict fields are ReadOnly. * fix(logging): redact bridged text completions for opted-out callbacks and pin the sync logged shape and usage ack * test(integration): look up bridged chat spend rows by the id the client got Bridged chat spend rows are keyed by the client-facing id and logged with the caller's call type, so the helpers that decoded the old resp_<base64> key or pinned call_type responses now assert the new contract --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-11-03 10:02:54 +00:00
tessellark	5b546259d5	fix(cache_control)!: keep one deployment's prompt-cache opt-in from reaching siblings on the same model (#49685) ... * fix(cache_control): read the routed deployment's own prompt-cache flag, not a sibling's shared cost-map entry Two deployments that share one model string share one key in the live litellm.model_cost map, and router registration writes each deployment's supports_prompt_cache_breakpoint into that key. The cache control hook read the live map, so a flagless deployment picked up a sibling's opt-in and sent prompt_cache_options and prompt_cache_breakpoint upstream, which the provider rejects with a 400 When a call carries the routed deployment's model_info, the prompt cache checks now use its declared supports_prompt_cache_breakpoint, and fall back to the loaded catalog instead of the live map. This covers the hook on completion, acompletion, Responses and the /v1/messages route, the azure_ai chat transform, the OpenAI Responses transform under drop_params, and the /v1/messages to Responses adapter * fix(prompt caching): read each deployment's own breakpoint flag in the openai and bedrock readers A deployment's own supports_prompt_cache_breakpoint now picks OpenAI's prompt_cache_breakpoint only where the request goes out OpenAI-shaped on a provider other than openai, such as azure_ai OpenAI models and bedrock_mantle. Bedrock on every route and providers that build Anthropic messages keep cache_control, which Bedrock Converse turns into cachePoint The openai chat reader now passes the routed model_info and checks base_model. The Bedrock cache point reader takes the deployment's flag first and, for a routed call with no flag, reads the catalog as loaded instead of the live map that sibling deployments write into. model_info is threaded from the converse transform, the factory message and tool builders, and the responses bridge * fix(prompt caching): keep gateways on cache_control and read register_model opt-ins on routed calls A declared supports_prompt_cache_breakpoint picks OpenAI's breakpoint only on providers hosting OpenAI's API or for a model the cost map lists under the provider, so OpenRouter, LiteLLM proxy, and vLLM deployments keep cache_control as before. Routed calls read the loaded catalog plus the user's own register_model entries, never router registrations. * fix(prompt caching): honor a deployment's own supports_prompt_caching on Bedrock and its declared litellm_provider on gateways * fix(prompt caching): keep wildcard cache flags off matched models, read deployment flags for auto caching and base_model aliases * test(anthropic): cover auto caching on an unmapped Claude deployment * fix(cache_control): keep the merge base's reads where the per-deployment flag changed them A register_model entry under a provider-prefixed name reaches routed calls again, since the overlay now resolves the bare and prefixed names the way runtime registrations are stored. The hook and the openai chat reader no longer judge base_model, as at the merge base, so a client base_model sent under allow_client_pricing_override cannot pick OpenAI's breakpoint and get a 400. A Bedrock deployment flag that is not a real true reads as false again, as at the merge base. An azure_ai wildcard's supports_prompt_cache_breakpoint speaks for each model it matches again, as at the merge base. Other wildcards still leave both cache flags to the catalog. The s10/s11 Bedrock wire test asserted that a flagless deployment inherits its sibling's flag on a shared ARN, which is the leak this PR removes. It now asserts that each deployment keeps its own flag and that a flagless unmapped ARN keeps its cache points after the delete. * fix(cache_control): let the later register_model entry win between a bare and a provider-prefixed name Routed calls read register_model entries through an overlay that merged the bare name after the provider-prefixed one whatever order they were registered in, so a later opt-out under one name lost to an earlier opt-in under the other. Each registration now moves its key to the end of the kept registrations, and the overlay merges both names oldest first, which matches what SDK calls read from the live model map. Also lists the wildcard cache flag helper in the router coverage script, since the hook's wildcard tests drive it rather than calling it by name. * fix(cache_control): keep register_model entries visible while a cost map reload replays them The previous commit moved a re-registered name to the end of the saved registrations by removing and re-adding it, and the overlay walked that dict to order a bare and a provider-prefixed name. A reload replays every saved entry, so a request on another thread could briefly miss an entry or fail on a dict that changed size mid-walk. Entries are written in place again, a per-name sequence records which registration came last, and both the overlay and the reload replay order by it, so the later registration still wins after a reload. * fix(cache_control): read a string or numeric supports_prompt_caching as the bool it spells and find register_model entries under any route prefix A deployment flag stored as "true", 1, "false" or 0 (POST /model/new and quoted YAML keep strings) now means that bool on the Bedrock cachePoint reader and Anthropic auto caching, and any other value falls to the catalog. register_model records which names wrote to each catalog row, so an opt-out under bedrock/ or bedrock/converse/ reaches routed calls like one under anthropic/. --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-11-03 07:55:51 +00:00
Emerson Gomes	ecb8ef5ca8	feat(realtime)!: support latest OpenAI audio models (#35600) ... * feat(realtime): support latest OpenAI audio models * fix(transcription): type SDK stream duration metadata explicitly * fix(realtime): preserve transport settings and translation duration * refactor(realtime): reuse SDK client initialization * fix(audio): bill finalized streams and authorize transcription models * refactor(audio): keep transcription pricing selection together * fix(realtime): reconcile model metadata and test contracts * fix(realtime): preserve provider duration and model capabilities * fix(realtime): keep transcription validation compatible with cost map guard * fix(realtime): address transcription and billing review findings * fix(realtime): preserve input usage and provider routing * fix(realtime): bill retained output and route Azure aliases * fix(realtime): reject malformed provider durations * fix(realtime): preserve batch cache rates and Azure v1 routes * fix(transcription): reject nonfinite provider durations * refactor(realtime): align annotations with current lint rules * fix(realtime): preserve translation deployment pricing and input usage * test(transcription): verify synchronous stream finalization * fix(realtime): retain missing translation input usage at close * fix(transcription): reject streams that bypass output guardrails * fix(realtime): reconcile audio protocols billing and current CI * fix(realtime): trust confirmed formats and authorize default models * fix(realtime): preserve absent models and reject unsupported guardrails * fix(realtime): reject translation policy pipelines * chore(ci): rerun transient infrastructure timing failure * fix(realtime): reconcile audio protocols with current upstream * fix(audio): validate provider responses at SDK boundaries * fix(realtime): preserve minimum OpenAI SDK compatibility * fix(realtime): preserve deferred Live session authorization * fix(realtime): preserve upstream authentication and transcription behavior * fix(audio): scope provider duration precedence to configured models * fix(typing): avoid Final declarations inside transcription cost loop * refactor(audio): reuse transcription mapping and duration resolution * fix(realtime): enforce model budgets and preserve translation routing * fix(audio): log streamed transcripts whose done event reports detected languages * fix(audio): pass stream, languages and keywords through unchecked for non-OpenAI transcription providers Providers without a streaming transcription handler got UnsupportedParamsError for stream=True, languages and keywords, where the merge base forwarded them as provider-specific params. Only openai and azure validate them now. * fix(audio): bill streamed gpt-transcribe from the provider's reported duration The stream path computed cost after logging had replaced the transcription's duration usage, so it fell back to the measured file length while the unstreamed call billed the provider's seconds. * fix(transcription): log streamed transcription usage from the provider response The stream logging path built the spend payload from the response after its duration usage was transformed away, so whisper-1 with stream=true and streamed gpt-transcribe logged an empty usage object. Bill and log from the provider's TranscriptionResponse in both the sync and async paths. * fix(realtime): keep main's routing, protocol and transport behavior around the new translation routes - Send OpenAI realtime and translation client_secrets and calls through litellm's httpx client again, so ssl_verify and custom CA settings apply and upstream errors keep main's shape - Use the input transcription model as the session model only for transcription sessions - Drop the new /openai/v1 translation aliases and record each lazy feature's own routes, so the /openai pass-through and the MCP routes resolve as on main - Let a configured Azure realtime_protocol win for every model, with GA as the default only when none is set - Return x-litellm headers on streamed transcriptions and mark the request done - Refuse translation sessions before connecting upstream when realtime guardrails apply or the provider is not OpenAI or Azure - Accept json and text for gpt-transcribe, and drop other formats under drop_params instead of failing - Bill translation sessions on output audio seconds only, at the single published rate - Document the 201 SDP answer on the translation calls routes and the transcription 200 response * fix(realtime): restore main behavior for guardrails, Azure transcription, and realtime clients Streamed transcription runs unstreamed when a deployment post_call guardrail applies, so the guardrail still sees the transcript. Azure transcription maps v1, latest and preview to the default dated version on the deployments route for every model. The SDK client secret keeps session.model ahead of model, realtime WebSockets connect to api_base again, realtime metadata keeps only guardrails, and managed agents need an explicit transcription model. Reverts the two unrelated streamed Responses error codes and the duplicate cached_tokens_details field, and the gpt-transcribe format check only reads the request provider's cost map entry. * fix(realtime): keep main's transcription, client secret and route behavior where the PR changed it Transcription reaches every provider as on main again, sends stream only when asked, forwards language and languages together, keeps form key order, bills provider seconds only when above zero, honors base_model for the new flags, keeps main's Azure api_version on the proxy path, and streams carry chat's headers, an error event on a mid-stream failure, and a spend row when the stream ends early. Translation cost reads both per-second rates from one source and spend rows carry the billed seconds. Realtime transcription WebSockets keep model in the upstream URL outside OpenAI's hosts, client secrets forward explicit nulls, accept conversation and delay, and answer 400 on a malformed body, the /openai translation paths are back to main, translation refuses only for guardrails realtime sessions run, nested aliases resolve and later session.update models are authorized, translation client secrets run per-model auth, beta clients bridged to GA get beta-shaped session events, and transcription sessions send the deployment model in both shapes. * fix(realtime): address review findings on transcription billing, streaming and realtime auth Per-second pricing replaces token billing only when the priced entry sets no token rates, translation rates fall back per key to the cost map, transcription cache keys include languages and keywords, a stream request a post_call guardrail forces unstreamed comes back as redacted SSE, Azure beta clients are bridged only for GA-only deployments, nested transcription models on the realtime WebSocket are checked against the key, and translation spend carries key and team metadata * fix(azure): pick the GA realtime protocol for deployments whose model_info.base_model is GA-only * fix(realtime): preserve prior merge resolutions after rebase * fix(realtime): enforce translation policies and normalize Azure transcription URLs * fix(cookbook): stop translation once output idles after the upload ends * fix(realtime): bill translation on input audio and drop pre-read stream cost headers Translation spend now uses the input audio seconds the session sent, priced through input_cost_per_second, instead of the translated output duration. A streamed transcription no longer returns cost headers before the stream is read, since the cost is only known after the last event. Also removes an unused realtime guardrail helper. * fix(realtime): drop top-level session nulls, bill dropped transcription streams, and bridge beta response fields Client secret, translation and transcription session bodies drop top-level nulls again, as main did, while nested nulls such as turn_detection still pass through. Transcription client secrets send the deployment model as session.model. A streamed transcription the client drops still runs its success callbacks, and the stream ends with data: [DONE] like OpenAI's. The beta to GA bridge maps max_response_output_tokens, speed and noise reduction, remaps response.create, and drops temperature under drop_params. The Entra token for realtime HTTP routes is fetched off the event loop * test(e2e): cover realtime translation, live transcription, client secrets, and streamed transcription * fix(realtime): close review gaps in translation billing, transcription streams, and Azure GA Bill translation sessions from the deployment's own per-second rates, add per-second prices for Azure gpt-4o transcribe models, keep stream-scoped guardrails seeing the caller's stream flag, map mid-stream transcription errors to LiteLLM exceptions with a [DONE] frame, map GA responses back to the beta shape for bridged beta clients, honor the minting key's guardrail opt-out on translation calls, restore the Azure transcription session URL, and flag GA-only Azure realtime models * fix(realtime): bill a translation rate the deployment omits from the session model * test(e2e): check translated speech is not English instead of exact French wording * fix(realtime): keep main's behavior for non-bridged backends, stream error frames, gateway URLs and Azure transcribe prices Remap beta events to GA only for bridged beta clients and translation sessions, bridge beta clients on GA-pinned Azure deployments, log error-only upstream sessions as failures, flag dated Azure GA realtime models, drop the unpublished Azure per-second transcribe rates, log a mid-stream transcription failure once, keep main's SSE error frame outside transcription, drop unsupported transcription params under drop_params, keep the caller's Azure auth headers, keep a gateway /v1 path, let key opt-outs release string-callback guardrails on translation, and keep the cookbook waiting while a long upload is still sending * fix(realtime): keep the proxy's realtime session gate off translation sessions A key that opts out of a legacy callback guardrail now gets a working translation session, since the transcription-guardrail gate no longer sends a realtime session.update the translation upstream rejects * fix(realtime): keep websocket model limits, bill measured transcription length, route transcription sessions on the allowed model A WebSocket opened without a query model now authorizes as an empty model, transcription cost prefers the measured upload length over the provider's reported seconds, transcription sessions overwrite nested models with the authorized top-level model again, and a non-boolean transcription stream value answers 400 or is dropped under drop_params. * fix(ci): copy the rust bridge extension into the openapi docs gate's base worktree * fix(model-catalog): accept the realtime translation endpoints in the cost map * fix(cookbook): close the translation session and read until session.closed * fix(realtime): treat an empty transcription stream as unset and pass the cost-map guard An empty `stream` form value runs the transcription unstreamed again, as OpenAI does, instead of answering 400. The four translate cost-map entries drop the translation routes from supported_endpoints until the main branch's catalog parser accepts them, and the nested transcription model lookup uses the router registry entry point. * test(proxy): drop the mid-stream error frame pin that main's error shape change replaced * fix(transcription): bill uploaded seconds when a stream fails mid-transcript * fix(proxy): bill uploaded seconds when a streamed transcription breaks mid-stream * fix(audio): keep transcription's positional arguments and redact streamed error details - Move the new languages, keywords and stream options to the end of transcription's parameters, so a prompt or format passed by position keeps its meaning - Redact internal hostnames, paths and tracebacks from a streamed transcription's mid-stream error event, as chat streams already do - Move the CA-bundle check for the realtime HTTP routes, which serves real TLS on loopback, from the unit suite to the SDK integration suite * fix(audio): stream the guarded transcript to SDK callers that asked for a stream * fix(logging): keep a streamed transcript out of callbacks that turned off message logging A streamed transcription's assembled response reached success callbacks with its text intact under turn_off_message_logging or a callback's message_logging=False, because the shared streaming redactor had no transcription branch. It now redacts the text, as the unstreamed call already does * fix(proxy): stream a transcription the deployment streamed when the client sent no stream field --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-11-03 07:27:52 +00:00
devin-ai-integration[bot]	408773bb69	test(spend): assert Anthropic envelope for Messages budget refusals (#49800) ... Co-authored-by: mateo <277851410+mateo-berri@users.noreply.github.com> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-11-03 06:10:08 +00:00
devin-ai-integration[bot]	3da6029977	test(bedrock): assert native structured output when the model rejects forced tools (#49799) ... Co-authored-by: mateo <277851410+mateo-berri@users.noreply.github.com> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-11-03 06:10:00 +00:00
Kerry Lu	fcef9cd0a3	feat(videos): serve custom_provider_map BaseVideoConfig providers on /v1/videos (#49785) ... * feat(videos): serve custom_provider_map BaseVideoConfig providers on /v1/videos A provider registered in custom_provider_map with a BaseVideoConfig handler now resolves on every video call type (create, status, content, list, remix, edit, extension, characters), the same way CustomLLM handlers already resolve on /v1/images/generations. The video entry points and the HTTP handler stop casting the provider name to LlmProviders, which raised before any lookup ran, and a provider without a video config gets a 400 instead of a 500. * fix(videos): type the custom provider lookups so the type gate holds Pick the CustomLLM for a custom provider through ProviderConfigManager.get_custom_llm_handler, which narrows the now wider custom_provider_map handler union, read custom_provider_map instead of the private _custom_providers list, and annotate the untyped custom_llm_provider parameters on the video entry points * test(videos): cover a CustomLLM shadowing a built-in video provider name and type the new tests Mark custom_handler ReadOnly on CustomLLMItem, type the new tests, and assert that a CustomLLM registered under openai still leaves openai video on the built-in config --------- Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-11-02 21:28:46 -08:00
Devin Gupta	216041e243	feat(decisions): add Parallel AI web-grounded decisions (#49188) ... * feat(decisions): add Parallel AI web-grounded decisions * fix(decisions): name the Parallel model decision * fix(parallel_ai): restrict environment credentials to trusted decision endpoints * fix(decisions): keep Parallel evidence fields off other providers and handle missing confidence Only fields a provider names reach the OpenAI response, so existing providers keep their base shape. A malformed api_base answers 400 on both credential paths. The complexity router classifier and the Playground handle a provider that omits confidence. Regenerates the lazy OpenAPI snapshot and the dashboard API types * chore(cost-map): declare the Decisions routes on parallel_ai/decision The provider endpoint files are generated from the cost map, so the row must name both routes for the generated System One flag to hold * fix(decisions): report no classifier confidence when a provider omits it A choice answer with no confidence is a valid classifier answer, so the complexity router picks its tier and leaves classifier_confidence and the confidence signal out instead of reporting a different metric in its place. Also show the Parallel AI logo in the provider picker and add a Parallel translation fixture captured from the live API for both Decisions routes. * fix(decisions): keep a token-priced row on token billing when it also sets a query price A plain input_cost_per_query bills flat only on a row with no token price, and input_cost_per_query_decisions always bills flat, so a custom Decisions row that already bills tokens costs the same as before this PR * fix(decisions): bill a deployment's query price and accept Parallel bases that end in a route A proxy request logs through an object built before the router picks the deployment, so a query price on litellm_params never reached Decisions billing. Parallel Search accepts a base ending in /v1/search and shares PARALLEL_AI_API_BASE with Decisions, so Decisions now accepts that base and one ending in /v1/decisions. * fix(decisions): keep per-query billing and optional confidence on Parallel only A query price on any other Decisions provider moved that provider off token billing, and a choice or score answer with no confidence became a 200 with a null for every provider. Both now apply to Parallel alone, so other providers bill and fail as they do on main. A Parallel deployment with explicit zero token rates and a query price read as free, so every budget check was skipped while it billed per call. It now counts as priced. The dashboard key for Parallel is now PARALLEL_AI so a served bundle that predates the provider still sends parallel_ai as the provider slug, and the README table lists the provider. --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-11-03 04:47:02 +00:00
Mateo Wang	2a94bb3835	fix(spend): bill web search interception follow-ups once under the outer request (#49762) ... * fix(spend): bill web search interception follow-ups once under the outer request Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(spend): keep web search rounds inside interception follow-ups billed Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(spend): keep the outer request as parent for nested interception follow-ups Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * revert(spend): drop parent override for nested interception follow-ups Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-11-02 20:08:45 -08:00
ryan-crabbe-berri	be50cbe9d8	fix(responses)!: authorize and decode comparison response IDs (#49402) ... * test(responses): reproduce comparison response ID ownership and wire failures Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(responses): guard stale retries and malformed comparison envelopes Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(responses): authorize and unwrap comparison response ids Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(responses): require comparison IDs to follow response security policy Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(responses): deny raw comparison response ids by default Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(responses): reject raw comparison response ids by default Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(responses): enforce managed ownership with unmanaged ID opt-in Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-11-03 03:40:46 +00:00
yuneng-jiang	20cb3ea439	test(integration): repair release qualification timing and labels (#49737) ... * test(integration): require deployment IDs on cache-hit failures * test(integration): await completed callbacks and observe locks directly * test(integration): compare long-input bursts with matched benign traffic	2026-11-02 18:02:28 -08:00
tavren27	94b5b6386a	fix(bedrock): send native structured output for json_schema on Claude models that cannot force a tool (#49703) ... * fix(bedrock): send native structured output for json_schema on Claude models that cannot force a tool Opus 5.5 and Sonnet 5.5 on Bedrock cannot force the json tool, so a json_schema request without thinking sent an unforced json tool the model ignored. The opt-in native output key now also applies when the model map says the model cannot force a tool, and the Opus 5.5 commercial Bedrock rows carry the opt-in key. The GovCloud Opus 5.5 rows carry it as false, like the Sonnet 5.5 GovCloud rows * fix(bedrock): keep json tool with thinking on Opus 5.5 and Fable 5.1, and GovCloud on the tool Native output without thinking now applies to every Bedrock Claude model that cannot force the json tool unless supports_bedrock_native_structured_output is false, so Opus 5.5 needs no opt-in row and keeps the json tool when thinking is requested, as #46870 chose. The invoke chat path keeps the native output_format it built instead of inlining it, and GovCloud region-path ids read their own false rows. * fix(bedrock): clean caller output_format schemas before sending them natively on invoke * fix(bedrock): keep Mantle Claude on the json tool and send a caller's output_config.format natively on invoke * fix(bedrock): keep the faked Converse stream for native output that replaces the json tool A json_schema stream that goes native only because the model can't force the json tool now keeps the json tool's faked stream, so it still needs only bedrock:InvokeModel. /v1/messages builds a native-output retry only where it sends native output, and the Mantle test drops its mutable capture. * test(bedrock): cover native output and the opt-out in the invoke json-mode refusal tests --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-11-03 01:56:52 +00:00
mika-lt	183212e8be	feat(anthropic): pick the beta allowlist row per deployment and surface input_transformations (#49689) ... * feat(anthropic): pick beta allowlist row per deployment, add mid-2026 beta rows, surface input_transformations Foundry-hosted anthropic deployments now filter anthropic-beta by the azure_ai row on /v1/messages and chat completions, the bundled config gains the mid-conversation-system-clear-at-2026-08-21 row plus the tool-changes and output-config betas on azure_ai and vertex_ai from a live probe, and Anthropic's input_transformations array reaches chat completions in provider_specific_fields (non-streaming and the first streaming chunk). Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(types): type the provider_specific_fields source dict and beta row test helpers Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): input_transformations reaches chat completions and /v1/messages Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(types): read the Anthropic response as a Mapping when building provider_specific_fields Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(anthropic): fold deployment-row tests into the filtering file and mirror the new beta in the Rust table Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(anthropic): drop the duplicated beta tuple and mark filter results Final Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(azure_ai): forward the seven betas Foundry accepts on the azure_ai beta row A Foundry-hosted anthropic deployment now filters through the azure_ai row, which dropped compact-2026-01-12, fast mode, fine-grained tool streaming, inline tools, oauth, output-128k, and token-efficient tools. Foundry answers 200 to each, and without compact-2026-01-12 a compaction request answers 400, so the row now allows all seven on both the Python map and the Rust table * fix(anthropic): keep pass-through and beta-less requests off the Foundry beta row An openai/ deployment that passes /v1/messages through to a Foundry host lost every beta, and a request without betas read the beta table from GitHub. Only an anthropic deployment on a Foundry host now takes the azure_ai row, and the check reads the provider and host before the table. * test(anthropic): point the beta-gated wire legs whose row lacks clear-at at a native Databricks deployment * style(rust): wrap the azure_ai beta header test case the way rustfmt does * fix(anthropic): forward unlisted betas to Foundry and keep Foundry calls off the Rust messages route An anthropic deployment on a Foundry host now drops only the betas the azure_ai row marks unsupported on /v1/messages, and forwards betas the table does not list as sent, as main did. cognitiveservices.azure.com counts as a Foundry host. The Rust messages route sends an async call with a Foundry api_base to Python, which filters by row. A sync call stays on Rust, since Python has no sync /v1/messages handler. The json_mode chat response keeps input_transformations, and the first /v1/messages call with betas loads the beta table off the event loop. * fix(anthropic): run the advisor loop for an anthropic deployment on a Foundry host The advisor interceptor treated every anthropic deployment as native, so a Foundry-hosted one received the advisor_20260301 tool and Foundry answered 400. Interceptors now get the deployment's beta row from resolve_beta_provider, so that deployment runs litellm's advisor loop the same way an azure_ai deployment does * test(anthropic): drive the Foundry advisor loop test through recorded HTTP The test now intercepts the Foundry and Anthropic HTTP calls with respx instead of patching litellm's own sub-call helper, so it checks what Foundry receives: no advisor_20260301 tool and no advisor beta * fix(anthropic): leave the advisor loop to the providers it ran for on main and tidy the PR's tests Reverts 083990f6d7, since review found it reaches a pre-existing gap that is now with security engineering. The beta row tests drive a real HTTP transport, the beta table fetch test checks the fetch thread instead of a 2 second deadline, and the touched test files keep main's formatting * test(anthropic): give the Foundry beta row logging fixture a fixed datetime start time * test(anthropic): check that chat completions pick the beta row by api_base --------- Co-authored-by: mateo <mateo@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-11-03 01:55:26 +00:00
alberto	e63067a23b	fix(anthropic_endpoints)!: return Anthropic error envelopes on /v1/messages auth failures (#48466) ... * fix: Return Anthropic error envelopes on Messages auth failures Signed-off-by: Alberto Perdomo <aperdomo@redhat.com> * fix: Answer Anthropic envelopes on Messages auth failures without relabelling unmapped statuses Signed-off-by: Alberto Perdomo <aperdomo@redhat.com> * fix(proxy): answer the Anthropic error envelope on the /anthropic pass-through The pass-through relays upstream errors in Anthropic's shape, but the proxy's own refusals there (bad key, model access, budget) still answered the OpenAI body, so an Anthropic SDK pointed at /anthropic could not read them * fix(proxy): answer the Anthropic envelope for 404, 405 and unhandled 500 errors on Messages routes A disabled Claude Code gateway 404, a wrong-method 405 and an unhandled 500 on the Messages and /anthropic/ routes kept FastAPI's detail body or the OpenAI shape. The central HTTP and unhandled-exception handlers now pick the Anthropic envelope by route. The dashboard playground toast titles Anthropic error types when the body has no code. * fix(proxy): keep LiteLLM error types and Anthropic envelopes on Messages routes Error bodies on the Messages routes now carry error.litellm_error_type when the LiteLLM type differs from the Anthropic type, so clients and the Admin UI keep causes like budget_exceeded. Unlisted 4xx statuses map to invalid_request_error per Anthropic's errors page, model-list requests that send anthropic-version get the Anthropic envelope, and the admission, decompression and request-size middleware errors use it on these routes too. * test(proxy): expect litellm_error_type in Messages integration tests and type middleware tests * fix(proxy): keep handler error bodies unchanged and timeouts as api_error Errors the Messages and count_tokens handlers already returned in the Anthropic shape keep their original bodies, so litellm_error_type now appears only on errors the proxy raises before the handler. A 408 stays api_error as before, and the playground toast titles the new Anthropic types and ignores long numbers in front of an embedded error * fix(proxy): keep the base error types inside the Messages handler and scope toast titles to Anthropic bodies The wider status table (402 billing_error, 409 conflict_error, 504 timeout_error, other unlisted 4xx invalid_request_error) now applies only to errors the proxy raises before the handler. Errors inside the Messages handler, streamed error frames and the SDK streaming adapters keep the merge base's table. The dashboard toast reads a status before embedded JSON and titles by Anthropic type only when the body is an Anthropic error envelope, so other pages keep their titles. * test(anthropic): mark the new test variables Final --------- Signed-off-by: Alberto Perdomo <aperdomo@redhat.com> Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-11-03 01:53:29 +00:00
tavren27	640a87f70d	fix(gemini): resend once with the placeholder signature when the other backend rejects a replayed thought signature (#49697) ... * fix(gemini): resend once with the placeholder signature when the other backend rejects a replayed thought signature Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): gemini tool-call signature rejected by the backend is resent once with the placeholder Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(gemini): keep the placeholder resend inside the cache rebuild so either rejection order ends in an accepted body Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(gemini): resend a cached-prefix signature inline and cover native generateContent When the rejected signature sits inside a cachedContents prefix, the placeholder resend now carries the history inline, since replacing the signature in the request body cannot reach the cache. The native generateContent handlers get the same one-time resend. * fix(gemini): send the history inline when Vertex rejects a signature while creating the cache * fix(gemini): keep the context cache on a signature resend and go inline only when that is rejected too --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-11-03 01:24:55 +00:00
devin-ai-integration[bot]	0bd677afef	test(integration): look up streamed Responses spend rows by the client-facing id (#49752) ... Co-authored-by: mateo <277851410+mateo-berri@users.noreply.github.com> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-11-02 17:13:41 -08:00
quietjules	4187e1ae89	fix(managed_files): keep the upload the managed id encodes when two target models share a deployment (#49420) ... * fix(managed_files): keep the upload the managed id encodes when two target models share a deployment * fix(proxy): skip cost tracking for a file delete routed to a deployment * fix(managed_files): delete repeat uploads after the upload answers Run the repeat-copy deletes together in a background task so a slow provider delete no longer holds the upload's response, log the unbilled file-delete cost skip at debug, and type the new test helpers * fix(proxy): run managed file repeat-upload cleanup as a drained internal call Delete repeat uploads after the record is saved, through the shared background task list the shutdown drain waits on, and as an internal call so logging, spend tracking and alerts skip it. Leave litellm_proxy deployments on the old keep-last behavior, since their per-target copies are distinct files. Revert the cost tracker skip change, which the internal call makes unnecessary. * fix(managed_files): delete the copy a managed id names on DELETE and isolate cleanup metadata * fix(managed_files): read the copy DELETE removes from the id's gateway-written fields * fix(managed_files): delete the copy a managed id carries as an internal call * fix(managed_files): pass the parent call id to the cleanup deletes * fix(managed_files): file each repeat-upload delete under the upload that made its copy POST /v1/files sets no request call id, so the cleanup delete passed none and its usage ledger row had no parent. Each copy now keeps the call id of the router upload that made it, and its delete runs under that id. --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-11-03 00:58:41 +00:00
Mateo Wang	4e268c2c55	test(gemini): assert vertex lists on content chunks (#49734) ... Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-11-02 15:40:32 -08:00
paveln83	7fe6b1de34	fix(proxy)!: ignore stored general_settings keys no settings API manages (#48967) ... Apply database general settings only for managed keys, warn about ignored copies, and allow admins to delete stale stored keys Move intended database-only settings into config.yaml before upgrading, including authentication and access restrictions	2026-11-02 14:58:46 -08:00
simonp9	250af8f027	fix(spend)!: save streamed /v1/responses spend rows under the encrypted resp_ id the client received (#49493) ... * fix(responses): log the encrypted response id on streamed /v1/responses spend rows With response id security on, the stream hook encrypts the resp_ id on every frame, but logging had already received the SDK's own copy of the terminal response, so the spend row, the standard logging payload and callbacks got the managed id (or the chatcmpl id on a bridged stream) the client never saw. The hook now records the exact id it advertises on the request's logging object, and the logging layer stamps that id on the logged streamed response * fix(responses): return the id the client sent on GET and cancel, and keep read rows off the create row's id Every GET /v1/responses/{id} re-encrypted the response id, so a client that continued a bridged conversation with the id a GET returned referenced a ciphertext no spend row was filed under. The follow-up lost turn 1 and opened a new session. GET and cancel now answer with the exact id the client sent when it is an id this proxy issued and it decodes to the response the provider returned. A create with previous_response_id still gets a fresh id for its own response, and a raw provider id still gets a managed id Because a read now carries the create's id, a GET or cancel spend row is filed as <id>_retrieve_<call id> or <id>_cancel_<call id>. That keeps one row per read, as before, and a read row can never take the create row's request_id, so the create row survives whichever row is written first and a follow-up replays turn 1 once. The background settlement row keeps <id>_background_cost, and the cost poll's own status checks keep the response id * fix(responses): log a fresh id for each read and cancel, echo the held id only to its owner A GET or cancel of a stored response now logs a new encrypted id of its own, so callbacks and spend rows never reuse the create's id, while the owner of the id still gets back the exact id they sent. An admin reading someone else's response gets an id bound to the admin, as before. The spend row id suffix for reads and cancels is gone, so with response id security off the rows and session history match the default branch. A streamed GET follows the same rules. * fix(responses): echo the sent id to every reader that passes the response access check A GET, streamed GET or cancel of a response now answers with the id the caller sent whenever the caller passed the access check, not only when the caller is the user or team the id was issued to. An admin, the master key or a teammate who read someone else's streamed response got a fresh id back, and the session lookup could not reach the first turn from it, so a bridged follow-up through that id lost the conversation history Every read and cancel still logs a fresh id bound to the reader, never the create's id * fix(responses): keep a read or cancel row in session history only when it brings new output A GET, streamed GET or cancel sent with a session header joins the create's session, and its stored response repeats the create's output. The replay then sent the same assistant turn once per read. Session history now keeps an aget_responses or acancel_responses row only when its stored Responses output has an item id that no earlier kept row carried. A polled background response keeps the first poll that returns the finished output, since its queued create row has none, and later polls of the same output are dropped. Every other row keeps today's rule * fix(responses): replay each history item once, echo the sent id on delete, keep background cost files apart A create row now owns its output items in session history, and a read or cancel row adds only items no create and no earlier read carried, in one linear pass, so a background response polled while it grows, or a read that ends before its streamed create, replays the answer once and after its prompt DELETE /v1/responses/{id} answers with the id the caller sent, on a copy, and logs a fresh encrypted id like a read does Azure Data Lake Storage per-event files and unbatched GCS objects are named after the event's spend row id, so a streamed background create and its cost event, which now share the client's id, keep separate files The integration tests that decoded the managed id now match the exact id the client received * fix(responses): keep main's answer for a raw-id delete and main's storage names outside background cost events A DELETE by a raw provider id returns the provider's id as main did, and logging still records a fresh encrypted id. A streamed GET matches the held id by upstream response id, like a plain GET Only a background response's cost event gets the _background_cost suffix on its Azure Data Lake Storage file and unbatched GCS object, so batch retrieves and events with no id keep main's names An item a session read replays takes the newest read's copy at the first read's place, so an item a poll caught half written replays finished Tests pin the client's id over the router's continued response id, and the storage test no longer reads the clock after the upload * fix(spend): give background cost events their own OpenMeter and DynamoDB ids, replay id-less read items A streamed background create and its later cost event share the client's id, so OpenMeter dropped the second event and DynamoDB overwrote the create's item. Both now key the cost event as <id>_background_cost, like its spend row and storage file. Session history again replays a read's output items that carry no id, as main did. * fix(integrations): keep OpenMeter and DynamoDB logging working without the proxy extras * fix(integrations): give background cost events their own ClickHouse request id and DeepEval span id A streamed background create and its later cost event now share the id the client received, so ClickHouse and DeepEval saw both events under one id. Route both through logged_event_key, the way the Azure storage, GCS, OpenMeter and DynamoDB loggers already do, so the cost event's ClickHouse request_id matches its spend row's `<id>_background_cost` and its DeepEval span gets its own uuid. ClickHouse response_id stays the raw provider id. * test(deepeval): annotate the background cost span test's locals with Final * test(responses): run the DELETE wire test that pins the echoed client id DELETE /v1/responses/{id} now answers with the id the caller sent, so the skip that marked it as a known bug no longer applies. The test passes on the /v1, /openai/v1 and bare routes. --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-11-02 14:13:17 -08:00
yuneng-jiang	0ea7e96a66	test(integration): repair agent reads and bulk deletion setup (#49719)	2026-11-02 13:44:45 -08:00
tessellark	c720b23238	feat(proxy)!: honor model_info.max_input_tokens in the Codex catalog for stock models (#49701) ... * feat(proxy): honor model_info.max_input_tokens in the Codex catalog for stock models A Codex model catalog entry copied from Codex's stock entry (a stock slug, or an alias whose upstream is one) now takes the max_input_tokens every routable deployment of the group sets as its context_window, capped at the stock entry's max_context_window like Codex's own model_context_window. With nothing set, the stock window stays. * feat(proxy): honor the smallest max_input_tokens a group's deployments set in the Codex catalog A deployment without a limit counts as Codex's own window, so a limit on only some deployments can lower the window but never raise it * fix(proxy): skip a zero or negative max_input_tokens when sizing a Codex catalog window * test(proxy): stub the max_input_tokens group field and type the Codex window tests * fix(proxy): skip a max_input_tokens a saved /model/info response copied from the cost map --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-11-02 13:27:58 -08:00
Mateo Wang	f106a09102	test(integration): isolate stranger batch authorization from the batch-cost poller (#49522) ... * test(integration): align stream and A2A contracts and isolate batch authorization Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(sdk): keep EOF assertion compatible with pytest lint Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(sdk): collect cut stream content without mutation Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-11-02 12:57:56 -08:00
yuneng-jiang	044bd824c0	test(integration): align remaining release qualification checks (#49714) ... * test(integration): align remaining release qualification checks * test(integration): scope offline resolution to the restricted runner	2026-11-02 12:43:15 -08:00