122 commits

Author	SHA1	Message	Date
yujonglee	11af2eb22d	refactor(rust_bridge): route every Rust-to-Python call through host modules (#50822) ... * refactor(rust_bridge): route every Rust-to-Python call through host modules Rust now reaches Python only through litellm/rust_bridge/host/ (plus the execution driver), and no Python module outside host/ imports it. Library calls that reproduce Python behavior (httpx and OSError error text, huggingface_hub downloads, the random generator, os.environ) go through host functions too. The rule is written into host/AGENTS.md The live litellm.model_cost table, its version and every Python writer move to litellm_core_utils/model_cost_table.py, and host/model_cost.py keeps only the functions Rust calls. The cost map fetch passes use_http_settings instead of handing Rust an http_settings() snapshot * refactor(rust_bridge): read REDIS_USERNAME from the process environment in Rust The Redis bridge asked Python for all of os.environ to read one key. CPython's os.environ writes through to the process environment, so Rust reads it with ProcessEnvironment like the rest of the module. Also drops doc comments that restated their code. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * ci(rust): raise the native extension size cap to 55 MB Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-11 01:57:54 +00:00
mrinal-berri	9ba551b871	feat(proxy): let teams use their own allowed S3 buckets for batch files (#50571) ... * feat(proxy): let teams send allowed S3 buckets for batch files and read them back A team's metadata can now list allowed_s3_buckets. Keys on that team may send one of those buckets as s3_bucket_name on POST /v1/files and as s3_output_bucket_name on POST /v1/batches, and the upload and batch-create handlers check the bucket against the team list plus general_settings.allowed_client_side_s3_buckets. The early auth body check still rejects bucket fields on every other route and method Reads of managed files now trust the bucket LiteLLM recorded in the stored s3:// uri through an internal stored_s3_bucket_name param, so content, retrieve, delete and the batch cost job can read files written to a request-chosen bucket. Callers can never send that param, and files in the model bucket resolve exactly as before * test(proxy): assert outcomes in team bucket tests and type the managed content read Gives the admit-only tests a real assertion, has the refresh test check what the caller gets back, and narrows the provider file id before reading its bucket so the type gate stays at its base count * fix(proxy): keep team S3 bucket lists proxy-admin only and read prefixed request buckets back A team admin allowed to edit team metadata could add any bucket to the team's allowed_s3_buckets, so the key now sits with the other proxy-controlled team metadata keys. The stored bucket for a managed S3 file now keeps any prefix that comes before LiteLLM's own path, so a file uploaded to a request bucket like team-bucket/tenant reads, refreshes and deletes from that prefix * fix(proxy): keep the request's approved S3 bucket when merging deployment credentials Model-routed uploads and batch creates merge the deployment's credentials into the request, and that merge let a deployment's s3_bucket_name or s3_output_bucket_name replace the bucket the request had already been approved for, so Bedrock wrote to the deployment bucket. The merge now keeps any bucket the request carries and fills it from the deployment only when the request has none * fix(proxy): keep the request bucket only on upload and batch create, not on reads The previous commit let a request's S3 bucket win over the deployment's on every credential merge, which undid #47496 on reads, where the deployment's bucket bounds which buckets a model-routed file id may point at. Deployment credentials win again by default, and only model-routed and provider-only upload and batch create opt in to keep the bucket the request was approved for * fix(proxy): only proxy admins can set a team's allowed_s3_buckets Org admins and users allowed to create teams could put any bucket on a team's allowed_s3_buckets through /team/new or a team update, which grants that team the proxy's AWS access to the bucket. The key now goes through the same proxy-admin-only metadata check as the batch limits, so only a proxy admin can add, change or remove it, and resending the stored value still passes	2026-11-10 11:06:17 -08:00
ryan-crabbe-berri	87be092af7	fix(proxy): send the whole Codex model catalog to Codex 0.166.0-alpha.1 and newer (#50736) ... * fix(proxy): send the whole Codex model catalog to Codex 0.166.0-alpha.1 and newer Codex removed its 1 MiB limit on model_catalog_url responses in openai/codex#56525, first shipped in 0.166.0-alpha.1 (0.165.0 stable was cut without it). Older or unknown Codex versions still get the catalog cut to 1 MiB, since they drop a larger one. * fix(proxy): order patched Codex alphas like 0.166.0-alpha.5.1 when gating the catalog Codex tags patch releases of an alpha as X.Y.Z-alpha.N.M, which PEP 440 rejects, so those clients counted as too old for the uncut catalog, their custom reasoning levels and stock entries. Map them to aN.postM so they sort between alpha.N and alpha.N+1 as semver does. * test(proxy): type the Codex catalog byte-limit test * refactor(proxy): gate the uncut Codex catalog on the 0.166 release line Read only the major and minor numbers of client_version instead of a shared semver comparison, and leave the reasoning level and stock entry version checks as they are on main. * refactor(proxy): read every Codex client_version with one helper Codex sends semver tags such as 0.166.0-alpha.5.1 that the generic version parser rejects. The catalog byte limit, the per-model minimum version and the custom reasoning level checks now share one reader for Codex's format, which also bounds digit counts so an absurd version can't fail the listing.	2026-11-10 07:10:30 +00:00
Mateo Wang	ac08eb5663	fix: make scheduled key, user, and team spend resets idempotent (#50298) ... * fix: make scheduled spend resets idempotent Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test: cover reset roundoff with shared Redis workers Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test: isolate reset workers and type raw adapters Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix: keep updated_at moving on zero-spend budget resets * fix: never leave spend below zero after a budget reset * fix(proxy): keep the stored budget reset time when budget_duration is resent unchanged Key, user, and team updates recomputed budget_reset_at whenever the body carried budget_duration, and the Admin UI key form sends it on every save. Under the guarded reset, a resend during a sweep skipped that period's reset, and a resend after the boundary skipped it on main too. An unchanged duration now keeps the stored reset time * fix(proxy): keep a resent budget reset time only once it is due A save that resends the same budget_duration keeps the stored reset time only when that time has already passed, so the scheduled sweep still resets the period it is due for. A row that is not due yet realigns its reset time on save, the same as before. --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-11-09 22:21:43 -08:00
ryan-crabbe-berri	e099925923	fix(proxy): list images in the Codex catalog only for models known to read them (#50738) ... * fix(proxy): list images in the Codex catalog only for models known to read them A translated Codex entry used to offer image input whenever no deployment declared supports_vision, so Codex let users attach images to models that may not read them. Image input now goes out only when every deployment behind the name says it reads images. * test(proxy): type the vision combination test parameters * refactor(proxy): rename the vision helper and drop the extra vision docstring sentence	2026-11-10 03:46:37 +00:00
ryan-crabbe-berri	16fad64e95	fix(proxy): serve Codex its own Bedrock catalog entries for Bedrock deployments (#50743) ... * fix(proxy): serve Codex its own Bedrock catalog entries for Bedrock deployments Codex builds separate catalogs for Bedrock's runtime and Mantle endpoints that turn off what Bedrock does not support yet. A model whose deployments run on Bedrock now takes its entry from those catalogs, vendored from Codex 0.166.0, instead of from Codex's OpenAI catalog. * fix(proxy): keep Bedrock catalog entries from Codex releases that cannot decode them codex debug models leaves out fields such as supports_parallel_tool_calls that Codex releases before 0.148.0 require, so a Bedrock entry now needs at least 0.148.0 as well as its source entry's minimum.	2026-11-09 19:00:22 -08:00
yujonglee	523b3efea2	refactor(model-cost): move the cost map inputs, fetch and register_model overlay into Rust (#50501) ... * refactor(model-cost): move the cost map inputs, fetch and register_model overlay into Rust * fix(model-cost): keep main's compat helpers pure and move network tests to integration ModelCostMapReloaded.model_cost_map now returns the alias-sharing table, so an alias registration after adopting a reload still reprices the owner. _finalize_model_cost_map reads a caller's map without size limits and installs nothing, through the new NativeModelCostMap.finalize. The unit tests that started a local server or a child interpreter now live in tests/integration/sdk/test_model_cost_fetch_sdk.py, and run_isolated gets direct tests for joining its blocking pool and leaving the fork gate alone. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-10 02:37:35 +00:00
jesus-berri	c76a95fc7e	feat(proxy): let a key's logging entry reference a proxy credential (#50572) ... * feat(proxy): let a key's logging entry reference a proxy credential A virtual key's metadata.logging entry can name a proxy credential with litellm_credential_name. At request time the credential's values, which config credential_list resolves through the secret manager at boot, are merged under the entry's callback_vars, with the entry's own values winning. An entry whose credential is not defined is dropped, so it never falls back to the proxy's own callback environment. Only proxy admins may change the logging entries of a key when the stored or requested list holds a reference (403), and a reference to an undefined credential is a 400. The check runs on key generate, update, team bulk update, regenerate and /user/new, and compares decrypted entries so re-saving metadata read back from /key/info is not a change. Resolves LIT-10061 * fix(proxy): drop a key logging entry whose referenced credential holds no logging values * fix(proxy): only require newly added credential references to exist on key writes * fix(proxy): log a dropped credential reference as an error so /key/health reports it * fix(proxy): gate credential references on bulk-created user keys * fix(proxy): keep failing requests on keys with non-object logging metadata A key whose stored metadata.logging is not a list of objects failed every request with a 500 before this branch. The credential reference rewrite turned those values into no entries, so the request quietly succeeded without the key's logging. Raise a TypeError for those shapes again so the request path behaves as it did on main, while empty values and objects with unknown fields still go through the existing skip. * test(proxy): audit cells for credential references across surfaces, v1, dropped entries and malformed logging The malformed logging cell expects the base behavior that 7ac4ab1356 restored, a 500 for a key whose stored metadata.logging is not a list of objects and a 200 for an empty list * fix(proxy): keep a 500 for teams with non-object logging metadata On main, a team whose stored metadata.logging held non-object items failed chat requests with a 500, because AddTeamCallback(**item) raised a TypeError. This branch switched to AddTeamCallback.model_validate, whose before-validator raises AttributeError on those items, and the proxy maps AttributeError to a 400 invalid_request_error. Raise a TypeError for non-mapping items before validation so the status matches main again, as the key path already does.	2026-11-09 16:02:46 -08:00
zoveki	abe3b11f7a	fix(proxy): show committed spend in the remaining budget gauges (#49199) ... * fix(proxy): report live key and team spend to budget gauges within the key cache TTL * fix(prometheus): carry live spend for the user, org and customer budget gauges too The remaining budget gauges for users, orgs and customers read the same cached rows as the key and team gauges, so they read high for about a minute after a charge. Auth now carries the live counter for each budgeted entity in one snapshot that only the gauges read, floored at the spend auth loaded. The x-litellm-key-spend header and logging spend fields keep their current values. * fix(prometheus): show committed spend in the remaining budget gauges Reads twin counters that take only charged cost, so a worst-case budget hold never shows as spent and a request that finishes last cannot set a gauge back. The twins exist only while the Prometheus logger is registered, and every path that resets a spend counter drops them. Failed requests now set the org gauge from the carried org row and committed spend, not the cached org row. * fix(prometheus): count a charge the cached customer row already holds once * fix(prometheus): keep committed spend only for budgeted entities and drop it on spend resets Auth now reads committed spend through the admission MGET and names the counters a charge may move, so the master key and unbudgeted entities get no twin. Organization spend edits and the global spend reset drop their twins, and a long request no longer reads as charged. * fix(prometheus): read the committed org spend on failed requests without the auth object * test(prometheus): cover a failed request with no carried org * fix(prometheus): move the committed counters for charges with no auth on the row A billed background response is charged from its creator's stored attribution, after the request's auth is gone. Its charge moved the live counters but not the committed ones, so every gauge read one charge behind. It now moves the committed counters that already exist, and its own gauge adds its cost to them. * fix(prometheus): keep the gauges right for pass-through rows, overlapping charges and slow DB spend writes Pass-through rows carry the auth object, so a cold committed counter gets seeded there too. The gauges read a charge still waiting in the post-call pipeline as not counted, whatever other charges reached the counter. Committed counters outlive two DB spend writes. Cold seeds read the DB once per worker, and the global spend reset reads keys and teams in bounded pages. * fix(prometheus): count a charged failure's own cost in the org gauge A stream cut mid-response still charges for what it delivered, and that charge is usually still queued when the failure logs. The failure path passed a cost of 0, so the org gauge left that charge out until the worker's next request * fix(prometheus): skip twins for window-named ids, drop twins on delete, keep auth out of PostHog * fix(prometheus): keep the auth object off pass-through rows and drop user twins on every user delete Pass-through rows now carry only the committed spend in litellm_params, so loggers that dump the row's metadata never get the auth object. Every user delete path, SCIM and bulk delete included, drops the deleted users' committed spend twins * fix(prometheus): keep committed spend values out of logger dumps of pass-through rows CommittedBudgetSpend fields are repr=False, so Athina's payload, Sentry breadcrumbs and debug logs that str() a pass-through row's litellm_params show no spend values * fix(prometheus): keep the gauge fallback on a slow committed read and refresh committed twin TTLs on cluster The success-path committed read now gets half the per-request budget, so a slow Redis falls back to stored spend before the outer timeout cancels the gauges. Post-call charges refresh a committed twin's TTL on Redis Cluster too, so a twin never expires while its charges are still on their way to the DB. Also applies cargo fmt to the PR's Rust files * fix(proxy): key the global reset's committed-spend page queries so the Prisma span check reads them --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-11-09 20:05:37 +00:00
tavren27	2f430873ae	fix(bedrock): derive the web identity session policy per AWS caller (#50592) ... * fix(bedrock): derive the web identity session policy per AWS caller Each AWS caller (Bedrock embeddings, batches, files, SageMaker, Polly, S3 Vectors, and the S3 and SQS loggers) now sends its own STS session policy on AssumeRoleWithWebIdentity, so a role that allows those actions is no longer capped by the Bedrock-only policy. The default Bedrock policy is unchanged, and the credential cache key includes the policy so callers never share a narrower session. * fix(bedrock): let async invoke write its S3 output under the web identity session policy Bedrock writes an async-invoke job's output with the caller's session, so the embedding session policy now allows s3:PutObject only when Bedrock makes the call (aws:CalledViaLast bedrock.amazonaws.com). * fix(opensearch): give OpenSearch vector store search its own web identity session policy * fix(bedrock): give every remaining AWS caller its own web identity session policy Secrets Manager, knowledge base ingestion, model listing, the S3 config file reader, the SageMaker chat handler, and the Comprehend Medical and Transcribe pass-throughs each send a session policy with only the actions they call Bedrock embeddings now use the Bedrock policy, so chat and embeddings share one STS session. That policy adds async-invoke status, S3 reads and writes only through Bedrock, KMS only through S3, and Mantle token counting bedrock:InvokeModel gets its own statement without aws:SecureTransport, since Bedrock checks a Nova Sonic realtime stream as InvokeModel with that key false KMS grants are limited by kms:ViaService, file downloads get a read-only S3 policy, and batches and speech config export skip underscore attributes * test(batches): expect config export to skip every underscore attribute The session policy class attribute is private, so batches config export no longer returns single-underscore names * fix(bedrock): scope knowledge base ingestion role grants to the roles it creates Knowledge base ingestion only creates and passes litellm-bedrock-kb-* roles, so its session policy no longer lets the session create, edit, or pass any other role, the proxy's own included * fix(bedrock): give embeddings and the agent-runtime pass-through their own web identity session policies * fix(bedrock): grant enforced guardrails and the remaining agent-runtime operations over web identity Every web identity ceiling that grants bedrock:InvokeModel now also grants bedrock:ApplyGuardrail, so an account or organization enforced guardrail no longer denies embeddings, the agent-runtime pass-through, or realtime The agent-runtime ceiling grants every POST operation the pass-through forwards, adding flow executions, document content, and ingested document ACLs, and a test derives that list from botocore Only the realtime caller gets bedrock:InvokeModel without the aws:SecureTransport condition, since chat and embeddings never open a bidirectional stream --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-11-08 19:13:53 +00:00
Mateo Wang	7ea4028625	fix(proxy): align ownership and scope handling (#50600)	2026-11-08 07:03:36 +00:00
Kerry Lu	51b49973c9	fix(proxy): hold DB_RECONCILE_LOCK around credential create, update and delete (#50551) ... * fix(proxy): hold MODEL_RECONCILE_LOCK around credential create, update and delete The config reload snapshots the credential table, awaits config rows, then prunes and upserts litellm.credential_list from that snapshot. A credential write landing inside that window was undone in memory until the next reload. The three endpoints now take the same lock the reload holds. Fixes #50273 Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(proxy): take MODEL_RECONCILE_LOCK before the credential update reads its row Two PATCHes queued behind a reload read the same old row and the second overwrote the first. The relay now keeps enough bytes to spot a split after marker, and the new unit tests are fully typed. * test(credentials): annotate the reconcile lock test locals with Final * refactor(proxy): rename MODEL_RECONCILE_LOCK to DB_RECONCILE_LOCK The lock gates every db-to-memory reconcile and now the credential writes too, so the model-only name undersold what it protects. --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-11-07 16:02:44 -08:00
yucheng-berri	b603e84c02	fix(mcp): derive MCP bridge and session keys with the FIPS provider's KBKDF (#50032) ... * fix(mcp): derive MCP bridge and session keys with the FIPS provider's KBKDF With OpenSSL enforcing FIPS, hashlib.scrypt raises, so every MCP session token and bridge envelope request failed with a 500. Under FIPS the keys now come from LITELLM_SALT_KEY through SP 800-108r1 KBKDF run by the provider's EVP_KDF. Without FIPS the scrypt keys are unchanged * fix(mcp): keep master key rotation invalidating MCP tokens under FIPS Pass the master key as the KBKDF context, so rotating it changes the three FIPS-mode keys as it changes the scrypt keys * test(mcp): annotate new FIPS key test locals with Final * fix(mcp): derive the FIPS MCP keys only when general_settings.fips_mode is on On a FIPS OpenSSL 1.1.1 host (RHEL 8) scrypt still works, so keying the switch on openssl_enforces_fips() sent those hosts to the OpenSSL 3 only KBKDF and broke MCP sessions and bridge envelopes that worked on main. fips_mode_active() is the same reader B1 adds, set by enforce_fips_mode at startup. * refactor(fips): move the salt key KBKDF into encrypt_decrypt_utils The KBKDF helper derives keys from LITELLM_SALT_KEY, so it now sits next to the other salt key helpers, and fips.py goes back to main's version. * Pin the MCP key KBKDF fetch to the FIPS provider Fetch KBKDF with the property query fips=yes, so a change to the process's default properties after boot can't move the derivation to a non-FIPS provider. The shared fips_openssl test fixture drops the pin, since the test OpenSSL has no FIPS provider.	2026-11-07 13:33:27 -08:00
ryan-crabbe-berri	ab3b12b951	feat(proxy): serve Codex its own release's model catalog, fetched from GitHub (#50318) ... * feat(proxy): give Codex accurate model entries for every model in /v1/models * fix(router): read a deployment's upstream model without a not-required key access * fix(proxy): keep Codex tiers for unpriced deployments and match stock entries by declared provider A deployment with no token price no longer votes a Codex service tier off, an undescribed model keeps Codex's default reasoning summary parameter, a Bedrock deployment naming its provider in custom_llm_provider gets Codex's stock entry, and Bedrock provider namespaces follow the AWS model id format. * fix(proxy): import cost tracking and provider helpers lazily in the Codex catalog lite codex imports the catalog before the SDK is initialized, so importing cost tracking, the Bedrock helpers or provider resolution at module level hit a circular import and broke every lite command. * fix(proxy): drop Codex's fast speed tier wherever its priority tier is dropped Codex shows /fast for a leftover fast speed tier even when service_tiers no longer offers priority, so an alias, an empty configured tier list or a configured list without priority showed a /fast toggle that sends nothing. One helper now sets service tiers, speed tiers and the default together. * fix(proxy): give an older Codex a translated entry for models past its version Codex's own entries name the oldest release that can read them in minimal_client_version. A release older than that can fail to decode the entry, such as a reasoning effort it has no name for, which fails the whole catalog, so it now gets the translated entry, whose levels are trimmed to that release * feat(proxy): serve Codex its own release's model catalog, fetched from GitHub When Codex asks /v1/models for its catalog, the proxy now reads models.json and prompt.md at that release's tag on GitHub and serves those entries, falling back to the bundled 0.166.0 copy when the fetch fails, times out after 2 seconds, does not validate, the version has no tag, or LITELLM_OFFLINE_MODE is on. A failed fetch is remembered per version for an hour and only the eight most recent versions are kept. The integration tests ask as the bundled release so their expectations hold whether or not the proxy can reach GitHub * fix(proxy): bound the GitHub fetches a caller can trigger through Codex catalog versions Failed versions now live in their own cache so they never push out a fetched catalog, requests for a version already being fetched share that fetch, each worker starts at most six fetches a minute and serves the bundled catalog past that, and the version check only accepts ASCII digits * fix(proxy): keep a failed Codex version's cooldown past the fetched catalog cap Failures are tiny, so they get their own cap of 256 instead of sharing the eight-catalog cap, which let nine failing versions retry each other well before their hour was up. The TypeVar takes Final like the rest of the codebase	2026-11-06 21:58:54 +00:00
Mateo Wang	3f38730d2c	refactor(typing): clear about 1,100 basedpyright Any errors across 303 files (#49986) ... * refactor(typing): clear basedpyright Any errors (wave 1) * refactor(typing): clear basedpyright Any errors (wave 2) * refactor(typing): clear basedpyright Any errors (wave 3) * refactor(typing): clear basedpyright Any errors (wave 4) * refactor(typing): clear basedpyright Any errors (wave 5) * refactor(typing): clear basedpyright Any errors (wave 6) * fix(typing): keep the cached-plan fallback's direct query_first calls so span naming resolves * fix(logging): keep callbacks running when litellm_params is a params model should_run_callback and the sync success and failure handlers read litellm_params through any object with a get method again, as they did before this branch, so a logging hook that returns a GenericLiteLLMParams no longer stops sync logging for that request Adds behavior tests for the changed lines codecov listed as not run * fix: read the regional uplift for every data residency again and reuse one dict TypeAdapter The typed lookup only read the eu and us multipliers, so a deployment that set regional_processing_uplift_multiplier_fedramp stopped applying it. /model/update and the model write paths now share one module-level TypeAdapter instead of building one per call. * fix: bill regional uplift multipliers float() accepts, bytes included * fix: raise the in operator's own error from one shared json_contains, and name the bad uplift type	2026-11-06 13:31:15 -08:00
joshua-berri	de0b389205	feat(router): add the Switchyard stage router as an auto router strategy (#50407) ... * feat(router): add Switchyard Stage Router strategy * feat(router): decision-model classifier and safer fallbacks for the Switchyard stage router Moves to nemo-switchyard 0.4.0a1, which adds decision-model classifiers and makes ModelCall.respond and fail async. The classifier is now a tagged union of an LLM judge and a decision model, and decision calls go through the router's adecisions so any decisions provider works. Algorithm misbehavior and library errors route to the capable model instead of failing the request, and the session id plus client session headers are always passed so capable holds apply without a classifier Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(router): keep the Switchyard router inside the type-check gate The gate's environment did not install the optional switchyard extra, so every libsy symbol counted as Unknown. nemo-switchyard joins proxy-dev like the other optional runtime packages, and imports go through switchyard_rust.libsy, which ships py.typed. Step handling moves into helpers so no Final is rebound in the loop, and the dashboard API types are regenerated for the new routing fields Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(router): route Switchyard on every endpoint and reject unsafe router saves The router now passes each request's call type to routing strategies, and Switchyard converts the request to chat messages with the existing converter for that call type before translating it. /v1/messages turns with tool or thinking blocks used to fall back to the capable model, and now route on their signals. The Anthropic converter can keep tool_result.is_error on an analysis copy, so Switchyard sees tool failures there, while requests sent to models never carry it Saves of a strategy router that lists itself now fail with a 400 instead of a 500 with a dead row, the capable and efficient models must differ, judge and decision calls are not retried before failing open, the Switchyard routing fields are classified for redaction, and the edit dialog returns the saved router instead of the stale one Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(router): catch self-routing on patch and rename, and trim Switchyard tests The self-routing check now reads the stored params through the decrypting helper and runs when a patch carries no litellm_params, so a PATCH that points a router at itself or renames it onto one of its targets returns 400 instead of a 500 with a dead row. The Switchyard tests share one transcript builder and router fixture, and reformatting noise in unrelated test files is reverted Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * refactor(ui): make Switchyard a strategy in the existing auto router form Switchyard is now a card in the existing strategy picker. Picking it swaps the tier and classifier fields for the Switchyard fields, and the shared name, team and save button create it through one submit branch, the same flow as the other strategies. The separate add form, the extracted picker and the hidden-form swap are gone, and the edit dialog keeps its own Switchyard form The judge's session_affinity and message_hash_fallback settings are removed, since Switchyard rejects both for the stage router Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(ui): accept partial tool semantics and show server errors on model edit The Switchyard tool semantics field rejected mappings that list only some categories, although the server accepts them. Model PATCH errors were reduced to a generic message, so it now goes through the shared API client like model create and surfaces the server text Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * feat(ui): show the routing strategy picker up front in the auto router form Every strategy was hidden behind an Advanced configuration link, so the quick setup read as the only option and never said it creates a heuristic router. The picker now always shows with Heuristics as the labelled default, the quick setup sits under it, and picking another strategy opens that strategy prefilled with the chosen models Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(ui): show the server's reason when a model settings or API key update fails Both screens replaced the rejection with a fixed message, so a 400 such as a router routing to itself read only as a generic failure Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * feat(router): route Switchyard on the Rust router core main replaced the Python router core with the Rust one, which left Switchyard routers registered nowhere. Register them with the other pre-routing strategies, hand each strategy the endpoint the request came through, and evict Switchyard routers on reload and delete like the rest Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(router): scope Switchyard sessions per API key and fix review gaps in the Switchyard form Switchyard session state is keyed by a hash of the caller's key hash and session id, and client session headers no longer reach Switchyard unscoped. The add form carries the quick setup models into Switchyard, sends the team and access groups, clears a stale tool semantics error, and the schema reads API-saved configs with nulls and structured decision instructions Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test(router): make the Switchyard test folder a package * fix(router): read the Switchyard session key hash only from the proxy's own metadata * fix(proxy): keep nested nulls as null when masking deployment params * fix(router): send Switchyard judge calls with the caller's own metadata bucket so tag routing applies * fix(router): put Switchyard judge metadata in the bucket the judge endpoint reads --------- Co-authored-by: Joshua Valluru <326636767+joshua-berri@users.noreply.github.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-06 12:07:57 -08:00
simonp9	1f15088fc9	feat(proxy): mixed Codex model picker and LiteLLM routing on native codex routes (#49408) ... * feat(proxy): mixed Codex model picker and LiteLLM routing on native codex routes The native Codex catalog now lists the key's other chat and responses models after ChatGPT's, within Codex's catalog size limit, and drops the ETag so the merged list refreshes on Codex's timer. A request for a model configured by exact name on another provider goes through /v1/responses without the ChatGPT bearer or account header, while ChatGPT names and wildcard-only matches keep the subscription passthrough. Reasoning items another provider produced are dropped before forwarding to ChatGPT. * fix(proxy): keep Codex's picker from turning reasoning off for LiteLLM models Codex fills a missing default_reasoning_level with effort "none" when a model is picked in /model, while `codex -m <model>` sends no effort at all. A thinking-only provider model then answers 400. On the native /backend-api/codex/responses route, drop effort "none" for a LiteLLM model whose catalog entry names no default, so the provider applies its own default exactly as `codex -m` does * fix(proxy): harden Codex mixed-model routing on the native ChatGPT routes Keep the ChatGPT credentials out of hook raw headers on the inner /v1/responses request, route to LiteLLM and merge LiteLLM catalog rows only for keys that may call /v1/responses, drop unpersisted reasoning ids on the passthrough when store is not true, keep any slug the caller's ChatGPT account lists on the subscription via a shared account catalog cache, and keep X-Models-Etag for keys whose catalog added no LiteLLM rows. The guide covers the new behavior and the /v1 wildcard precedence over a catch-all deployment. * fix(proxy): keep Codex's native route on the subscription when the account list is unknown A name OpenAI lists for the caller's ChatGPT account now goes to LiteLLM only when the account's list is known and leaves it out. A failed catalog read is remembered for 60 s and the Codex version comes only from a User-Agent whose product matches the originator header. The merged catalog serves a stable ETag digest of OpenAI's ETag and the key's LiteLLM rows, and /responses replaces X-Models-Etag with the same value, memoized per key for 300 s, so Codex stops refetching or dropping its catalog cache. LiteLLM-routed native replies wrap their reasoning with the model id, so an OpenAI or Azure blob is dropped before chatgpt.com and unwrapped for LiteLLM, and the chat-completions bridge decodes wrapped thinking blocks with their signature. The access-group recipe uses chatgpt/ model names so /v1 routing does not change, and the native route checks the chatgpt/ form of a name the key holds only through that group. A none reasoning effort is dropped only under drop_params. The guide covers app tools, every routing case, and the ETag, and the catalog logs how many models the byte limit left out. * fix(fireworks_ai): move Codex additional_tools items into tools on the Responses API * fix(proxy): keep Codex native turns working when the key's model listing fails A listing callback that raises no longer turns chatgpt.com's catalog or responses answer into a 500. The catalog keeps upstream's body and ETag, and responses keep upstream's X-Models-Etag. Under drop_params, a `none` reasoning effort is now dropped only when a candidate deployment does not explicitly support it, so an alias to a model like openai/gpt-5.4 keeps it. The account slug list and its unavailable marker are keyed on the account id plus the caller's ChatGPT Authorization header, so a failed fetch with one bearer cannot turn off LiteLLM routing for another caller on the same account. The auth step checks the Codex route before importing the Codex module. * fix(proxy): keep Codex model switches working across LiteLLM providers On the Codex native route, a LiteLLM model now gets only the encrypted reasoning every deployment it can route to is able to decrypt, so ChatGPT or another deployment's blobs no longer make xAI answer 400 Codex additional_tools items are hoisted into tools for every provider except the OpenAI API, and namespace tools become flat tools named mcp__docs__search, the shape xAI accepts A JWT caller's models ETag is cached on a hash of its LiteLLM credential, so ChatGPT turns stop rebuilding the catalog rows * fix(proxy): keep Codex namespace tool calls intact across LiteLLM providers Fireworks and Bedrock Mantle accept Codex namespace tools natively, so the native route now hoists them without flattening, like OpenAI and Azure and the chat completions bridge. Other native Responses providers still get flat names, now built with the bridge's namespace_tool_name_map, and each function_call they return gets its tool name and namespace back in the non-streaming body and in streamed output_item.added, output_item.done, and completed/incomplete/failed events. Every other event passes through unchanged * fix(proxy): bridge Codex turns for providers whose Responses API rejects Codex tools Codex sends its freeform exec tool as a custom tool on every turn, and xAI's Responses API answers 422 for custom, namespace and additional_tools shapes. On the Codex native route, a model whose deployments include a native Responses provider other than OpenAI, Azure, Fireworks or Bedrock Mantle now asks for the chat completions bridge with use_chat_completions_api, which already turns custom and namespace tools into functions, hoists additional_tools, and puts the namespace back on each returned call OpenAI and Azure still get the request unchanged, and Fireworks and Bedrock Mantle still get it on their own Responses API with additional_tools hoisted. The route's namespace flattening and the response-side namespace restore are removed, since no path sends flattened names anymore * fix(proxy): drop reasoning ids a Codex turn's LiteLLM target cannot resolve A reasoning item another deployment produced, replayed with store false and no blob the target can decrypt, now reaches the target as its readable summary without id or status, and is dropped when nothing readable is left. OpenAI answered 400 on the status field and 404 on the id for a glm-5p3 item replayed whole to an openai deployment. Same-deployment replays and stored conversations keep the item whole * refactor(codex): reach encrypted reasoning stripping through the router registry The Codex route read Router.optional_callbacks and imported EncryptedContentAffinityCheck from litellm.router_utils, which the router surface fence forbids outside the router. RouterRegistry gains decryptable_reasoning(model_group, request_input, targets), which keeps the same rule: with encrypted content affinity on for the group, reasoning one target can decrypt stays, otherwise every target must decrypt it * fix(proxy): keep Codex's priority tier off LiteLLM rows in the native model catalog * fix(proxy): drop Codex's none effort for an alias whose native entry names no default * fix(proxy): start a native Codex alias at its pinned reasoning effort and build the catalog off the event loop * test(proxy): map the pinned reasoning efforts field in the router registry stub * fix(proxy): honor admin-only /v1/responses and keep Codex's default tier off native aliases Codex native routes now send a key to LiteLLM only when admin_only_routes allows it, a native alias with configured service tiers starts with no tier picked, and the account slug cache declares its Redis key family. * fix(proxy): check the key's tool allowlist on native Codex LiteLLM turns and additional_tools items * fix(proxy): check vector store grants on file_search tools inside additional_tools items A Responses request can define file_search inside an additional_tools input item, which the native Codex route hoists for LiteLLM-served models and OpenAI runs as is on /v1/responses. The vector store check now reads those tools in both strict and legacy modes. The input item scan uses a Mapping guard instead of a pydantic error per item, since auth now runs it on every list input. * fix(proxy): apply the proxy-wide allowed_routes to LiteLLM models on native Codex routes * fix(proxy): cap native Codex account catalog reads per LiteLLM caller before rate limits apply * fix(guardrails): check additional_tools items next to top-level function tools in tool_policy ToolPolicyGuardrail read tools from the request body only when no top-level function tool existed, so a blocked tool defined in an additional_tools item reached Fireworks once the Responses transform moved the item into tools after the pre-call hooks. Check every list on each request, including namespace members inside items. * fix(router): serve decryptable_reasoning from the Rust core's optional checks The merge with main removed the Python router core, so the registry's host has no optional_callbacks and no routing context, and native Codex requests with encrypted reasoning raised RustCoreUnsupported. The filter now lives with the optional checks, which own both. The ported pinned_reasoning_efforts field gets a test on the Rust core Also give the static litellm.Router a declared constructor. Since the Python core was deleted a bare __init__ assignment reads as RustCore's, so every Router(model_list=...) is a type error and make lint fails on main. TYPE_CHECKING only, no runtime change * fix(proxy): check the budget of a saved alias target on native Codex LiteLLM turns Native /backend-api/codex/responses binds {endpoint}, so auth skipped the key's or team's model_group_alias and read budgets on the requested name, while the later /v1/responses handoff served the target unchecked. Auth now resolves the alias on a turn that routes to LiteLLM, as it does on /v1, and leaves ChatGPT-only names on the subscription. Also makes the tool_policy tests build a real ToolPolicyRegistry and type their fixtures. * fix(proxy): keep a name the ChatGPT account lists on the subscription despite a saved alias * fix(codex): keep LiteLLM keys off ChatGPT, enforce tool policy on namespaces and pin reasoning on aliases Auth no longer asks ChatGPT for the account's model list when the bearer is missing or is the LiteLLM key, so a key sent as the bearer never reaches it. tool_policy now reads the members of a top-level namespace, which is where the native route's hoist puts them. A global alias now reads encrypted content affinity from its target group. A non-list top-level tools value and a non-string function name now reach the provider, which refuses them, instead of raising a 500. The catalog ETag no longer depends on the order a process iterates priced service tiers in, so workers and pods agree on it. * fix(proxy): list a team's models in the order the team set them get_team_models built the names in a set, so each process listed a team key's models in its own hash order and the native Codex catalog sent a different row order and ETag from each worker * fix(proxy): apply JWT route policy before native Codex routes serve LiteLLM models A standard JWT caller now gets a LiteLLM model on /backend-api/codex/models and /backend-api/codex/responses only when the JWT auth flow would admit a direct /v1/responses call for that model, so team_allowed_routes, admin_allowed_routes and RBAC route limits hold on the handoff too. The Fireworks Responses transform also leaves additional_tools input items in place when the tools param is not a list, instead of raising while hoisting * fix(fireworks): hoist additional_tools when tools is a tuple litellm.responses passes a tuple tools argument through to the Fireworks transform unchanged, so the non-list guard skipped hoisting and left the additional_tools item in the input. Treat tuples like lists and cover both shapes in the drop_params merge test --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-11-06 11:14:47 -08:00
yucheng-berri	4086d5ccaa	feat(proxy): hash UI passwords with PBKDF2 in FIPS mode (#49874) ... * feat(proxy): hash UI passwords with PBKDF2 in FIPS mode A FIPS OpenSSL refuses scrypt, so every password write fails there and nobody can sign in with a password. When OpenSSL enforces FIPS, passwords now use PBKDF2-HMAC-SHA256 at 600k iterations with a salt from OpenSSL's DRBG, stored as "pbkdf2:" plus base64 of salt and key. Any other row is refused with one error log and left as it is, and the boot migration does nothing, because FIPS mode is for new deployments. PBKDF2 at 600k takes about five times as long as scrypt, so in FIPS mode the five request paths run it on one thread instead of the event loop. A single thread keeps a login burst from using up a CPU quota, which is what stalls the loop with asyncio.to_thread. A login that queued for that thread asks the throttle again before it answers, so a burst of guesses can't get past a block that the failures ahead of it just set. The breached password check is off in FIPS mode, because it runs the password through SHA-1. A standard OpenSSL reads False at every new branch, so standard mode keeps scrypt, inline, on the same rows. Resolves LIT-8425 * fix(proxy): take FIPS logins one at a time and recheck the throttle first A login that waited for the KDF thread asked the throttle again only after deriving, so a burst for one user kept that thread busy long after the user was blocked. Logins now wait for their turn on a lock and ask the throttle when it comes, so a blocked login costs no PBKDF2 and only one login at a time holds a place in the KDF queue. Password changes and invite claims no longer wait behind a whole burst of logins. * fix(proxy): keep PBKDF2 rows on a standard boot and warn when FIPS skips the breach check A standard boot treated a pbkdf2: row as plaintext and hashed its text, so the real password stopped working and the old row text signed in. The migration now leaves a row of that exact shape alone. Any other pbkdf2: text still migrates. FIPS mode now logs a warning when it skips a breach check that is turned on. The one-thread test no longer depends on timing, and the new test variables are annotated Final. * fix(proxy): keep the breached password check on in FIPS mode * fix(proxy): choose the password KDF from fips_mode instead of the host OpenSSL * test(proxy): fake the scrypt probe in the FIPS fixture now that the gate checks scrypt	2026-11-06 01:31:59 -08:00
yujonglee	b1b236826a	feat(redis): serve the proxy's common Redis call with Rust stores (#50170) ... * feat(redis): route the router, rate limiters and auth through the native store families * style(redis): satisfy the python lint gates Format the rate limit store, sort its __all__ and give the routing read batch and two tests concrete list and dict types. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(router): give the registry's default usage store a memory-only handle RouterUsageStore now takes a handle, so the default passed None and any usage read before RustCore swapped in its store raised AttributeError. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test(redis): build stores from native handles and wait for router tasks Tests still used the pre-refactor store constructors (Store(None), handle= factories, NativeIncrement, tuple scopes). The router outcome tests counted 20 loop ticks, too few now that the allowed-fails increment resolves on the native runtime, so settle() waits for the scheduled tasks instead. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-06 06:01:11 +00:00
ryan-crabbe-berri	9a473fbbb0	feat(proxy): give Codex accurate model entries for every model in /v1/models (#49825) ... * feat(proxy): give Codex accurate model entries for every model in /v1/models * fix(router): read a deployment's upstream model without a not-required key access * fix(proxy): keep Codex tiers for unpriced deployments and match stock entries by declared provider A deployment with no token price no longer votes a Codex service tier off, an undescribed model keeps Codex's default reasoning summary parameter, a Bedrock deployment naming its provider in custom_llm_provider gets Codex's stock entry, and Bedrock provider namespaces follow the AWS model id format. * fix(proxy): import cost tracking and provider helpers lazily in the Codex catalog lite codex imports the catalog before the SDK is initialized, so importing cost tracking, the Bedrock helpers or provider resolution at module level hit a circular import and broke every lite command. * fix(proxy): drop Codex's fast speed tier wherever its priority tier is dropped Codex shows /fast for a leftover fast speed tier even when service_tiers no longer offers priority, so an alias, an empty configured tier list or a configured list without priority showed a /fast toggle that sends nothing. One helper now sets service tiers, speed tiers and the default together. * fix(proxy): give an older Codex a translated entry for models past its version Codex's own entries name the oldest release that can read them in minimal_client_version. A release older than that can fail to decode the entry, such as a reasoning effort it has no name for, which fails the whole catalog, so it now gets the translated entry, whose levels are trimmed to that release	2026-11-05 19:31:32 -08:00
ryan-crabbe-berri	48ff826cbc	feat(proxy)!: only let clients pick S3 buckets the admin allowed (#47496) ... * feat(proxy): let admins allow specific S3 buckets and endpoints in request bodies, block bucket settings otherwise Client requests can no longer pick s3_bucket_name, s3_output_bucket_name or allow_legacy_cloud_file_ids unless the value is on general_settings.allowed_client_side_s3_buckets (buckets) or the admin opted in the usual way. s3_endpoint_url, already blocked, can now be allowed per value through general_settings.allowed_client_side_s3_endpoint_urls * refactor(proxy): drive the client-side S3 allowlist from one typed field-to-setting map * fix(proxy): treat a null client-side S3 allowlist in config as empty instead of failing the request	2026-11-06 02:50:15 +00:00
yucheng-berri	9f489430a6	fix(proxy): probe scrypt instead of MD5 in the fips_mode OpenSSL check (#50305) ... * fix(proxy): probe scrypt instead of MD5 in the fips_mode OpenSSL check * test(proxy): fake a working scrypt so the fips_mode check tests pass on FIPS hosts	2026-11-05 17:43:32 -08:00
joshua-berri	a12a6cd154	feat(accounting)!: make the SpendLogs ledger the default accounting path (#49872) ... * fix(accounting): keep one undecodable delivery from blocking the ledger A request whose metadata carried a 401-digit tokens_saved stored that number in its delivery payload. The query engine could not decode it when the worker claimed the batch, so every later delivery on the proxy stayed pending The worker now claims the payload as text and decodes each row in Python. A row that still cannot be decoded is parked alone while the rest of the batch applies. The payload builder also clamps integers to the JSON-safe range, so caller-supplied counts can no longer reach the engine at that size Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(accounting): take spend locks in one order so the ledger cannot deadlock A ledger delivery updates many spend tables in one transaction, so it can deadlock with any writer that locks the same rows in another order. It took the team advisory lock after the user and team rows, while team roster writers take it first The delivery now takes every team advisory lock before any row, then writes tables in one canonical order with ids sorted. It also runs under the rollup lock_timeout, applies 500 deliveries per batch by default, and spreads retries with jitter The writers that could still collide with it now follow the same order. The budget reset cascade, model allowlist rename, global spend reset, bulk user update, org delete and the delete-by-list paths lock their rows in key order first, and access group endpoints update teams and keys in one sorted pass Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test(accounting): prove a ledger delivery waits instead of deadlocking Real Postgres tests hold the first lock of a team roster change, an access group update and a budget reset cascade, start a delivery, and let the other writer finish. The delivery must wait and then charge once. Before the lock order fix each case ends in a Postgres deadlock Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * feat(accounting)!: make the SpendLogs ledger the default accounting path use_spend_logs_ledger now defaults to true, so every request is charged through LiteLLM_SpendLogDelivery. Setting it to false rolls back to the in-memory and Redis spend queues, which stay in place. Tests that cover those legacy writers pin the setting off, and the rest check the queued charge the ledger stores Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test(proxy): let the mocked client open a transaction for key and team deletes Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(proxy): take team locks first when a model rename rewrites allowlists A rename locked team rows before user rows while a roster change holds the team advisory lock and a user row, so the two could deadlock. The rename now takes the advisory lock for every team that lists the model, sorted, before any row lock. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(accounting): drain a backlog in one worker run, one batch per transaction One run applied a single fresh batch, so at 500 per batch charges fell behind any load above that rate. The worker now keeps claiming fresh batches while they come back full, for up to five seconds, and each batch still commits on its own. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(accounting): queue only the charge when a request's spend log is suppressed With logging off for a request, the ledger still queued the full row, so a failed flush or a shutdown could leave its prompt and response in Redis. The queue now holds only the fields the charge needs. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * style(proxy): format the rename team listing query Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(accounting): remove expired delivery receipts as fast as the worker applies them The worker now applies several batches per run but still removed one cleanup batch of applied receipts, so the table could outgrow its retention window. Cleanup now repeats while each pass removes a full batch, for up to five seconds. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(proxy): retry a model rename when a team starts listing the model mid-lock The team advisory locks come from a snapshot, so a team that gained the old name before its row lock was locked without its advisory lock. That attempt now commits without writing and the next one locks the fresh set of teams. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Joshua Valluru <326636767+joshua-berri@users.noreply.github.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-05 09:14:10 -08:00
ishaan-j-berri	71fde48016	feat(alerts): show recommended model provenance and provider notices (#50162) ... * feat(alerts): show recommended model provenance and provider notices * fix(ui): avoid nested recommendation source conditionals	2026-11-05 04:34:17 +00:00
Krrish Dholakia	c5da69cd13	feat(ui): show model deprecation banner on the models page (#50080) ... * feat(ui): show model deprecation banner on the models page Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * style(ui): format model deprecation banner Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(ui): refresh model deprecations when models change Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(ui): address review feedback on model deprecation banner Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(ui): cover deprecation role gate and agent builder refresh Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(ui): separate deprecation banner toggle from alerts link Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(ui): dedupe and allow dismissing the model deprecation banner Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(ui): refresh table once and key banner entries on successor Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix: quiet successor logs, count banner models, use local dates Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(ui): pin banner test date and bucket upcoming entries locally Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test: cover mixed-successor log level through the deprecations endpoint Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(ui): move deprecation banner cases into integration tests Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(ui): keep deprecation banner when a refetch fails Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Krrish Dholakia <krrishdholakia@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-11-04 18:50:14 -08:00
joshua-berri	906fa7ed28	feat(accounting): add opt-in SpendLogs ledger (#49856) ... * feat(accounting): add opt-in SpendLogs ledger delivery * fix(accounting): preserve missing organization daily spend parity * fix(accounting): preserve ledger receipts when replaying migration * fix(accounting): preserve callback windows and tune ledger batches * fix(accounting): preserve window charges and bound ledger statements * test(accounting): verify budget callback registration order * test(accounting): verify captured charges after repeated startup * feat(accounting): store durable payloads independently of spend logs * fix(accounting): preserve first delivery and trace ledger writes * fix(accounting): keep SDK imports independent of proxy tracing * fix(accounting): honor log retention during batch takeover * test(accounting): cover retained batch alerts and invalid settings * test(accounting): verify downgrade command against pending work * fix(accounting): drain retries without starving fresh deliveries * test(accounting): cover delivery projection, metrics and window key reads Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test(accounting): read delivery metrics without the shared registry Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(accounting): charge a leftover $0 batch row when its claim fails A failed batch claim buffers the retrieve's row and reports it charged. At flush the row conflicted with the pre-upgrade $0 SpendLogs row, so its delivery was rolled back and the charge never reached the spend tables. The flush now takes that row over in the same transaction and keeps the delivery. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(accounting): trace the batch takeover at flush as a SpendLogs update Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(accounting): keep local spend logs when SPEND_LOGS_URL has no export client The export client is never created, so main always writes SpendLogs to the proxy's own database even with SPEND_LOGS_URL set. The ledger skipped local rows whenever the URL was set, which dropped those request logs. It now treats a flush as an export only when the client exists, and batch claims write local rows as before. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(accounting): keep every request's charge when its log row is shared, evicted or rejected - Key non-batch deliveries on a per-request delivery_id so a reused client call id is still charged once per request, while buffer replays stay deduplicated - Persist the redacted api_key instead of an opaque bearer token - Keep the accounting payload when the spend-log queue evicts a row or Postgres rejects it - Strip NUL from payloads and skip non-text tags so projection never parks a paid request Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(accounting): isolate a raw-SQL spend log row Postgres rejects instead of requeueing the batch A rejected row on the ledger insert path raises RawQueryError, which the exact DataError check missed, so the whole batch was requeued every flush. Data and integrity SQLSTATE classes now bisect like before, while missing tables and deadlocks keep retrying Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(accounting): keep charging a request whose tag is too long for the daily tag index A tag past Postgres's btree entry limit failed the DailyTagSpend upsert, and on the ledger path that rolled back the request's key, team and user charges with it. Such tags are now left out of the daily tag rollup only Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(accounting): store only guardrail billing fields in the accounting payload The ledger payload copied whole guardrail records, including responses and match details, into LiteLLM_SpendLogDelivery even when spend logs were disabled. It now keeps the id, name, usage and cost fields the projection reads. NUL stripping reuses safe_json_structure instead of a recursive walk Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * refactor(proxy): move spend payload stamping out of update_database Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test(proxy): restore the modules the import-layering guard evicts The guard popped litellm.proxy.utils from sys.modules for good, so later tests on the same xdist worker patched a fresh module while calling functions bound to the old one, and spend queue tests failed depending on scheduling Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * refactor(proxy): keep update_database's payload stamping on its own local Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * refactor(accounting): type guardrail entries and use a public key redaction helper Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(accounting): keep charging requests whose end-user id is too long for an index An oversized end-user id failed the SpendLogs insert with SQLSTATE 54000, which was requeued forever, and failed the end-user upsert in the delivery, which parked the key and team charge with it. Class 54 rows are now isolated like other rejected rows, and such ids are left out of the end-user rollups only Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * refactor(proxy): type the spend log queue snapshot Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(accounting): leave out any projected row Postgres cannot index or store Caller-supplied ids and counters, such as a long tag or end-user id or an out-of-range compression count, failed the delivery transaction and rolled back the request's key, team and user charges. One check in project() now drops only the row that cannot be stored, replacing the per-field cutoffs Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(accounting): keep the guardrail fields compression savings are counted from Slimming the guardrail records dropped the provider and the tokens_saved figure that Headroom compression savings are read from, and dropped a bare-dict record entirely. The payload now keeps both while still leaving out guardrail content Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(accounting): cap projected counters so daily totals cannot overflow Counters such as a caller-supplied compression count fit one by one but overflowed the bigint daily total once summed across requests, rolling back the key, team and user charges in the same transaction. Each counter is now capped at 2^40, far below any total that could overflow Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(accounting): let a buffered batch charge take over its zero-cost receipt A batch charge queued after a failed lookup hit the $0 receipt an earlier poll left behind, and ON CONFLICT dropped it. The buffered insert now runs the same receipt takeover as a live retrieve Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(accounting): read a rescued batch row's spend from its accounting payload Rows rescued from an overflowing queue keep only the accounting payload, so the buffered takeover never saw their positive spend and left the $0 receipt uncharged Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(accounting): read budget windows through the delivery transaction Window seed and existing-row reads used a separate connection while the delivery transaction held one, so with a one-connection pool they waited on their own transaction until it timed out. Inside an enclosing transaction they now share its connection Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(accounting): retry a transient failure while keeping a dropped row's charge The rescue swallowed every error, so a database blip right after a row was rejected lost the unpaid charge. Only a rejected row is still logged and skipped, anything else propagates so the flush retries Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test(accounting): give delivery projection payloads their charged entities project() now reads entities only from accounting_entities, so the hand-built delivery payloads carry them like real stored payloads do. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * refactor(accounting): own ledger rows by their payload and key batches on the batch id Drop the accounting_path stamp and its SpendLogs column. A buffered row that carries an accounting payload is ledger owned, one without it is legacy, and spend counters get a plain windows_from_ledger flag. Window seeding tells applied ledger rows apart by their delivery row. A ledger batch is plain transaction A keyed by the batch id, so a repeated completed retrieve conflicts and charges nothing. A $0 retrieve records no delivery, a batch a legacy retrieve already charged is not charged again, and the legacy batch path is unchanged from main apart from refusing a batch the ledger already owns. Batch takeover and its delta bookkeeping are removed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test(accounting): drop a projection test duplicated while rebasing Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * refactor(accounting): drop previous_payload now that deliveries are never replaced Batch takeover was the only writer of previous_payload, so the worker applies each delivery's payload as stored and the column leaves the migration. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * refactor(accounting): drop the ledger downgrade checklist script The use_spend_logs_ledger flag is the rollback path, so a separate downgrade gate is no longer needed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(accounting): name the window seed reads so their Prisma span resolves Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * refactor(accounting): remove the unused accounting event types and protocols types.py, protocols.py and measurements.py were left from the old event scaffolding. Nothing outside the package imports them, so they go with their tests. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(accounting): park only deliveries whose own data fails A delivery that keeps failing on a database outage, lock timeout or deadlock stays pending and retries with capped backoff, so a recovered database applies it. Only a data error or an invalid payload parks the isolated row. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(accounting): isolate a spend log row the insert statement rejects The raw SpendLogs insert validates columns and JSON before Postgres sees the row, so a row that fails there is now bisected out and dropped alone while its charge is kept, instead of failing and replaying the whole flush. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(accounting): carry a job's model window on every completed poll in ledger mode The ledger keeps the poll whose delivery claims the job, which need not be the poll that won the limiter's in-memory charge. Every completed poll now carries the job's window, so the delivery that lands always has it, while the live model budget is still charged once. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * refactor(accounting): leave model-budget windows on the existing window writer Model-budget windows are written by the model budget limiter rather than derived from a request's spend, so they stay on main's window queue like live budget counters do. This drops window capture from the accounting payload, the deferred limiter registration and the job-window capture, and the ledger keeps owning key and team budget windows. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(accounting): keep a delivery pending when the database is unreachable Prisma reports a can't reach database server outage as a DataError, so the worker now checks for an unavailable database before treating a data error as the delivery's own fault. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(accounting): isolate only rows the raw insert cannot encode Catching every ValueError also swallowed unexpected insert errors that main re-raises. The raw insert now raises RejectedSpendLogRow for an unknown column or non-JSON value, and only that is isolated as a rejected row. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(accounting): store zero savings when caller data makes them uncomputable A huge caller-supplied tokens_saved overflowed the savings math inside build_accounting_payload, which dropped the request's charge in ledger mode. Savings that cannot be computed are now recorded as zero and the charge lands. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Joshua Valluru <326636767+joshua-berri@users.noreply.github.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-04 17:49:15 -08:00
Mateo Wang	80b18d165b	chore(proxy): use neutral example names for baggage tag mappings (#50011)	2026-11-04 14:49:55 +00:00
tin-berri	48bd9b01f6	feat(skills): upload local skill ZIPs from the dashboard (#50008)	2026-11-04 00:56:40 -08:00
yucheng-berri	ffbbb6366b	feat(proxy): add general_settings.fips_mode startup gate (#48600) ... * feat(proxy): add LITELLM_FIPS_MODE startup gate With LITELLM_FIPS_MODE=true the proxy refuses to start when TLS verification is off, when Python's OpenSSL does not enforce FIPS, or when the startup password migration fails. A value that is not true/false also refuses. * feat(proxy): refuse FIPS mode when cryptography's OpenSSL is not FIPS The stock cryptography wheel bundles its own non-FIPS OpenSSL, which serves JWT, Fernet and AES-GCM even when Python's OpenSSL enforces FIPS. * fix(proxy): refuse FIPS mode when cryptography's FIPS provider cannot hash The stock cryptography wheel reads the host's FIPS config and reports FIPS on, but its bundled OpenSSL cannot load the host's FIPS provider, so every operation fails. Require a working SHA-256 alongside the FIPS flag. * fix(proxy): refuse TLS verification off anywhere in FIPS mode A model's own ssl_verify: false reached the HTTP client after the startup check passed. get_ssl_verify now raises in FIPS mode whenever verification resolves to off, which also covers the startup check. * refactor(proxy): return FIPS boot refusals as values and harden the FIPS tests * fix(proxy): stop refusing TLS verification off in FIPS mode Skipping certificate verification is a security choice, not a FIPS one: the TLS connection still only uses algorithms the FIPS provider allows. * feat(fips): refuse a database used without FIPS mode, drop the password migration change FIPS mode marks a new database with a fips_mode row in LiteLLM_Config and refuses one that already holds LiteLLM rows without it. Without FIPS mode, the proxy refuses a database carrying the marker. * test(fips): close the database test clients with aclose so the engine stops * feat(fips): read FIPS mode from general_settings.fips_mode instead of LITELLM_FIPS_MODE * refactor(fips): keep fips_mode out of the API-writable general settings, it is read from the config file only * fix(fips): retry the startup marker and state reads once after a dropped connection A dropped connection during the non-FIPS marker read used to count as an unmarked database, so a blip at boot let a non-FIPS proxy start on a FIPS database. Both reads now go through call_with_db_reconnect_retry, the reconnect-and-retry the other startup reads use. A read that still fails keeps its old outcome. * fix(fips): refuse a fips_mode key that has no value fips_mode: os.environ/X with X unset, or a bare fips_mode: / null, resolved to None and started without FIPS mode and without a message, while an empty string refused. A key that is present now has to be a boolean; only a config without the key means off. * refactor(fips): move the database check out of the boot gate The boot gate keeps the setting and the two OpenSSL checks. The check that a database matches the FIPS setting moves to its own PR, so a boot without the flag runs no extra database read. * test(fips): pin how a fips_mode string is read through enforce_fips_mode	2026-11-03 14:56:14 -08:00
Emerson Gomes	ecb8ef5ca8	feat(realtime)!: support latest OpenAI audio models (#35600) ... * feat(realtime): support latest OpenAI audio models * fix(transcription): type SDK stream duration metadata explicitly * fix(realtime): preserve transport settings and translation duration * refactor(realtime): reuse SDK client initialization * fix(audio): bill finalized streams and authorize transcription models * refactor(audio): keep transcription pricing selection together * fix(realtime): reconcile model metadata and test contracts * fix(realtime): preserve provider duration and model capabilities * fix(realtime): keep transcription validation compatible with cost map guard * fix(realtime): address transcription and billing review findings * fix(realtime): preserve input usage and provider routing * fix(realtime): bill retained output and route Azure aliases * fix(realtime): reject malformed provider durations * fix(realtime): preserve batch cache rates and Azure v1 routes * fix(transcription): reject nonfinite provider durations * refactor(realtime): align annotations with current lint rules * fix(realtime): preserve translation deployment pricing and input usage * test(transcription): verify synchronous stream finalization * fix(realtime): retain missing translation input usage at close * fix(transcription): reject streams that bypass output guardrails * fix(realtime): reconcile audio protocols billing and current CI * fix(realtime): trust confirmed formats and authorize default models * fix(realtime): preserve absent models and reject unsupported guardrails * fix(realtime): reject translation policy pipelines * chore(ci): rerun transient infrastructure timing failure * fix(realtime): reconcile audio protocols with current upstream * fix(audio): validate provider responses at SDK boundaries * fix(realtime): preserve minimum OpenAI SDK compatibility * fix(realtime): preserve deferred Live session authorization * fix(realtime): preserve upstream authentication and transcription behavior * fix(audio): scope provider duration precedence to configured models * fix(typing): avoid Final declarations inside transcription cost loop * refactor(audio): reuse transcription mapping and duration resolution * fix(realtime): enforce model budgets and preserve translation routing * fix(audio): log streamed transcripts whose done event reports detected languages * fix(audio): pass stream, languages and keywords through unchecked for non-OpenAI transcription providers Providers without a streaming transcription handler got UnsupportedParamsError for stream=True, languages and keywords, where the merge base forwarded them as provider-specific params. Only openai and azure validate them now. * fix(audio): bill streamed gpt-transcribe from the provider's reported duration The stream path computed cost after logging had replaced the transcription's duration usage, so it fell back to the measured file length while the unstreamed call billed the provider's seconds. * fix(transcription): log streamed transcription usage from the provider response The stream logging path built the spend payload from the response after its duration usage was transformed away, so whisper-1 with stream=true and streamed gpt-transcribe logged an empty usage object. Bill and log from the provider's TranscriptionResponse in both the sync and async paths. * fix(realtime): keep main's routing, protocol and transport behavior around the new translation routes - Send OpenAI realtime and translation client_secrets and calls through litellm's httpx client again, so ssl_verify and custom CA settings apply and upstream errors keep main's shape - Use the input transcription model as the session model only for transcription sessions - Drop the new /openai/v1 translation aliases and record each lazy feature's own routes, so the /openai pass-through and the MCP routes resolve as on main - Let a configured Azure realtime_protocol win for every model, with GA as the default only when none is set - Return x-litellm headers on streamed transcriptions and mark the request done - Refuse translation sessions before connecting upstream when realtime guardrails apply or the provider is not OpenAI or Azure - Accept json and text for gpt-transcribe, and drop other formats under drop_params instead of failing - Bill translation sessions on output audio seconds only, at the single published rate - Document the 201 SDP answer on the translation calls routes and the transcription 200 response * fix(realtime): restore main behavior for guardrails, Azure transcription, and realtime clients Streamed transcription runs unstreamed when a deployment post_call guardrail applies, so the guardrail still sees the transcript. Azure transcription maps v1, latest and preview to the default dated version on the deployments route for every model. The SDK client secret keeps session.model ahead of model, realtime WebSockets connect to api_base again, realtime metadata keeps only guardrails, and managed agents need an explicit transcription model. Reverts the two unrelated streamed Responses error codes and the duplicate cached_tokens_details field, and the gpt-transcribe format check only reads the request provider's cost map entry. * fix(realtime): keep main's transcription, client secret and route behavior where the PR changed it Transcription reaches every provider as on main again, sends stream only when asked, forwards language and languages together, keeps form key order, bills provider seconds only when above zero, honors base_model for the new flags, keeps main's Azure api_version on the proxy path, and streams carry chat's headers, an error event on a mid-stream failure, and a spend row when the stream ends early. Translation cost reads both per-second rates from one source and spend rows carry the billed seconds. Realtime transcription WebSockets keep model in the upstream URL outside OpenAI's hosts, client secrets forward explicit nulls, accept conversation and delay, and answer 400 on a malformed body, the /openai translation paths are back to main, translation refuses only for guardrails realtime sessions run, nested aliases resolve and later session.update models are authorized, translation client secrets run per-model auth, beta clients bridged to GA get beta-shaped session events, and transcription sessions send the deployment model in both shapes. * fix(realtime): address review findings on transcription billing, streaming and realtime auth Per-second pricing replaces token billing only when the priced entry sets no token rates, translation rates fall back per key to the cost map, transcription cache keys include languages and keywords, a stream request a post_call guardrail forces unstreamed comes back as redacted SSE, Azure beta clients are bridged only for GA-only deployments, nested transcription models on the realtime WebSocket are checked against the key, and translation spend carries key and team metadata * fix(azure): pick the GA realtime protocol for deployments whose model_info.base_model is GA-only * fix(realtime): preserve prior merge resolutions after rebase * fix(realtime): enforce translation policies and normalize Azure transcription URLs * fix(cookbook): stop translation once output idles after the upload ends * fix(realtime): bill translation on input audio and drop pre-read stream cost headers Translation spend now uses the input audio seconds the session sent, priced through input_cost_per_second, instead of the translated output duration. A streamed transcription no longer returns cost headers before the stream is read, since the cost is only known after the last event. Also removes an unused realtime guardrail helper. * fix(realtime): drop top-level session nulls, bill dropped transcription streams, and bridge beta response fields Client secret, translation and transcription session bodies drop top-level nulls again, as main did, while nested nulls such as turn_detection still pass through. Transcription client secrets send the deployment model as session.model. A streamed transcription the client drops still runs its success callbacks, and the stream ends with data: [DONE] like OpenAI's. The beta to GA bridge maps max_response_output_tokens, speed and noise reduction, remaps response.create, and drops temperature under drop_params. The Entra token for realtime HTTP routes is fetched off the event loop * test(e2e): cover realtime translation, live transcription, client secrets, and streamed transcription * fix(realtime): close review gaps in translation billing, transcription streams, and Azure GA Bill translation sessions from the deployment's own per-second rates, add per-second prices for Azure gpt-4o transcribe models, keep stream-scoped guardrails seeing the caller's stream flag, map mid-stream transcription errors to LiteLLM exceptions with a [DONE] frame, map GA responses back to the beta shape for bridged beta clients, honor the minting key's guardrail opt-out on translation calls, restore the Azure transcription session URL, and flag GA-only Azure realtime models * fix(realtime): bill a translation rate the deployment omits from the session model * test(e2e): check translated speech is not English instead of exact French wording * fix(realtime): keep main's behavior for non-bridged backends, stream error frames, gateway URLs and Azure transcribe prices Remap beta events to GA only for bridged beta clients and translation sessions, bridge beta clients on GA-pinned Azure deployments, log error-only upstream sessions as failures, flag dated Azure GA realtime models, drop the unpublished Azure per-second transcribe rates, log a mid-stream transcription failure once, keep main's SSE error frame outside transcription, drop unsupported transcription params under drop_params, keep the caller's Azure auth headers, keep a gateway /v1 path, let key opt-outs release string-callback guardrails on translation, and keep the cookbook waiting while a long upload is still sending * fix(realtime): keep the proxy's realtime session gate off translation sessions A key that opts out of a legacy callback guardrail now gets a working translation session, since the transcription-guardrail gate no longer sends a realtime session.update the translation upstream rejects * fix(realtime): keep websocket model limits, bill measured transcription length, route transcription sessions on the allowed model A WebSocket opened without a query model now authorizes as an empty model, transcription cost prefers the measured upload length over the provider's reported seconds, transcription sessions overwrite nested models with the authorized top-level model again, and a non-boolean transcription stream value answers 400 or is dropped under drop_params. * fix(ci): copy the rust bridge extension into the openapi docs gate's base worktree * fix(model-catalog): accept the realtime translation endpoints in the cost map * fix(cookbook): close the translation session and read until session.closed * fix(realtime): treat an empty transcription stream as unset and pass the cost-map guard An empty `stream` form value runs the transcription unstreamed again, as OpenAI does, instead of answering 400. The four translate cost-map entries drop the translation routes from supported_endpoints until the main branch's catalog parser accepts them, and the nested transcription model lookup uses the router registry entry point. * test(proxy): drop the mid-stream error frame pin that main's error shape change replaced * fix(transcription): bill uploaded seconds when a stream fails mid-transcript * fix(proxy): bill uploaded seconds when a streamed transcription breaks mid-stream * fix(audio): keep transcription's positional arguments and redact streamed error details - Move the new languages, keywords and stream options to the end of transcription's parameters, so a prompt or format passed by position keeps its meaning - Redact internal hostnames, paths and tracebacks from a streamed transcription's mid-stream error event, as chat streams already do - Move the CA-bundle check for the realtime HTTP routes, which serves real TLS on loopback, from the unit suite to the SDK integration suite * fix(audio): stream the guarded transcript to SDK callers that asked for a stream * fix(logging): keep a streamed transcript out of callbacks that turned off message logging A streamed transcription's assembled response reached success callbacks with its text intact under turn_off_message_logging or a callback's message_logging=False, because the shared streaming redactor had no transcription branch. It now redacts the text, as the unstreamed call already does * fix(proxy): stream a transcription the deployment streamed when the client sent no stream field --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-11-03 07:27:52 +00:00
zoveki	704e73d6fe	fix(cost-map): give the bare gpt-4o, gpt-4o-mini, gpt-4.1 and gpt-4.1-mini aliases their 2027-01-14 shutdown date (#49707) ... * fix(cost-map): give the bare gpt-4o, gpt-4o-mini, gpt-4.1 and gpt-4.1-mini aliases their 2027-01-14 shutdown date * fix(proxy): report a deployment's own provider retirement date before the bare model's An Azure deployment declared as model gpt-4o with custom_llm_provider azure, or as azure/<deployment> with base_model gpt-4o, now resolves azure/gpt-4o's date instead of the bare OpenAI row's. The GPT-4 series alias test now reads both cost maps with typed rows * fix(proxy): read Azure's retirement schedule for azure_ai deployments An azure_ai deployment with no azure_ai row of its own now reports the azure/<model> date, and a cost-map row wins over the provider-less copy the router registers under the deployment's key, which carries the bare model's date. * fix(proxy): let custom and registered base_model rows keep their deprecation date Only the copy the router writes at startup, which has no provider and inherits the bare row's date, gives way to a later candidate * fix(proxy): report a pinned snapshot's own earlier deprecation date over its base_model's * fix(proxy): recognize the router's copied cost-map rows by the loaded catalog * fix(proxy): read Azure's schedule for a provider-prefixed azure_ai base_model --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-11-03 02:58:39 +00:00
Etienne Chabert	523b87ec71	fix(proxy): charge budget windows when the cached key or team expired before the request ended (#47608) ... * fix(proxy): charge budget windows when the cached key or team expired before the request ended The post-call spend update reads a key's or team's budget_limits from user_api_key_cache and returned early on a miss, skipping both the window counter increment and the LiteLLM_BudgetWindowSpend row. The cached object expires after user_api_key_cache_ttl (60s by default), so any request that ran longer than that without another request refreshing the entry was counted in the key's lifetime spend but never in its budget windows. With budget reservation disabled nothing else charges the windows, so keys kept running past their window caps On a miss the update now reads budget_limits straight from the key or team row. It does not re-cache the object, since a write from the spend path could race a /key/delete and put a just-deleted key back into the auth cache * test(proxy): assert the budget window fallback never re-caches the key or team * fix(proxy): read a missing key or team row once per cache TTL in the budget window fallback The master key and UI session tokens are logged under names no key row carries, so every one of their requests missed the auth cache and read the database. A missing row is now remembered for the cache TTL in this process only. A failed read is not remembered, and a row that exists is read fresh on every miss so a key update is never served stale. * fix(proxy): skip the budget window row read for a key or team with no windows The fallback remembered only a missing row, so a key or team that exists but holds no budget_limits was read from the database on every request that outlived the key cache. It now remembers both cases under one cache key built in user_api_key_cache.py, for the same TTL as the auth cache * fix(proxy): keep no-window markers in their own cache and clear them when an update adds windows Markers for keys and teams with no budget windows lived in the shared 200-entry auth cache, so a burst of late costs could evict cached teams and users. They now live in a dedicated in-memory cache on UserApiKeyCache, reached by local and broadcast evictions. /key/update and /team/update now evict the marker on every worker, so a late cost after an update charges the windows the update added. * test(proxy): give the spend counter tests the proxy's real key cache and type the window spend helper * fix(proxy): keep a row read that an update outran from restoring the no-window marker * fix(proxy): clear finished no-window reads and evict a marker-shaped user id from both caches * fix(proxy): apply other spend counters before a window row read and let no-window notes follow the key cache settings --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-11-02 14:41:59 -08:00
tessellark	c720b23238	feat(proxy)!: honor model_info.max_input_tokens in the Codex catalog for stock models (#49701) ... * feat(proxy): honor model_info.max_input_tokens in the Codex catalog for stock models A Codex model catalog entry copied from Codex's stock entry (a stock slug, or an alias whose upstream is one) now takes the max_input_tokens every routable deployment of the group sets as its context_window, capped at the stock entry's max_context_window like Codex's own model_context_window. With nothing set, the stock window stays. * feat(proxy): honor the smallest max_input_tokens a group's deployments set in the Codex catalog A deployment without a limit counts as Codex's own window, so a limit on only some deployments can lower the window but never raise it * fix(proxy): skip a zero or negative max_input_tokens when sizing a Codex catalog window * test(proxy): stub the max_input_tokens group field and type the Codex window tests * fix(proxy): skip a max_input_tokens a saved /model/info response copied from the cost map --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-11-02 13:27:58 -08:00
tavren27	a4de720144	fix(count_tokens): count grok-4-1, grok-4.3 and DeepSeek V4.1 tool requests the way their hosts bill them (#49074) ... * fix(count_tokens): count grok-4-1, grok-4.3 and DeepSeek V4.1 tool requests like their chat templates Render grok-4-1, grok-4.3 and DeepSeek V4.1 tool schemas the way each chat template does, and add a per-deployment token_count_tool_framing next to token_count_framing for the hidden tool preamble a host bills. The Azure AI Foundry grok-4-1, xAI grok-4.3 and Fireworks DeepSeek V4.1 cost map rows carry the measured framing, overridable in model_info * fix(count_tokens): count Azure grok-4-1 Responses requests with their own framing and use the tool framing in place of the base framing * fix(count_tokens): keep operator framing on tool and Responses counts, frame bridged Responses like chat, follow base_model for tool rendering, count grok tool_choice * fix(token_counter): count Azure Responses strict default, skip false strict and built-in-only tools, per-kind operator framing * fix(token_counter): count an explicit null strict like false on Azure Responses * fix(count_tokens): follow the chat completions bridge flag and Azure's strict default on azure/ deployments Keep Azure grok-4-1 chat tool counts at or over the bill by raising its tool framing to 302 * fix(token_counter): keep strict false for non-grok templates, coerce the bridge flag, frame DeepSeek API counts * fix(proxy): frame Responses counts for the api_base router defaults send inference to * fix(proxy): keep the api_base helper's name so main's /v1/messages count caller resolves --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-11-02 12:37:35 -08:00
zoveki	33457d3ac9	fix(proxy): check the model in every OpenAI websocket passthrough frame (#48979) ... * test(proxy): cover per-frame model checks and session pricing on the OpenAI websocket passthrough * fix(proxy): check the model in every OpenAI websocket passthrough frame and price the session * fix(proxy): bill each OpenAI websocket turn on its own row and run full auth per frame Each finished turn (completed, incomplete, or failed) now logs its own spend row under the model and access groups that allowed it, so group budgets see spend as turns end instead of at close. Every model a frame names goes through the key's full auth, and a stored prompt reference with no model is refused for a key held to some models. A frame still in flight when the client leaves is recorded before the relay gives up on the send * fix(proxy): keep the stored prompt refusal's first sentence inside the websocket close reason The close reason carries the refusal's first sentence, and the stored prompt sentence ran past the 123 bytes a close frame allows, so it was cut mid-word. The sentence is shorter now and the rest of the explanation stays in the error event. * fix(proxy): bill pipelined websocket turns to their own frame and close the review gaps * fix(proxy): log no extra websocket spend row for frames after the last turn * fix(proxy): price websocket Responses turns by served tier and hosted tools, and bill turns a client walked out of A websocket Responses turn is now priced through the same Responses cost path the HTTP passthrough uses, so a priority-tier answer and a hosted web search cost what they cost over HTTP. When the client leaves mid-answer, the gateway keeps reading the provider's socket until that turn ends, up to 10 minutes, so the turn the provider still bills reaches spend * fix(proxy): bill websocket turns by lane, skip turns OpenAI refuses, and stop frame checks counting the session's own reservation An error event refuses a queued response.create, so the next turn is no longer charged to the refused frame's model. Turns on separate stream_id lanes are charged to the frame that started each one. The frame model gate settles the connect reservation before it re-runs auth, so a key with less budget left than one turn's max cost is no longer refused on its first frame. * fix(proxy): bill each websocket lane cut off at session end on its own row When a client leaves while turns on several stream_id lanes are still running, each unfinished turn now gets its own spend row under the groups and model its own frame asked for, instead of one merged row. Also pins the PR's new single-assigned test variables as Final and swaps datetime.now() in the logging-handler and success-handler fixtures for fixed timestamps. * test(proxy): mark the remaining new websocket test variables Final Builds the logging mocks with model_call_details in the constructor instead of setting it afterwards. * fix(proxy): check every model the realtime URL names before the OpenAI websocket relay connects * fix(proxy): keep reading running answers after a refused websocket frame so they are billed * fix(proxy): bill realtime transcription sessions on the OpenAI websocket passthrough A transcription session names its model only in session.updated under audio.input.transcription, and reports usage on each completed transcription instead of response.done, so its row logged unknown at $0 and a budgeted key never spent. * test(proxy): mark the turn-taking client's delivered list Final * fix(proxy): hold a websocket session's parallel slot until the socket closes Each finished turn logged its spend row under the session's own call id, so the first turn's success callback freed the key's max_parallel_requests slot while the socket was still open, and a key limited to one session could open several at once. The session now claims its slot under its own call id, every turn row logs under a call id of its own, and the slot is released when the session ends, as before Priced turn rows also take their row's call id as their id, so two turns that start in the same instant no longer share a spend log id and drop one row * fix(proxy): bill realtime transcription clips that finish outside an answer A conversation.item.input_audio_transcription.completed frame that arrived after a session's last response.done was never billed, because the session end only owes a row for a turn still running. A clip that completes while no answer is running now logs its own row, and a clip inside a running answer stays in that answer's row Each row also carries the session's latest session.updated frame, since later rows lost the transcription model it names and priced clips against the realtime model. A newer session.updated replaces the older one, so the stale model is never read first It also drops the docstrings this PR had added, since the names already say what each helper does * fix(cost): price a realtime transcription clip at the transcription model the session set A realtime conversation session opens with session.created naming the realtime model and no transcription model, and the client sets the transcription model later in session.update. The cost code took the first session frame and fell back to its session model, so a clip was priced at the realtime model. That model has no per-second rate, so a gpt-transcribe clip cost $0, and a token-billed clip used the realtime model's audio rate instead of the transcription model's. Now the newest session frame that names a transcription model sets the price, and the first session model is only the fallback when no frame names one, as before * fix(cost): price each realtime transcription clip at the model set when it ran A session that changes its transcription model mid-way kept every logged clip on the newest model. Each clip now takes the transcription model from the last session event before it, and falls back to the session's own model * fix(passthrough): keep the session update a buffered clip ran under until its answer is billed A transcription that completes while an answer streams waits in the buffer for that answer's response.done. A session.updated in between used to drop the earlier update, so the clip was priced at the opening configuration. An earlier update now stays while an unbilled clip follows it, and drops at the next update once that clip is billed. * fix(passthrough): keep reading a websocket turn the client left before OpenAI started it A client that sends response.create and hangs up before OpenAI answers with response.created made the walkout checks see no turn in flight, so the gateway closed the provider socket and the turn OpenAI still ran and billed never reached spend. The relay now counts every forwarded create until the provider starts or refuses it, and keeps draining while one is owed * fix(passthrough): settle a websocket turn that ends without its start frame A turn the provider ended without sending response.created first left its create counted as not started, so a client leaving later held the provider socket for the whole walkout drain with nothing left to bill. A turn's end now settles one waiting create when its lane saw no start frame * fix(proxy): refuse unparseable and unroutable websocket frames and hold legacy slots per socket A client frame too deep or too long for the JSON parser closed the provider socket mid-turn, so the turn in flight was never billed. The gate now refuses that frame with an error event and keeps draining the open turn. A stream_id OpenAI rejects comes back as an error that names no lane, which dropped another turn's authorization. The gate now refuses those frames before they reach OpenAI. With the legacy parallel request limiter, each per-turn spend row freed the key's slot while the socket stayed open. Websocket passthrough rows now free nothing, and the socket frees its slot once when it closes. --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-11-02 09:48:59 -08:00
Nate Armstrong	87d91d78b7	refactor(router): route the router family through RoutingContext and close the outside surface (#49118) ... Strategies, pre-call checks, cooldown and fallback helpers, and the auto, complexity and quality sub-routers now take a RoutingContext instead of the Router, and read the router only through registry(), settings(), health() and the context's forwarding methods. Router-free helpers move from router_utils to litellm_core_utils, outside callers of router state switch to entry points, and the surface allowlist shrinks to the endpoint methods plus the entry points	2026-10-30 19:18:54 -07:00
Kerry Lu	298911d61b	fix(batches): cost an unmanaged gs:// batch through the deployment that created it (#49301) ... * fix(batches): cost an unmanaged gs:// batch through the deployment that created it The create now stores the router deployment id as model_id inside the managed object row's file_object JSON and CheckBatchCost reads it before falling back to the first deployment matching the base model in the gs:// path. Rows without the field keep the old lookup * fix(batches): tolerate a non-string or malformed file_object when reading the stored deployment * fix(batches): read the stored creating deployment from the writer, not a lagging read replica * fix(batches): resolve the writer database once so the type gate count does not grow * fix(batches): decode a dict-shaped file_object when reading the stored deployment, accumulate wire samples, annotate test locals --------- Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-10-30 15:25:47 -07:00
Mateo Wang	cbfb1abdb3	fix(proxy)!: label shared-handler errors the way the route catch-alls do (#39555) ... * fix(proxy): name shared-handler errors by the status they answer handle_exception_on_proxy pinned type=internal_server_error on every exception it wrapped, so a management route rejecting a caller's own argument answered {"type":"internal_server_error","code":"400"} and an SDK branching on the type read its own bad request as a gateway outage. Derive the type from the status the response actually carries, keeping a type the exception already names. * test(proxy): pin a 500 for exceptions whose status is None * fix(proxy): keep internal_server_error for timeouts and 5xx, redact hosts in shared handler errors Provider 5xx and timeouts keep internal_server_error so retry loops still retry them, the shared handler redacts internal hosts from raw exception messages, and /key/list routes its HTTPException through the shared handler instead of its own copy of the old label. * fix(proxy): label shared-handler errors like the chat route, pick the status before the type, redact only gateway failures * fix(proxy): pass shared-handler error messages through as on main Redacting timeout and 5xx messages changed what every shared-handler route returns, which relabeling the error type does not need, so messages go back to the verbatim text main sends. * fix(proxy): redact only the status-less failure main never sent * fix(proxy): answer a generic 500 for a status-less failure and expect auth_error on the /user/new 403 A failure with no status now answers "Internal server error" instead of its own text, since that text can name a guardrail's private host and pattern redaction misses cluster DNS names and IPv6 addresses. The /user/new integration test now expects the auth_error type main's route catch-alls give a 403 the proxy raises * fix(proxy): answer only a 4xx or 5xx carried status and fall back to the route default otherwise A numeric-string status outside 400 to 599 made /v1/messages, images, rerank and the OpenAI pass-through answer HTTP 200 with an error body or drop the connection, and non-ASCII digits or a 5000-digit string slipped through isdecimal. error_status_code now accepts an int or a three-digit ASCII string in 400 to 599, the range the chat route already enforces * refactor(proxy): parse a three-digit string status without recursing	2026-10-30 06:58:05 -07:00
tessellark	080e643edc	fix(proxy): apply new model rows on every worker as soon as they are written (#49034) ... * fix(proxy): apply published model rows on every worker before the debounced resync Config sync messages now carry the ids of model rows a create, update, or upsert wrote. Every worker that receives one loads those rows by id right away, including while its debounced full resync is still waiting, so a group added one deployment at a time is complete on every worker as soon as the write returns. Deletes, bulk writes, and messages from older pods carry no ids and keep the full resync path * fix(proxy): load published model rows from the primary under the reconcile lock Read the by-id rows through the writer so a lagging read replica cannot put an old row back, and read them inside MODEL_RECONCILE_LOCK so a concurrent delete or reload stays the last word. Look the ids up in bounded chunks, reload DB credentials only when a row names one this worker lacks, and leave auto-router rows to the debounced full resync. Add an integration test that boots two proxies and needs the peer to fall back to a just-added backup within 4 s. * test(proxy): cover a published model row that names no credential * fix(proxy): keep the unknown model name read-through on the replica and outside the lock Only the by-id load that follows a published write reads the primary inside the reconcile lock. A request for a model name a worker does not know reads through the replica routing before taking the lock again, as on main, so unknown names put no load on the primary and do not wait behind a reload. * fix(proxy): keep access group edits from dropping staying deployments and batch published config ids * fix(proxy): read access group rows from the primary so quick edits are not lost to a lagging replica * fix(proxy): install ordinary rows queued with an auto router and check access group edits before writing A batch of published model ids that held one auto router skipped every row in it, so ordinary rows written at the same moment waited for the full reload. The by-id load now leaves only the auto router rows to that reload. An access group update now reads every deployment from the primary, answers 404 for an unknown group and 400 for an unknown model id or name before it writes anything, then removes the group from the deployments that leave before it adds it to the ones that join, so a failed write leaves the group narrower than asked, never wider. * fix(proxy): write an access group update in one transaction after a fresh read The update now checks model names first, reads every deployment from the primary, and commits every join and leave in one batch right after that read, so a failed write changes nothing and a field another admin set during validation is kept. Other workers hear about the rows only after the commit. * fix(proxy): lock the model rows while an access group update reads and rewrites them * test(proxy): run the model resync tests on the real installer and a real router The by-id resync tests now install rows through ProxyConfig's real installer and credential loader on a real Router, faking only the database, and the config sync test row model is frozen. * fix(proxy): lock only the rows an access group update touches and answer 404 before 400 An access group update locked every model row, so a lock on any unrelated row stalled it for 30 s and then failed it with a 500, and unrelated model edits on other proxies waited behind it. It now takes a per-group advisory lock, locks only the rows that carry the group or that it selects, and reads them again once locked. The group existence check runs before the model name check again, so a missing group answers 404 as on main. * fix(proxy): never wait on a locked row in an access group update The update now takes its row locks with NOWAIT. When another edit holds one of the group's rows, it rolls back, holding nothing, and tries again for about 10 s before answering 409. Holding rows while waiting deadlocked against team deletes and let a concurrent model edit undo the update's change. The per-group lock also moves to the two-key advisory space, apart from team ids. * fix(proxy): retry access group row locks for 30 s with SKIP LOCKED and lock only the rows model_ids select * fix(proxy): take the access group lock with pg_try_advisory_xact_lock and retry while another edit holds it --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-10-30 03:19:25 -07:00
Mateo Wang	326f17594e	fix(batches): return real batch token counts next to OpenAI's batch usage fields (#47740) ... * fix(batches): read OpenAI batch usage into the batch token counts * fix(batches): serve OpenAI's batch usage shape from retrieve and the cost poller * fix(batches): keep the chat usage keys next to OpenAI's batch usage fields Batch usage now carries every chat key it had plus OpenAI's five batch fields with the same counts, so poller-billed batches keep prompt_tokens and cache-write tokens, and older proxies still read the new rows. Usage with missing or null detail counts reads its real input and output counts. * test(batches): check the batch usage counts in the e2e terminal-state test * fix(types): apply nested include and exclude to both batch usage shapes * fix(batches): log no cost for a batch retrieve that is not final yet A retrieve now reads OpenAI's batch usage into the chat counts, so a retrieve that saw the batch completed before its output file arrived priced those counts for the logging callbacks, and the final retrieve priced the whole batch again. A batch that is not final now logs no cost, as it did before. Batch usage dumps also pass exclude_unset and exclude_defaults through to the usage keys. * fix(batches): pass through every usage detail field the provider sends Batch dumps rebuilt input_tokens_details and output_tokens_details from the chat keys, which dropped any detail field past cached_tokens and reasoning_tokens that the merge base passed through. The dump now starts from the provider's stored details and only sets the two counts derived from the chat keys * fix(batches): skip sync loggers on the cost poller's own retrieve and honor exclude_none on provider details * fix(batches): keep sync logger events at $0 on poller-owned retrieves, pass unparsed provider counts through, log batch usage in chat shape * fix(batches): store only chat usage extras on batch spend rows Spend rows for batch retrieves copied OpenAI's batch fields (input_tokens, output_tokens and their details) into metadata.additional_usage_values next to the chat counts they duplicate. Build the row from the batch's chat usage, shared with the logging payload through batch_chat_usage.	2026-10-29 18:29:30 -07:00
quietjules	d302096b5a	feat(lite): offer each model group's reasoning efforts in pi, OpenCode and Codex (#48949) ... * feat(lite): offer each model group's reasoning efforts in pi, OpenCode and Codex lite pi, lite opencode and lite codex now read supported_reasoning_efforts from /model_group/info and write them into each client's model config. pi gets reasoning and a thinkingLevelMap, OpenCode gets reasoning_options plus one variant per effort, and Codex fallback entries get supported_reasoning_levels with a default level. The proxy-served Codex catalog on /v1/models?client_version= offers the same levels Codex before 0.138.0 fails the whole catalog on an effort it cannot name, so those releases, and an unreadable version, get only the six efforts every release parses. A group with unknown efforts keeps today's output, and a group with none is written as non-reasoning The strength order and nearest_declared_reasoning_effort move to an SDK-free module so the lite CLI, which loads without initializing the SDK, can import the Codex catalog * fix(lite): describe team models by their public name and keep Codex's none default /model_group/info now looks up a team's own deployments when the caller's team serves the requested name, so a team model reports its reasoning levels. Codex fallback entries default to none when the group lists it, and the proxy builds the Codex catalog in a worker thread * fix(lite): round Codex's default level down and describe a team's shared-name model from its own deployments Move the reasoning strength helper to litellm_core_utils so lite and the Codex catalog import it without the SDK or the router, and let make lint-format-check-changed diff the working tree so a staged rename passes. * fix(lite): keep pi's system role on reasoning model groups pi switches its system prompt to the developer role for any model marked reasoning, so an operator's cache_control_injection_points entry for role system stopped matching on Claude groups. lite pi now sets compat.supportsDeveloperRole false on those entries, so pi sends system as it did before. * fix(codex): treat a client_version too long to parse as an old Codex and keep undescribed stock levels A client_version past Python's integer digit limit made the Codex catalog answer 500. It now gets the levels every Codex parses. A stock Codex level with no description validates again, as on base, and lite codex treats undecodable codex --version output as an unknown version * fix(lite): keep REASONING_EFFORT_STRENGTH_ORDER importable from router_utils and skip the Codex sync on undecodable debug models output --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-10-29 18:15:44 -07:00
Nate Armstrong	ead221995e	refactor(router): move router callers onto the entry points and invert the proxy imports (#48847) ... Code outside the router now reaches it only through the entry points (registry, settings, health, select, apply, observe, register_plugin, retire) or the endpoint methods; the outside-member allowlist shrinks from 139 to 45. litellm/router.py no longer imports litellm.proxy: it asks the installed RouterHost (litellm/router_host.py) for vertex pass-through credentials, member auto-router authorization and auto-router prompt compression, and the proxy installs its host on import. Call sites where a model name can be missing keep the old answer instead of reaching an entry point with None.	2026-10-29 17:26:33 +00:00
Mateo Wang	6cdb9959df	feat(router): opt in to non-chat deployment filtering (#46257) ... * feat(proxy): reject chat routes on models declared non-chat via model_info.supported_endpoints A model whose every deployment sets a non-chat model_info.mode and a supported_endpoints list with no chat endpoint now answers /v1/chat/completions, /v1/messages and /v1/responses with a 400 before any provider call. Models without both fields keep today's behavior. * fix(proxy): let fallbacks answer and cover more chat routes in the non-chat model check The check now steps aside when a request, key, team or router fallback would answer, skips such models in the rate-limit fallback loop, ignores mode and supported_endpoints copied from the cost map by old Admin UI saves, treats a blank mode as unset, and also covers /v1/completions, the Responses WebSocket, Gemini generateContent and /v1beta/interactions * fix(proxy): only step aside for fallbacks that can answer chat, gate the Responses WebSocket after alias resolution, and compare declarations to the loaded catalog * fix(proxy): gate the Responses WebSocket with the router after aliases resolve, and ignore client-set internal fallback flags * fix(proxy): walk fallback groups breadth first and release reservations on a rejected WebSocket Each fallback group is looked up once however often the request repeats it, and a WebSocket closed by the endpoint gate runs the failure hook so rate-limit reservations are released * fix(proxy): walk fallback layers without recursion and gate the compact Responses route The fallback walk now folds one layer at a time, so the recursion check passes, and /v1/responses/compact on a model declared non-chat gets the same 400 as the other chat-shaped routes * fix(proxy): keep chat on rows whose cost map entry serves chat A row that carries the cost map key of an entry that serves chat counts as declaring nothing, so a catalog edit that adds a chat path never turns old Admin UI copies into 400s * fix(proxy): check chat sent to a deployment id or backend model name * fix(proxy): check a model the read-through loads from the database * fix(proxy): step aside for fallbacks the check cannot follow and wildcards that serve a backend name A fallback target with no deployments of its own, a client routing stamp, and a wildcard that does not match the requested backend name no longer turn a working request into a 400 * fix(proxy): ignore a client model_group that is not a string when building rate-limit headers A client metadata.model_group sent as an int or a list crashed the error handler with a 500 once the embedding-only check rejected the call before routing. The header helper now treats a non-string stamp as no model group. Drops the pattern clause in the backend-name lookup, since the group lookup already returns wildcard matches. Adds the third audit round's routing-stamp cells. * fix(proxy): keep a non-string client model_group out of spend rows * fix(proxy): validate standard fallbacks before the typed fallback-group lookup * feat(router): opt in to non-chat deployment filtering * fix(router): skip unresolved local rate-limit fallbacks * fix(router): ignore caller-provided endpoint selector values * fix(router): normalize declared chat endpoint prefixes * test(router): cover preflight error-value behavior * test(router): derive replay metadata defaults from model info * fix(router): preserve fallback aliases and check synchronous text calls * test(routing): rewrite the WebSocket model with a pre-call callback, since main no longer runs guardrails at socket admission * test(routing): keep the endpoint gate's owned proxies on the worker's database * fix(router): keep the endpoint gate on model_info patches that omit it and on same-group stream retries A partial PATCH of model_info reset a stored enforce_supported_endpoints true back to false, because the field defaulted to False and the update merges model_dump(exclude_none=True). The field now defaults to None so an omitted flag keeps the stored value The same-group retry of a stream dropped before content called the provider without the endpoint selection context, so it could land on an enforced embedding-only deployment of the same group. It now runs inside endpoint_selection like every other router attempt * fix(router): probe rate-limit fallbacks through get_routable_deployments so the router surface does not grow Main's router fence lets code outside the router use only listed Router members and router_utils modules. The proxy's fallback probe now calls the listed get_routable_deployments with the request's call type, and the gate's 400 moves to litellm.exceptions as EndpointNotSupportedByModelError. It stays separate from main's UnsupportedEndpointError, which the router retries on the group's other deployments, while the gate refuses the whole group * test(responses): cover the WebSocket error frame for a model that refuses the endpoint * fix(proxy): let the rate-limit fallback probe try a fallback this worker has not loaded yet The probe now skips a fallback only when the endpoint check removed every deployment it has, so a model added on another worker still reaches route_request and its database lookup * fix(proxy): keep key router settings off the Responses WebSocket, read unrooted or single-string endpoint lists, and add status to the refusal frame * fix(router): warn when enforce_supported_endpoints has no usable list and match chat endpoints regardless of case, trailing slash, or Gemini path * test(router): type the load-warning test's caplog and router * test(routing): match the endpoint gate's integration tests to base fallback behavior on unloaded fallbacks and the Responses WebSocket --------- Co-authored-by: Mateo <mateo@Mateos-MBP-2.localdomain>	2026-10-29 08:33:36 -07:00
paveln83	032342d2b7	fix(responses): resume background response streams with stream and starting_after (#48819) ... * fix(responses): resume background response streams with stream and starting_after Retrieving a response with stream=true and starting_after=N now returns the replayed event stream from OpenAI and Azure instead of one JSON object, on the SDK (get_responses / aget_responses) and on the proxy's GET /v1/responses/{id} Providers that cannot replay a stored stream drop the params under drop_params and raise UnsupportedParamsError without it. Malformed values answer 400 with the param named. A replay that reaches its terminal event settles the background response's bill once and passes the provider's usage through without estimating one. An empty replay (cursor at the end) ends normally * fix(responses): encrypt replayed stream ids and resume Azure AI Foundry streams A replayed background stream now gets the same response id encryption as a created one, so the key that made the response can reuse the id. Azure AI Foundry deployments resume streams like Azure ones, and the chat-bridged Responses iterator closes its wrapped stream instead of raising. * fix(responses): drop malformed resume params under drop_params and keep JSON answers to a streamed retrieve A bad stream flag drops to a plain retrieve and a bad cursor to a full replay when drop_params is on, and a cursor without stream is ignored even when malformed. A streamed retrieve that the upstream answers with JSON returns the response object instead of an empty stream. get_responses forwards only the include keys from extra_query. * fix(responses): give proxy stream resumes the deployment stream_timeout and document retrieve query params * fix(responses): keep deployment stream off retrieves and reject oversized cursors * fix(responses): keep router default stream off plain retrieves * test(responses): black box coverage for resume refusals, drop_params, stream timeout and billing * test(responses): replay a real stream in the billing test and type the fake wrapper chunks Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-10-29 02:31:22 -07:00
Mateo Wang	65d912fc2e	feat(responses): support Codex client tool_search on Anthropic and other bridged providers (#48972) ... * feat(responses): support Codex client tool_search on Anthropic and other bridged providers * test(router): cover get_search_tool_support provider defaults and model_info overrides * fix(responses): keep tools loaded by earlier Codex tool searches across previous_response_id * perf(responses): read stored turns once on client tool search continuations * fix(router): avoid OAuth refreshes when listing search capabilities * test(cli): expect disabled search on fallback Codex models	2026-10-29 00:29:40 -07:00
tavren27	cbf4e40d38	fix(proxy)!: stop labelling management and health rejections as auth failures (#48739) ... * test(proxy): cover management and health routes labelling rejections as auth failures * fix(proxy): name management and health route errors by their status instead of auth_error * fix(proxy): keep auth_error for 401 and 403 and answer database outages with 503 in route catch-alls /key/list and /key/aliases now share the catch-all helper too, so they stop adding the Authentication Error prefix and typing a 400 as internal_server_error * fix(proxy): answer 503 when the database drops mid-query and prisma fails to name the error A real outage surfaces from prisma's query engine as a builtin AttributeError, which the prisma-module check skipped, so the route catch-alls still answered 500 or 400 * fix(proxy): report wrapped and stalled database outages as 503 and label upstream failures by their own status * fix(proxy): answer the upstream status on the catch-all, keep auth_error on tag denials, report model insert outages as 503 * fix(proxy): answer 503 for database failovers and label provider auth errors like chat A database that refuses connections while starting up, shutting down, in recovery, or read-only after a failover now answers 503 no_db_connection instead of 400. A provider's 401 or 403 keeps the type /chat/completions gives it, while a 401 or 403 the proxy raises keeps auth_error. An exception with no message is named by its type. * fix(proxy): scope failover 503s to management route errors and keep a re-raised error's cause * test(proxy): expect the OpenAI error shape from management route refusals Team-not-found 404s now answer invalid_request_error, and every handler-raised refusal answers param null instead of the string None. --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-10-28 21:49:03 -07:00
tessellark	5bd47442fb	fix(cli): give lite codex aliases of a Codex-known model that model's catalog entry (#48894) ... * fix(cli): give lite codex aliases of a Codex-known model that model's catalog entry lite codex built its Codex model catalog from /v1/models ids only, so a proxy alias of a model Codex knows (api-gpt-6-luna for openai/gpt-6-luna) got the fallback entry. That entry has no supports_search_tool or tool_mode, so Codex sent every MCP and app tool inline and each turn cost far more input tokens. lite now reads each listed model's upstream from the proxy's /v1/model/info (team_public_model_name when set, else model_name, first deployment wins) and passes it to the shared catalog builder, the same way the proxy's own Codex route does. If that request fails, the catalog is built as before. * fix(cli): give a lite codex alias an upstream only when all its deployments route to one OpenAI model * fix(cli): cap a lite codex alias's context window at its listed max_input_tokens and count only OpenAI-hosted deployments * fix(cli): keep Codex's tier, upgrade and reasoning defaults off lite codex aliases * fix(cli): keep Codex's own effort, verbosity and promo off lite codex aliases and skip credential-hosted deployments --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-10-28 21:23:55 -07:00
tessellark	a0c9a5993b	fix(batches): stop unbillable batches from blocking file deletes and new-batch polling (#48752) ... * test(batches): cover the batch cost poller gaps left after #47190 * fix(batches): keep unbillable batches from blocking file deletes and new-batch polling The file delete check now ignores batches past MANAGED_OBJECT_STALENESS_CUTOFF_DAYS, a provider 404 for the output object retires the batch, each poll page takes the arrival lane first, a batch expires only after the poller visits it past the cutoff, and the compare-and-swap matches batch_processed instead of updated_at * test(batches): cover the full arrival page and a failed expiry write in the batch cost poller Old batches still get polled every other cycle while new batches fill the page, old batches that fill a page never push a new batch out after boot, and a failed stale_expired write no longer stops the rest of the page from being polled * fix(batches): share the poll page between old and new batches and hold output files until billed Each lane now gets at least half of the poll page, a lane with nothing left gives its half to the other, and a lane read that resumed mid-lane wraps to the start, so neither old nor new batches can starve the other. The delete guard applies the 7 day cutoff to input files only. Output and error files stay locked until the batch is billed or retired, and stale_expired rows no longer lock anything. Only a lasting provider failure expires a batch past the cutoff, so a timeout or a 5xx no longer gives up on a batch that may still bill. Past the cutoff a lasting failure, a lasting cost-tracking failure, or an output file id that never appeared retires a finished row instead of polling it forever. * fix(batches): retry refused finished batches past 7 days, expire unfinished ones on a failed retrieve A finished batch the provider refuses after 7 days (rotated key, bucket 403, a 404 that does not name the batch) now keeps retrying so its spend is written once the config is fixed, instead of retiring unbilled. A batch the provider still reports running keeps polling. A batch that is not finished in the database becomes stale_expired once a retrieve fails after 7 days, so an unreachable provider is not polled forever. An S3 NoSuchKey for the output object now counts as gone. * fix(batches): expire an unfinished old batch only after 3 failed retrieves in a row * fix(batches): keep retrying an unfinished old batch through timeouts, 429 and 5xx * fix(batches): wait 3 polls before expiring an old batch whose output id lags * fix(batches): match a resource id in a 404 message followed by punctuation * fix(batches): keep refusal and pending-output expiry streaks apart, and void a pending-output retirement on a client write * fix(batches): give up on unsettled batches past twice the staleness cutoff Replace the per-reason refusal and pending-output streaks with one give-up age, and inject a clock into the poller and the delete check so tests pin exact age boundaries * fix(batches): keep polling finished batches past the give-up age so their spend stays recoverable * test(batches): expect a restarted poller's first page to split between old and new batches * test(batches): pin the delete-guard tests to a fixed clock and type a new journal list * test(batches): type the provider helper and the storage 404 cases --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-10-28 12:19:36 -07:00
tessellark	43ce3e22aa	fix(responses)!: answer 400 before counting a lone Responses built-in tool on Claude (#48798) ... * test(responses): audit #47363 Adds integration cells for the input token count paths #47363 changed: the provider id of a managed file and the 403 for another team's file on OpenAI, file_search and additional_tools parity plus cache-mark stripping and untranslatable tools on Claude, and the Bedrock runtime count leaving out the options it refuses. * fix(count-tokens): reject a lone tool with no chat equivalent before the Claude count call A Claude input token count whose only tool was mcp or computer_use_preview skipped the pre-call check and went to the provider, so the caller got the provider error instead of the 400 naming the tool. The check now runs whenever a tool would get a placeholder name. * fix(responses): count a lone Anthropic toolset and name the uncountable tool The lone-tool 400 now fires only for OpenAI Responses built-in tool types, so a nameless Anthropic toolset still counts. The error says LiteLLM cannot count that tool type for the model. Adds a Bedrock unit cell for the lone-tool 400 and pins the runtime wire test's tool shape per request type. * fix(responses): take the uncountable tool types from the OpenAI SDK and keep the region hint off LiteLLM's 400 The Responses built-in tool types now come from openai's ToolParam union, so a type the SDK adds is refused before the count call too. The AWS region hint no longer lands on an error LiteLLM raised before calling the provider. Tests cover two more lone types and the hint. * fix(responses): keep LiteLLM's own list of uncountable tool types and pass a non-string tool type through * fix(responses): count tool_search and namespace tools the way Claude chat inference sends them The count refused a lone tool_search with LiteLLM's 400 and sent a lone namespace raw or squashed it into an empty tool, while Claude chat inference maps tool_search to Anthropic's tool search tool and unpacks a namespace into its function tools. Both now go through the deployment's own map_openai_params, so the count sends what inference sends, and tool_search leaves the list of uncountable types. * fix(bedrock): count a Claude request with its deployment's own chat translation Bedrock invoke and Mantle inference turn a chat tool_search tool into Bedrock's tool search tool, but the Bedrock counter always translated with the Anthropic config, so the count answered 400 for a request inference serves. --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-10-28 12:12:45 -07:00
rj-cairn	ff5bf4826d	fix(count_tokens)!: audit #47441 and keep the framed estimate for Bedrock count models from the cost map (#48805) ... * fix(count_tokens): keep the framed estimate when a region-less token_count_model cannot count (audit #47441) * fix(count_tokens): let framing stand in for a shipped token_count_model, keep a configured one's error The framed local estimate now replaces a 400/404 from any count model the cost map ships, Mantle-pinned rows included, and an admin-set token_count_model keeps its error as the docs say. * fix(count_tokens): warn with the region when framing stands in for a pinned count model Test a Mantle row #47441 pinned in its own region, and keep the region hint check on an admin-set count model. * fix(count_tokens): name the deployment in the framing stand-in warning * fix(count_tokens): let framing stand in when a cost map count model's region denies the account A count model the cost map pins to a region can answer 401 or 403 when the account has not enabled that region or lacks Mantle count access there. With framing set, the estimate now stands in, as it does for 400 and 404. An admin set count model and a count model with no pinned region keep the error. The warning now names the model group, the count model and its region, and says the proxy falls back to the estimate. * test(count_tokens): cover framed 401 and 403 from a shipped count model region on the wire --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-10-28 19:11:35 +00:00