* feat(router): add Switchyard Stage Router strategy
* feat(router): decision-model classifier and safer fallbacks for the Switchyard stage router
Moves to nemo-switchyard 0.4.0a1, which adds decision-model classifiers and makes
ModelCall.respond and fail async. The classifier is now a tagged union of an LLM
judge and a decision model, and decision calls go through the router's adecisions
so any decisions provider works. Algorithm misbehavior and library errors route to
the capable model instead of failing the request, and the session id plus client
session headers are always passed so capable holds apply without a classifier
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix(router): keep the Switchyard router inside the type-check gate
The gate's environment did not install the optional switchyard extra, so every
libsy symbol counted as Unknown. nemo-switchyard joins proxy-dev like the other
optional runtime packages, and imports go through switchyard_rust.libsy, which
ships py.typed. Step handling moves into helpers so no Final is rebound in the
loop, and the dashboard API types are regenerated for the new routing fields
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix(router): route Switchyard on every endpoint and reject unsafe router saves
The router now passes each request's call type to routing strategies, and
Switchyard converts the request to chat messages with the existing converter for
that call type before translating it. /v1/messages turns with tool or thinking
blocks used to fall back to the capable model, and now route on their signals.
The Anthropic converter can keep tool_result.is_error on an analysis copy, so
Switchyard sees tool failures there, while requests sent to models never carry it
Saves of a strategy router that lists itself now fail with a 400 instead of a
500 with a dead row, the capable and efficient models must differ, judge and
decision calls are not retried before failing open, the Switchyard routing
fields are classified for redaction, and the edit dialog returns the saved
router instead of the stale one
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix(router): catch self-routing on patch and rename, and trim Switchyard tests
The self-routing check now reads the stored params through the decrypting
helper and runs when a patch carries no litellm_params, so a PATCH that points
a router at itself or renames it onto one of its targets returns 400 instead of
a 500 with a dead row. The Switchyard tests share one transcript builder and
router fixture, and reformatting noise in unrelated test files is reverted
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* refactor(ui): make Switchyard a strategy in the existing auto router form
Switchyard is now a card in the existing strategy picker. Picking it swaps the
tier and classifier fields for the Switchyard fields, and the shared name, team
and save button create it through one submit branch, the same flow as the other
strategies. The separate add form, the extracted picker and the hidden-form
swap are gone, and the edit dialog keeps its own Switchyard form
The judge's session_affinity and message_hash_fallback settings are removed,
since Switchyard rejects both for the stage router
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix(ui): accept partial tool semantics and show server errors on model edit
The Switchyard tool semantics field rejected mappings that list only some
categories, although the server accepts them. Model PATCH errors were
reduced to a generic message, so it now goes through the shared API client
like model create and surfaces the server text
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* feat(ui): show the routing strategy picker up front in the auto router form
Every strategy was hidden behind an Advanced configuration link, so the
quick setup read as the only option and never said it creates a heuristic
router. The picker now always shows with Heuristics as the labelled
default, the quick setup sits under it, and picking another strategy opens
that strategy prefilled with the chosen models
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix(ui): show the server's reason when a model settings or API key update fails
Both screens replaced the rejection with a fixed message, so a 400 such as
a router routing to itself read only as a generic failure
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* feat(router): route Switchyard on the Rust router core
main replaced the Python router core with the Rust one, which left
Switchyard routers registered nowhere. Register them with the other
pre-routing strategies, hand each strategy the endpoint the request came
through, and evict Switchyard routers on reload and delete like the rest
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix(router): scope Switchyard sessions per API key and fix review gaps in the Switchyard form
Switchyard session state is keyed by a hash of the caller's key hash and session id, and client
session headers no longer reach Switchyard unscoped. The add form carries the quick setup models
into Switchyard, sends the team and access groups, clears a stale tool semantics error, and the
schema reads API-saved configs with nulls and structured decision instructions
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* test(router): make the Switchyard test folder a package
* fix(router): read the Switchyard session key hash only from the proxy's own metadata
* fix(proxy): keep nested nulls as null when masking deployment params
* fix(router): send Switchyard judge calls with the caller's own metadata bucket so tag routing applies
* fix(router): put Switchyard judge metadata in the bucket the judge endpoint reads
---------
Co-authored-by: Joshua Valluru <326636767+joshua-berri@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* build(deps): lock filelock 4.2.4 on Python 3.11+ and 3.20.3 on 3.10 to clear osv-scan
Co-authored-by: Gilberto Plaza <keko950@hotmail.com>
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* build(deps): bump @azure/identity to 4.14.1 and @azure/msal-node to 7.1.2 to clear osv-scan
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* build(deps): bump oauthlib to 4.0.1 to clear osv-scan
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: Gilberto Plaza <keko950@hotmail.com>
Co-authored-by: mateo <277851410+mateo-berri@users.noreply.github.com>
* feat(proxy): give Codex accurate model entries for every model in /v1/models
* fix(router): read a deployment's upstream model without a not-required key access
* fix(proxy): keep Codex tiers for unpriced deployments and match stock entries by declared provider
A deployment with no token price no longer votes a Codex service tier off,
an undescribed model keeps Codex's default reasoning summary parameter,
a Bedrock deployment naming its provider in custom_llm_provider gets Codex's
stock entry, and Bedrock provider namespaces follow the AWS model id format.
* fix(proxy): import cost tracking and provider helpers lazily in the Codex catalog
lite codex imports the catalog before the SDK is initialized, so importing
cost tracking, the Bedrock helpers or provider resolution at module level hit
a circular import and broke every lite command.
* fix(proxy): drop Codex's fast speed tier wherever its priority tier is dropped
Codex shows /fast for a leftover fast speed tier even when service_tiers
no longer offers priority, so an alias, an empty configured tier list or
a configured list without priority showed a /fast toggle that sends nothing.
One helper now sets service tiers, speed tiers and the default together.
* fix(proxy): give an older Codex a translated entry for models past its version
Codex's own entries name the oldest release that can read them in minimal_client_version. A release older than that can fail to decode the entry, such as a reasoning effort it has no name for, which fails the whole catalog, so it now gets the translated entry, whose levels are trimmed to that release
* feat(packaging): ship litellm-proxy wheels with the proxy extra
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix(packaging): keep litellm-proxy out of the proxy extra until PyPI has it
The current release pipeline publishes litellm with whatever pyproject.toml lists, so pinning an unpublished package would break pip install litellm[proxy]. Images and CI still install the locally built gateway wheel
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* fix(nvidia_riva): send the api key once per call by talking to Riva over grpc directly
Drops nvidia-riva-client from the stt-nvidia-riva extra. Depending on its version it sent
the bearer token zero times on plaintext or twice on TLS, so token-checking Riva deployments
rejected every request (#50059). The provider now declares the few Riva messages it uses,
opens its own grpc channel and puts authorization and function-id on each call exactly once
* test(nvidia_riva): run the provider against a real grpc server built from nvidia's own definitions
* fix(nvidia_riva): validate endpointing settings with pydantic before building the riva message
* test(nvidia_riva): check the key is sent once over tls, and drop a comment that restated the code
Widen the huggingface-hub cap from <2.0 to <3.0. The locked version stays
at 1.32.0; only the matching specifier line in uv.lock changes.
Claude-Session: https://claude.ai/code/session_016isSPojcgVpmxSRshXQEPw
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* build(deps): bump pyasn1 from 0.6.4 to 0.6.5
Bumps [pyasn1](https://github.com/pyasn1/pyasn1) from 0.6.4 to 0.6.5.
- [Release notes](https://github.com/pyasn1/pyasn1/releases)
- [Changelog](https://github.com/pyasn1/pyasn1/blob/main/CHANGES.rst)
- [Commits](https://github.com/pyasn1/pyasn1/compare/v0.6.4...v0.6.5)
---
updated-dependencies:
- dependency-name: pyasn1
dependency-version: 0.6.5
dependency-type: indirect
...
Signed-off-by: dependabot[bot] <support@github.com>
* build(deps): bump pyasn1, pypdf and minimatch to clear osv-scan
pyasn1 0.6.4 -> 0.6.5 and pypdf 6.21.2 -> 6.21.4 in uv.lock, with a dated
exclude-newer-package cutoff that admits pypdf 6.21.4 before it clears the
3-day window. minimatch has no fixed 10.x release, so the dashboard's override
and a new vscode-extension override move it to 11.0.2.
---------
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* test(token_counter): price PDF pages for OpenAI and Azure models the way they bill
* fix(token_counter): price PDF pages for OpenAI and Azure models the way they bill
* fix(token_counter): count azure, data-zone and dated GPT names with their family's encoding
* fix(token_counter): price only drawn PDF images and keep other gateways' GPT pricing
Detect raster images from the page content stream, including inline
images on pages with empty resources, and walk each form once. Strip
only openai/, azure/ and Azure data-zone prefixes, bound the model
family scan, read Azure-only chat models from the cost map, and keep
cl100k_base for listed non-chat OpenAI models.
* fix(token_counter): price GPT names behind any prefix like the bare name and walk shared forms per resources
Prices a PDF by the model name's last path segment, the same name the proxy counts with,
keeps listed non-chat models on Claude pricing, walks each form once per inherited resources
dictionary so a shared child form drawing an image is not missed, and resolves PDF pricing
only when a message carries a PDF
* fix(token_counter): keep the rendered-page count for image pages, parse each form once, and match chat models only
A PDF page that draws a raster image keeps the rendered-page estimate, since newer GPT models bill such a page near
it. The image walk parses each form once and stops at a fixed number of drawn XObjects, so a form shared under many
resources dictionaries no longer slows the count. The family prefix match skips non-chat entries, and ft: fine-tunes
count like their base model
* fix(token_counter): count Azure-only chat models with o200k like other OpenAI chat models
* fix(token_counter): count reservation and project ITPM with the serving deployment's model
Budget reservation and project ITPM counted with the requested group name, so a
GPT-named group served by Claude reserved a PDF at OpenAI's price. They now count
with each deployment's model minus its provider prefix, the name count_tokens uses,
and take the largest. A custom OpenAI tokenizer with no model counts with cl100k
again instead of raising, and listed Azure gpt- chat entries price as OpenAI.
* perf(token_counter): find drawn PDF images with a byte scan instead of parsing every operator
pypdf parsed every operator of a page's content stream to find the images it draws, so a
compressed page of a million operators took about 30 seconds to price for an OpenAI model.
A linear scan of the decoded stream for name Do and inline images finds the same draws.
* fix(token-counter): count each team deployment by its PDF pricing and honor base_model
Budget reservation and the TPM limiter now look up the deployments that serve the
caller's team, by base_model when set. A deployment that reads PDF pages and prices
them unlike the group name counts with its own model, any other with the group name.
A comment between an XObject name and Do now reads as whitespace, and the timing
tests moved to CodSpeed benchmarks.
* fix(benchmarks): install pypdf in the benchmark environment
The PDF token-count benchmarks parse pages with pypdf, which only the proxy-runtime extra installed, so CodSpeed took the no-parser path.
* fix(proxy): count PDF pages for deployments served under multi-segment model names
* fix(proxy): count a team's wildcard route the way routing serves it and reserve OpenAI chat's per-PDF charge
* fix(proxy): count routing-group and alias members with the caller's team, as routing serves them
* build: allow filelock 4.x
filelock 3.x is the only line the current cap permits, and three CVEs
(CVE-2026-112940, CVE-2026-112913, CVE-2026-112907) are fixed only in
4.2.1+. The cap means every downstream that depends on litellm is held
on a vulnerable filelock with no way out, since pip has no override.
litellm uses FileLock(path, timeout=..., mode=...) and Timeout, in
litellm/proxy/client/cli/commands/configure_profiles.py and
claude_settings.py. Both keyword arguments still exist in 4.2.4 and
Timeout is still exported, so the two call sites are unaffected. The
CVEs are in the SQLite-backed ReadWriteLock / AsyncReadWriteLock, which
litellm does not use.
<5 rather than a floor bump because litellm supports Python 3.10 and
filelock 4.x requires >=3.11: widening the cap lets 3.11+ resolve 4.x
while 3.10 keeps a 3.x filelock automatically, through filelock's own
requires_python.
* build: sync core filelock bound and dependency lock
---------
Co-authored-by: Speculator55005 <50082482+fas89@users.noreply.github.com>
* chore(caching)!: remove the disk cache backend
Drop the disk cache type and its diskcache dependency, along with the
caching install extra that existed only to pull it in. diskcache has not
had a release since August 2023
Cache now raises a ValueError listing the supported types when it gets a
type it does not know. Before, an unknown type built a Cache with no
backend, which only failed on first use
BREAKING CHANGE: cache_params type "disk" and the litellm[caching] extra
are gone. A proxy configured with type disk now refuses to start. Use
local or redis instead
* test(proxy): expect the unsupported cache type error from _init_cache
The test pinned the old fallthrough, where an unknown type left the cache
without a backend and the proxy hit an AttributeError reading it
* chore(deps): drop the diskcache scan ignore main extended
diskcache is no longer in the lock, so the ignore matches nothing
* fix(caching): ignore a leftover disk_cache_dir instead of crashing redis and s3
Dropping the disk_cache_dir parameter sent the key into **kwargs, which
the redis and s3 branches hand to their clients, so a config that moved
from the disk cache to redis or s3 and kept the key failed with a
TypeError. Accept the key again, ignore it, and log a warning
The unsupported type test now checks the whole supported list, and two
proxy docstrings no longer name disk as a cache backend
* fix(caching): warn about a leftover disk_cache_dir only once the cache is built
The warning ran before the type check, so a config with type disk logged
that the key was ignored and then failed on the type anyway. It now logs
only after a supported backend is built. The test also checks that a
cache without the key logs nothing
* fix(proxy): default the uvicorn worker healthcheck timeout to 60 s so multi-worker boots survive a slow import
* fix(proxy): warn on uvicorn below 0.37 when multi-worker boots cannot get the longer healthcheck
uvicorn 0.33 to 0.36 has no timeout_worker_healthcheck, so the 60 s default never reaches it and a slow worker start still crash-loops with no hint. Print an upgrade warning when the proxy runs more than one uvicorn worker on such a version, keep granian out of the uvicorn-only path, and type the new tests.
* fix(proxy): name --run_granian in the healthcheck help text and add per-worker grace guidance to the helm chart
* fix(proxy): keep the litellm command light so spawned workers answer uvicorn's first health ping
Each worker uvicorn spawns re-runs the parent's main script before it starts
answering health pings. The litellm console script imported the whole SDK at
the top, so on a busy or CPU-limited host a new worker missed uvicorn's 5 s
ping and was killed in a loop. The script now runs litellm_cli.main:proxy_server,
which imports the proxy only when called, and gateway.launch imports LiteLLM
inside main the same way.
This replaces the earlier 60 s healthcheck default, so uvicorn's 5 s
healthcheck and SIGTERM timing stay as they are.
* fix(proxy): load the worker app before its log config and without GC pauses
* fix(gateway): ship litellm_cli in the gateway image
gateway.launch imports without_gc_pauses from litellm_cli, which the
gateway image never copied, so python -m gateway.launch failed at import.
Also build the gateway image when the launcher or litellm_cli changes.
* fix(proxy): apply the worker log config before the proxy app import, as earlier releases did
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* build(deps): suppress filelock Windows advisories and drop stale certifi entry to clear osv-scan
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* build(deps): suppress dev-only msal-node advisory in vscode-extension until azure identity allows 7.1
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* build(deps): bump ddtrace and msgpack to clear osv-scan
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* build(deps): bump pylint to 4.0.9 to clear osv-scan
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* build(deps): lock dnspython 2.9.0 on Python 3.11+ and defer 3.10 to clear osv-scan
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* build(deps): bump qs in vscode-extension to clear osv-scan
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* build(deps): extend the pillow 12.3.0 override for Python 3.10 to 2026-11-27
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* build(deps): extend the diskcache ignore, drop the withdrawn braces entry, and name the filelock cap in its reasons
* build(deps): bump langgraph and add dated ignores for the 2026-10-28 filelock advisories
* build(deps): move tornado to 6.6 on Python 3.11+, name huggingface_hub's filelock use, and scope a grype rule to the filelock High
* build(deps): say which packages import tornado in its ignore reasons
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* refactor(rust-bridge): require the native extension
_native ships in every wheel and the sdist will not install without it, yet the bridge loaded it optionally and carried None paths that could fail open for fork safety, exception mapping and OTLP errors. Delete loader.py and import _native directly. Tests inject a fake module through NativeBinding(source=...) or with_source() instead of patching the loader
The install smoke checks now verify the compiled module, and tests fail instead of skipping when _native is missing. The interactions spec canary installs the project so it has the extension
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix(ci): build Rust bridge in Windows smoke tests
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* build(rust): upgrade reqwest to 0.13 and move MCP out of litellm-http
The workspace moves to reqwest 0.13, so litellm-http no longer carries a
second reqwest just for rmcp. The mcp feature, its pool and its proxy
conversion go away, and the gateway hands rmcp a pooled NoRedirect client
that already carries the TLS, proxy and timeout settings
The Client test helpers collapse into for_test, a no-redirect client
without TLS or proxy settings, and wrap_for_test for the few tests that
need a hand-built reqwest client
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* build(deps): refresh compatible dependencies flagged by OSV
* refactor(rust): remove redundant client construction comment
* test(rust): cover configured MCP proxy and CA bundle
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(deps): upgrade vulnerable Python dependencies
* fix(deps): lock patched Python dependency versions
* fix(otel): preserve JSON export with the updated SDK
* test(otel): cover JSON export with the updated SDK
* test(otel): update SDK log records and worker discovery
* fix(ui): align Lens peer dependency with patched Next.js
* fix(otel): hand the AgentOps exporter a session it owns for the JWT header
OpenTelemetry 1.45 moved the OTLP HTTP exporter's requests.Session behind a
transport, so writing the minted JWT to the exporter's private _session raised,
was swallowed, and every AgentOps batch went out unauthenticated. The exporter
now creates the session, passes it through the public session= argument and
sets the Authorization header on it
The tests now drive a real export, serving the token endpoint with respx and
capturing the OTLP POST with requests_mock, instead of reading SDK internals
* fix(otel): skip FastAPI instrumentation once the app has started
opentelemetry-instrumentation-fastapi 0.66b1 wraps build_middleware_stack instead
of calling add_middleware, so instrumenting an app whose middleware stack is
already built no longer raises. The call silently did nothing while we still
registered the global response propagator. Check the built stack up front and
skip both, which is what the existing test expects
* test(otel): assert exported requests and span flags instead of SDK internals
OpenTelemetry 1.45 dropped the exporter's private _headers and sets the W3C
random-trace-id flag on new traces, so traceresponse ends in -03. The Langtrace
and Langfuse tests now export through requests_mock and check the URL and headers
the collector receives, and the traceresponse tests expect the server span's own
trace flags
* fix(deps): bump dashboard zod to 4.6.6 and make the vendored Lens use it
zod 4.6.5 is flagged by the dependency scan and blocks osv-scan on every PR. A
root bump alone leaves the vendored Lens tarball, which pins zod 4.6.5 exactly,
on its own nested copy, so a top-level zod override points every consumer at the
root version and the lock carries a single zod 4.6.6
* test(ui): stub the tag summary fetch the client panel retry refreshes
Retrying the model breakdown refetches /tag/summary, which this integration test
left on the real network. When that request failed before the assertion, the
panel switched to its summary error and unmounted the model row, so the test
passed or failed by timing
* fix(deps): drop the duplicated Lens override from the dashboard package.json
* style(ui): format the client panel fetch stub
---------
Co-authored-by: Yuneng Jiang <yuneng@berri.ai>
* fix(security): bump grpcio and boto3 so their bundled CA files drop the compromised eMudhra roots
grpc ignores SSL_CERT_FILE and verifies every channel against its own bundled roots.pem. In 1.78.0 that file still trusts emSign ECC Root CA - G3 and AAA Certificate Services, so Vertex AI, Google KMS, Chirp STT and OTLP gRPC exporters (Arize) accept all 14 advisory chains. 1.84.1 is the first release whose roots.pem drops both roots.
botocore uses certifi when it is installed and falls back to its bundled cacert.pem otherwise. Through 1.43.116 that file still trusts AAA Certificate Services. Requiring boto3 1.43.117 pulls in botocore 1.43.117, the first release whose cacert.pem rejects all 14 advisory chains, so the fallback stays safe if certifi is ever absent
* fix(docker): provide OpenSSL 3 when generating Prisma engines
Update certifi, Rust trust roots, and the Docker CA bundle to remove emSign ECC Root CA - G3
Require the fixed certifi release in litellm-core and retain libssl3 in Docker runtimes for Prisma compatibility
Co-authored-by: shin-berri <shin-laptop@berri.ai>
Pillow 13.0.0 requires Python >= 3.11, so the ci group pin is split by
Python version and the 3.10 resolution stays on 12.3.0 with a 7-day
osv-scanner override. Dashboard vite moves from 7.3.5 to 7.3.7. The
certifi fix (2026.10.19) is still inside the 3-day uv exclude-newer
window, so its advisory is ignored until 2026-10-23
* fix(mcp): upgrade to MCP SDK 2.3.0 and raise the per-event SSE size cap
MCP SDK 2.x reads Streamable HTTP responses through httpx2's SSE parser,
which rejects any single event over 1 MiB, so a large tools/call result
failed with 'SSE stream ended without a response'. mcp 2.3.0 exposes
max_sse_event_size, and MCPClient now passes a 64 MiB default,
configurable with LITELLM_MCP_MAX_SSE_EVENT_SIZE_BYTES
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(mcp): fall back for invalid SSE event limits
* test(mcp): verify configured SSE limits and recovery
* chore(mcp): remove redundant dependency comment
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: Joshua Valluru <326636767+joshua-berri@users.noreply.github.com>
* test(proxy): cover gzip and zstd request bodies on /v1/systemone
* fix(proxy): decode zstd and gzip request bodies before auth reads them
* fix(proxy): bound request body decompression and keep it inside CORS
Limit the decoded size by an expansion ratio and the operator's max_request_size_mb, pass bodies without the coding's magic number through, answer 415 when zstd is unavailable, and register the middleware innermost so errors carry CORS and security headers
* fix(proxy): budget decoded request bodies per worker and forward plugin bodies as sent
Each worker holds at most MAX_DECOMPRESSED_REQUEST_BODIES_IN_FLIGHT_MB of
decoded bodies and answers 503 past it. /plugin-proxy/ bodies reach the
plugin as sent, a zstd skippable frame counts as zstd, the decoded-size 413
names the decoded limit, and an integer max_request_size_mb stays an integer
in the 413 text.
* fix(proxy): read the magic number before size limits and reserve declared decoded sizes
* test(proxy): pin gzip mtime so parametrized test ids match across xdist workers
* fix(proxy): parse max_request_size_mb the same way in the auth size check
* fix(proxy): pass Claude Code OTLP export bodies through undecoded and skip the size parse without premium
* test(proxy): cover client disconnects, chunked bodies past the cap, and body-less request messages in the decompression middleware
* test(proxy): name the decompression test cases so CI logs stay small
anyio below 4.14.0 can leave a cancelled waiter queued on an asyncio Lock or
Semaphore, so the next task to acquire waits forever on a lock nobody holds
(agronholm/anyio#1145, fixed in 4.14.0). httpcore takes these locks for every
httpx request, including the prisma query-engine client
litellm imports anyio directly but never declared it, so pip and uv kept an
already installed anyio 4.13.0 when installing litellm and accepted an
explicit anyio==4.13.0 next to it. uv.lock already resolves 4.14.2, so Docker
and CI installs do not change. Both the litellm and litellm-core manifests get
the floor, and the lock only records the new direct dependency
Co-authored-by: yucheng-berri <293926326+yucheng-berri@users.noreply.github.com>
* perf(cli): start lite without initializing the litellm SDK
lite and litellm-proxy now enter through a small litellm_cli package that imports the CLI against an uninitialized litellm package and runs the SDK __init__ in place only when something first uses it, so commands start in a fraction of a second instead of loading about 3,000 modules first.
* perf(cli): import only the lite subcommand being run
Each lite subcommand now loads its module when Click looks it up, so
lite logout no longer imports the autoroute wizard, agents, or the other
command groups. importlib.import_module and attribute reads on the SDK
also initialize it on first use, and an already imported litellm is
kept as is.
* fix(cli): initialize the SDK on a star import and test command bodies in thin mode
A from litellm import * after lite loaded the CLI used to see only the names the thin package had, so it now runs the full SDK init first. The thin-vs-full parity test now also runs logout and whoami, not just --help
* fix(cli): keep the import hooks until the lazy SDK init finishes
A thread that imported SDK code after the init restored the interpreter
imports could hold that module's import lock while waiting for the init,
and the init then waited on the same lock, so both threads hung. Other
threads now wait for the init before they load any new SDK code
* fix(cli): retry a failed lazy SDK init on the next use instead of leaving it half-run
* test(cli): bind the patched credentials client and autoroute dir in an order-independent way
OSV started flagging claude-agent-sdk 0.1.44 today, which turns OSV Scan red on every PR. 0.2.165 is the newest release outside the 3-day uv cooldown. The package sits only in the ci dependency group and is only imported by the cookbook
* feat(ui): load the Home page What's new launches from /public/whats_new at runtime
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(ui): show the three newest What's new launches
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* build: ship whats_new.json in the wheel
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: kerry <kerry@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix: support OpenAI SDK 3.x while retaining 2.x compatibility
* fix: preserve HTTPX compatibility across OpenAI SDK versions
* refactor: trim OpenAI SDK compatibility patch
* refactor: simplify SDK client defaults
* test: cover SDK compatibility across provider routes
* fix: use shared names for SDK transport helpers
* fix: address SDK compatibility review feedback
* test(openai): send requests through the SDK API factory clients
* fix(exceptions): keep the provider response on PaymentRequiredError under httpx 2
* ci(base-sdk): count httpx2 as base-only when a base dependency brings it in
OpenAI SDK 3 depends on httpx2, so the litellm-core install at highest resolution pulls it in through openai. The base-only check now resolves the declared base dependency closure of litellm and litellm-core and accepts extras-only modules owned by a distribution inside that closure
* test: cover OpenAI SDK 3 client paths end to end and read SDK HTTP headers case-insensitively
* test: read C02 spend rows once, wait out the worker healthcheck on respawn, and resolve module owners on Python 3.10
---------
Co-authored-by: Marcus Wood <marcuswood@openai.com>
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* ci(lint): derive gate ceilings from merge-base counts and drop the budget files
The four lint gates (ruff strict, type discipline, basedpyright, test quality)
now fail a branch only when a rule's codebase count grows past its count at the
merge-base with litellm_internal_staging plus a fixed per-rule headroom, which
is zero everywhere except the LIT010/LIT011 and reportAny/reportExplicitAny
seeds. Base counts come from a disk cache, then the CI artifact the renamed
publish-lint-base-counts workflow uploads for every staging push (all four
checkers, one artifact per checker and sha), then a scan of the base worktree.
The four *-budget.json files, make lint-budget-update, budget_ratchet_check.py,
and the unratcheted check are gone, so no PR carries a budget edit again.
* test(lint-gates): assert checker identities by behavior and type the new gate tests
* test(lint-gates): resolve base points over an injected git so unit tests stay in-process
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(lens): use async-timeout on python 3.10 for budget reservation timeouts
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* chore(deps): keep uv.lock diff to the async-timeout entry
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(lens): cover real request deadline expiry in reserved_budget
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(lens): finish Python 3.10 timeout coverage and dependency checks
* test(lens): control event-loop time for deadline regressions
* test(tracing): include priced call count in trace fixture
* fix(lens): limit timeout compatibility changes to PR scope
---------
Co-authored-by: Moe Khalil <moe@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
uv only rebuilds the editable litellm package when its cache keys change, and
the default keys are pyproject.toml, setup.py and setup.cfg. Editing or
switching to a branch with different Rust code left the old
litellm/rust_bridge/_native.abi3.so installed. Key the build on the Rust
toolchain pin, Cargo config, lockfile, workspace manifest and every file
under litellm-rust/crates, since crates embed .sql, .json and .jinja files at
compile time.
Co-authored-by: Nate Armstrong <narmstrong@Nates-MacBook-Pro.local>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* feat(proxy): serve a Codex-native model catalog with per-model service_tiers from /v1/models
GET /v1/models and /models answer Codex CLI's catalog fetch (the request
carrying its client_version query parameter) with Codex's own
{"models": [...]} shape: a model Codex knows keeps the metadata of its
bundled 0.159.3 catalog (vendored), any other model gets Codex's fallback
entry, and model_info.service_tiers becomes each entry's service tiers so
Codex offers them as slash commands that send service_tier upstream.
Without the parameter the OpenAI list shape is unchanged. The CLI's
litellm agents codex catalog shares the same builder.
* fix(proxy): offer a Codex service tier only when every deployment of the model lists it
* fix(codex-catalog): an invalid service_tiers value offers no tier for the model
* fix(codex-catalog): read service tiers off the deployments the key's team can route to
A tier is offered to Codex only when every deployment of the model name a
request from the key's team can route to lists it, so another team's
deployment of the name and a deployment an admin paused via model_info.blocked
no longer withhold or add tiers for requests that never reach them
The catalog's always-null fields are annotated NoneType so the module imports
under pydantic 2.12.0 on Python 3.14, the lowest pin the MCP resolve job
installs, which rejects a None annotation with a None default
* test(codex-catalog): drop the redundant module docstring and sort the imports
* test(integration): add the Codex catalog audit cells and the multi-worker convergence note
* test(integration): clean up every catalog test model and answer the refresh GET
* fix(proxy): keep tiered models under Codex's catalog cut and resolve alias tiers
Under Codex's 1 MiB catalog limit the entries offering a service tier are kept
ahead of those offering none, each group in model_list order, with every kept
entry at its listing position, so the model an operator configured tiers for
survives a wide key's long listing. A model_group_alias row reads its target's
deployments, so it carries the target's tiers and stock metadata under the
alias name.
* fix(proxy): pick Codex catalog metadata per team and skip entries too large for the cut
The upstream model that selects Codex's stock entry was read off the first deployment of a name
without checking the key's team, so a team whose requests route to a different deployment could be
handed another team's prompt, reasoning levels, and tiers. The upstream model and the tiers now come
from the same team-aware selection routing uses, and a caller with no team reads the deployments no
team owns
The byte cut kept a prefix of the tier-first order, so one entry larger than the whole limit emptied
the catalog. An entry too large for the bytes left is now passed over and the smaller ones after it
are still kept
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
