65 commits

Author	SHA1	Message	Date
devin-ai-integration[bot]	c45b264024	fix(proxy): persist tag spend and drain success callbacks within the shutdown budget (#48988) ... * fix: drain shutdown callbacks and persist pending tag spend * fix: catch shutdown timeouts on Python 3.10 * fix: share shutdown callback budget with HTTP draining * fix: type shutdown cancellation tasks and document its clock * test: retain callbacks after bounded worker cancellation * test(proxy): inspect the lifespan body after the SIGTERM wrapper split Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(utils): keep the success callback enqueue on its own task The synchronous enqueue spawned the logging worker loop inside the awaited call and ran callbacks that per-call loops used to drop, which hung the SDK callback tests on the autouse worker flush. The shutdown worker flush covers the queued callback on its own, so the one-tick task is restored Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(logging_worker): keep the queue's unfinished count exact when a full queue rejects a callback The aggressive clear ran the rejected callback and marked it done against the queue it was never put in, so flush() could return while a real callback was still running and the shutdown drain cancelled it early. The rejected callback is now re-enqueued once the clear made room, and only runs uncounted when the queue is still full Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * refactor(logging_worker): build the aggressive-clear run set immutably Final tuple instead of an appended list, with the re-enqueue in a _requeue helper, and Final annotations on the overflow regression test Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(logging): track overflow callbacks through shutdown * chore(logging): explain shared callback ownership * test(logging): isolate deferred callback completion --------- Co-authored-by: Joshua Valluru <326636767+joshua-berri@users.noreply.github.com> Co-authored-by: joshua <joshua@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-10-29 09:32:18 -07:00
zoveki	c87a25a357	feat(proxy)!: general_settings.claude_code_marketplace_auth_required puts the Claude Code marketplace behind a LiteLLM key (#48980) ... * test(proxy): require a key for the Claude Code marketplace when configured and keep exception text out of its 500s * feat(proxy): general_settings.claude_code_marketplace_auth_required requires a LiteLLM key for the Claude Code marketplace When true, GET /claude-code/marketplace.json answers 401 without a key in Authorization, x-litellm-api-key, the litellm_key_header_name header, or ?key=, and GET /public/skill_hub returns an empty list. Both routes' 500 bodies carry a fixed message instead of the exception text. * fix(proxy): refuse ?key= and send WWW-Authenticate when the Claude Code marketplace requires a key * fix(proxy): close the marketplace key gate's outage, quoted-setting, and dashboard gaps Refuse the database-outage fallback identity with a 401, read a quoted "true" as on and an unreadable value as on, keep the skill hub's 500 detail a fixed string, and have the dashboard setup tab show a settings snippet with a key header when the marketplace requires a key * fix(proxy): keep the no-database 500, validate the marketplace key setting at boot, and show the key snippet until the probe answers * fix(claude-code): read the marketplace key setting from the UI config and answer database outages with 503 * fix(claude-code): keep the no-database hint on skill_hub and warn that the remove step uninstalls plugins --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-10-29 03:48:35 -07:00
Mateo Wang	c489aa7117	feat(proxy)!: add suppress_startup_output to turn off startup prints (#47708) ... * fix(proxy): keep every startup line JSON when json_logs is on Boot-time messages the proxy printed straight to stdout (banner, feedback box, model list, cache and callback notices, policy table, migration notices, CLI launch notes) and the first records of uvicorn- or granian-spawned workers came out as plain text while json_logs was on, breaking log pipelines that parse every line as JSON. - print_startup_message() prints unchanged in plain mode and, with json_logs on, emits one INFO record on the proxy logger (ANSI stripped) that skips the logger and handler level checks like the print it replaces; show_banner() routes to it in JSON mode only - the CLI exports JSON_LOGS=true when the config sets json_logs so spawned workers log JSON from their first record - the --enforce_prisma_migration_check deprecation notice prints after the config is read so a config-only json_logs covers it Resolves LIT-1624 * fix(proxy): keep show_banner() SDK-free, pick the JSON banner in initialize() Since #46803 the `lite` shell calls show_banner() against an uninitialized litellm package, and reading litellm.json_logs inside it initialized the whole SDK at the banner (0.14 s to 0.83 s for a bare `lite` here, the seconds that PR removed on small VMs). show_banner() is back to its main version; the proxy's initialize() now emits the JSON banner record itself and calls show_banner() only in plain mode. The banner unit tests move to the initialize() call site. * fix(proxy): keep the JSON banner when an operator --log_config names the proxy logger uvicorn applies --log_config after proxy_cli has switched JSON logging on and before initialize() switches it on again through load_config(). A logging config that names "LiteLLM Proxy" removes that logger's handlers in the window between the two, which is where initialize() emits the banner and the feedback box, so at one worker both went nowhere; two-worker children import litellm after uvicorn's logging setup and were not affected. print_startup_message() now falls back to a one-off JSON handler, on the stream the JSON handler uses, when the proxy logger has no handler of its own. Live, json_logs on plus a log_config naming the proxy logger, one worker: main prints the banner and the feedback box as plaintext, e2303a3925 prints neither, this head prints both as one JSON record each. New unit test fails on e2303a3925; new integration cell test_json_logs_with_operator_log_config_keeps_every_line_json passes at one and two workers. * fix(proxy): keep a long startup listing whole in its JSON record print_startup_message ran its record through the proxy logger's filters, so in JSON mode the required-migration SQL list and the policy list were middle-truncated at MAX_STRING_LENGTH_STDOUT_LOG (4096 by default) with the litellm_truncated marker, where the prints they replace showed them whole. That cap bounds echoed request payloads; a startup message is operator text, so the record now gets the uncapped processing of a DEBUG line: secret redaction and base64 collapse still apply, the text cap does not. Red on 41f92b745a, green here: - tests/unit/test_logging.py: a listing longer than the cap comes out whole; a credential in a startup message is still redacted - tests/integration/configuration/test_json_logs_startup.py: a 60-policy config boots with json_logs on and the policy listing record (9762 chars) carries every policy with no truncation marker (4096 chars, marker, on the unfixed tree) * fix(proxy): keep the --enforce_prisma_migration_check notice on --health and --test This PR moved the deprecation notice after the config is read, so that a json_logs set only in the config applies to it; --health and --test return before that point and stopped printing it, a visible change main did not have. The notice is now also emitted before those two returns, through the same helper, so a deployment running `litellm --health --enforce_prisma_migration_check` still learns the flag is dead, as on main. A normal server start emits it once, after the config, as in the previous commit. Red on 2d7a2d6844, green here: tests/unit/proxy/test_proxy_cli.py asserts the notice on --health and on --test. * fix(proxy): keep the JSON startup records on stdout under an operator handler on the proxy logger An operator --log_config that gives "LiteLLM Proxy" a handler of its own, a NullHandler that silences it or a plain WARNING-level handler on stderr, is in force in the window where the banner, the feedback box and the cache notice are emitted: uvicorn applies it after the CLI process switched JSON on and before initialize() re-installs the JSON handler through load_config(). 41f92b745a sent the startup records through the proxy logger's handlers and fell back to a JSON stdout handler only when there were none, so under such a config the records landed in the operator's handler, as plaintext on stderr, or vanished, where the prints they replace reached stdout whatever the logging config said. print_startup_message() now writes every startup record through the proxy's own JSON handler on stdout, filters included, and never through the proxy logger's handlers; the operator's handler gets nothing from a startup message, as it got nothing from the print. The normal JSON handler is the same class, formatter and filters, so the records a pipeline reads are unchanged when no operator config is in play. Tests: tests/unit/test_logging.py gets the two handler shapes (red on f6f5957d16: the record went to stderr as plaintext or nowhere); tests/integration/configuration/ test_json_logs_startup.py gets a silencing operator --log_config cell at 1 and 2 workers (red on f6f5957d16: banner, feedback box and cache notice missing; green here). * fix(proxy): print the --enforce_prisma_migration_check notice when the config fails to load f6f5957d16 printed the deprecation notice after the config is read, so a json_logs set only in the config makes it a JSON record. A boot whose config file is missing or does not parse then exited before that point and never printed it, where main printed it before the error (10-20 gauntlet finding). The notice is printed where main printed it, unless a config file can still switch json_logs on: then it is held until the config is read and printed right after the json_logs switch, and a click close callback prints it on the way out if the boot fails first. functools.cache keeps it to one print either way. Tests: tests/unit/proxy/test_proxy_cli.py covers a failed config read (red on 75043aed83), a plain config printing it once and a config-only json_logs making it one JSON record; tests/integration/configuration/test_json_logs_startup.py adds the JSON cell and a missing and an unparseable config cell (red on 75043aed83, green here), through refused_boot_log, which now takes extra CLI arguments. * feat(proxy): add suppress_startup_output to turn off startup prints * fix(proxy): print startup messages once and without color codes off a terminal * test(policy-engine): check the policy table keeps colors on a terminal and drops them on a pipe * fix(proxy): keep startup warnings when suppress_startup_output is on and apply it from CONFIG_FILE_PATH and before other settings Warnings that name something the operator must fix now print through print_startup_warning, which skips the suppress gate. The config key is read with the CLI's json_logs handling for --config and CONFIG_FILE_PATH, applied before the rest of litellm_settings in load_config, and can only turn the setting on. The Prometheus config panel obeys the setting, and startup prints use the plain log format's color rule. * fix(proxy): read CONFIG_FILE_PATH in the CLI only when workers would load it Workers load CONFIG_FILE_PATH only when it names an existing YAML file, so the CLI's early read of json_logs and suppress_startup_output skips any other value instead of failing a boot that used to start. * fix(proxy): read only suppress_startup_output from CONFIG_FILE_PATH in the CLI, without secrets The CLI now reads CONFIG_FILE_PATH as a plain file with its includes, skips it when a config bucket is set, and applies only suppress_startup_output, also when --config is passed. json_logs there stays with the workers as before. Fatal boot errors drop their color codes off a terminal like the startup prints. * fix(logging): emit startup warnings as WARNING records in JSON mode * fix(proxy): keep one JSON record per search tool in the startup listing * fix(proxy): keep booting with an ASCII stdout or an empty litellm_settings Startup messages now go through click.echo, which writes UTF-8 when stdout is ASCII, as the banner did before. An empty litellm_settings section reads as no settings in the CLI, as the workers already read it * test(proxy): boot the CLI tests on the real ProxyConfig config readers * test(proxy): cover the startup warnings that print with startup output suppressed * test(proxy): boot the deprecated resolver flag test on the real ProxyConfig * test(proxy): drop the startup output contract's narrating docstrings * test(proxy): cover the pgbouncer start failure message off a terminal --------- Co-authored-by: yucheng-berri <293926326+yucheng-berri@users.noreply.github.com>	2026-10-29 03:47:04 -07:00
devin-ai-integration[bot]	e85cb4052e	test: fix flaky spend-log cleanup budget, login lockout, and wedged Redis tests (#47108) ... * test: stabilize cleanup, lockout, and post-kill timing tests Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): bound the wedged Redis burst p95 by the socket timeout the test sets The burst p95 under a fixed 0.75s cap tracked how fast the box ran the 30 concurrent writes, so a loaded runner failed it at unchanged code. A handler that waits on the wedged publish takes at least the coordination Redis socket timeout, so the test now sets that timeout and asserts the p95 stays under it. * test: pin the lockout clock and bound the wedged Redis burst by the proxy's 2s wait cap The lockout tests now set the login throttle's clock instead of reading the real one, so Retry-After is pinned at 77 and a round-down fails on every run The wedged Redis burst p95 is now bounded by the 2s cap the proxy puts on its own Redis waits, so a burst wait capped under the 5s socket timeout still fails the test The fresh post-kill clients keep the harness's 2s keep-alive expiry, here and in the two tests #47080 changed * test: tell ceil from round in the lockout tests and bound the burst by the imported eviction cap The refused attempt now runs with 76.4s left on the 77s block, where ceil gives 77 but round and int give 76, so both lockout tests fail if Retry-After stops rounding up. The burst bound imports _BULK_EVICTION_WAIT_SECONDS instead of copying its 2s value, so lowering the cap moves the bound with it. * test: bound the wedged Redis burst by the shorter of the eviction cap and the socket timeout Any wait the proxy puts on the stopped Redis lasts at least the shorter of the two, so the bound now follows whichever one a later change lowers. * test: drive the tool index cleanup test with the manual clock and keep the post-kill block as on main The post-kill reset is already fixed on main by making kill() wait for the worker's last thread, so the fresh-client and keep-alive edits go back to main's version. The tool index cleanup test now advances the injected clock instead of timing the real one, and the lockout clock helper gets a type hint. * test: advance the spend log cleanup tests' manual clock through its own method * test: freeze the throttle clock in the login_utils lockout tests and drive the run budget tests with the manual clock --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-10-29 03:46:32 -07:00
Mateo Wang	65d912fc2e	feat(responses): support Codex client tool_search on Anthropic and other bridged providers (#48972) ... * feat(responses): support Codex client tool_search on Anthropic and other bridged providers * test(router): cover get_search_tool_support provider defaults and model_info overrides * fix(responses): keep tools loaded by earlier Codex tool searches across previous_response_id * perf(responses): read stored turns once on client tool search continuations * fix(router): avoid OAuth refreshes when listing search capabilities * test(cli): expect disabled search on fallback Codex models	2026-10-29 00:29:40 -07:00
Mateo Wang	f46adf80e4	fix(proxy): let typesafe/* allowlists cover /typesafe Jev calls and list Jev in the model picker (#47697) ... * fix(proxy): list TypeSafe Jev models in the admin model picker * fix(proxy): let typesafe/* allowlists cover bare /typesafe Jev ids Adds a typesafe-jev-ids routing rule to the cost map so a bare jev-* id resolves to TypeSafe, which lets typesafe/* on a key, team or project allowlist match /typesafe calls. The picker gate now only checks that a TypeSafe key is set, so an unresolvable OIDC reference cannot fail the listing, and the /v1/models route tests mock deployments with litellm_params like real ones. * fix(typesafe): match only published Jev ids and keep picker loads off the secret manager Narrows the typesafe-jev-ids rule to jev-latest, jev-preview and versioned ids so a model group such as jev-internal-router never resolves to TypeSafe, checks TYPESAFE_API_KEY in the environment instead of through the secret manager on each picker load, and moves the picker tests into the mapped test file. * test(typesafe): type the picker test fixture and tests * fix(typesafe): match only published Jev versions and cover non-version suffixes in tests * fix(typesafe): match only priced Jev ids and offer TypeSafe in the no-team key picker * fix(proxy): offer TypeSafe allowlist names only to the Admin UI allowlist pickers The dashboard sends include_model_access_groups on every /models call, so gate the TypeSafe names on wildcard routes too and skip access-group-only listings. Also say the typesafe-jev-ids rule matches case-insensitively. * fix(ui): offer only the TypeSafe names the cost-map rule routes, and keep /v1/models listing as on main A typesafe/* router deployment registers typesafe/* in the cost map, so the allowlist pickers offered a bare *, which allows every model. The pickers now list only priced names the rule sends to TypeSafe. The non-expand list also stops de-duplicating every entry and only skips TypeSafe names already listed.	2026-10-28 20:19:58 -07:00
Nate Armstrong	7650f766b0	refactor(router): fence the router surface and document the router-core boundary (#48839) ... Add rustdoc overviews for a new litellm-router-core crate and for litellm-model-catalog, a module docstring on litellm/router.py, and a shrink-only allowlist of the Router members and router_utils modules used outside the router, checked in CI by router_surface_check.py. Move the router-free helpers out of litellm/router_utils: batch_utils to litellm_core_utils, and the auto-router naming contract, tuning baseline and baseline request next to the auto-router plugin. No behaviour change.	2026-10-29 00:01:52 +00:00
mubashir1osmani	8bbce867d0	feat(proxy): accept request priority from the x-litellm-priority header (#47618) ... * feat(proxy): accept request priority from the x-litellm-priority header Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(proxy): only apply x-litellm-priority on scheduler-queued routes Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(proxy): apply x-litellm-priority only where the router schedules the request The header now maps onto priority only for chat and text completions that a router serves, so the Cursor route's responses call and proxies started without a config never forward it to the provider. /queue/chat/completions now honors the header too, with the header winning over a body priority. * test(proxy): use a real ProxyLogging in the priority header routing test * fix(proxy): ignore a malformed priority header under drop_params instead of dropping the body priority * fix(proxy): treat an empty priority header as unset and pass a valid one on as an integer --------- Co-authored-by: Mubashir Osmani <mubashir@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-10-28 15:58:52 -07:00
yujonglee	1f68cc5649	refactor(rust-bridge): require the native extension (#48852) ... * refactor(rust-bridge): require the native extension _native ships in every wheel and the sdist will not install without it, yet the bridge loaded it optionally and carried None paths that could fail open for fork safety, exception mapping and OTLP errors. Delete loader.py and import _native directly. Tests inject a fake module through NativeBinding(source=...) or with_source() instead of patching the loader The install smoke checks now verify the compiled module, and tests fail instead of skipping when _native is missing. The interactions spec canary installs the project so it has the extension Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(ci): build Rust bridge in Windows smoke tests --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-10-28 13:16:56 -07:00
shrey	0f49db5c3b	fix(proxy): store boolean settings as JSON booleans on /config/field/update (#48595)	2026-10-28 17:39:01 +00:00
tin-berri	5276e5435f	fix(ui): persist model parameter deletions after refresh (#48456)	2026-10-26 18:50:12 -07:00
Mateo Wang	464932b1d1	refactor(typing): clear 1181 basedpyright Any errors across 431 files (#47789) ... * refactor(typing): narrow Any sources across providers, proxy and guardrails (wave 1) * refactor(typing): narrow Any sources across hubs, providers and proxy (wave 2) * refactor(typing): validate RunwayML task JSON and let dotprompt loaders accept exported prompts * fix: restore pass-through behavior on paths the typing pass tightened MCP client IP extraction trusts X-Forwarded-For only when use_x_forwarded_for is True, dotprompt skips a malformed JSON prompt entry instead of raising, RunwayML task payloads pass through unvalidated, and a malformed provider body raises the same AttributeError as before at the CompactifAI, Gemini, and mock transport parse sites * fix(runwayml): keep a malformed 2xx task body on the 500 path * fix(edenai,cli): keep a non-object body in the error and fail on a malformed migration report EdenAI's embedding and image generation parse dropped a non-object body from the error they raised, so the message no longer named what the provider sent. A helper now raises with the body in the message, and both paths have a regression test. `lite encryption migrate` exited 0 with no message when the proxy's report was not an object or its residual_legacy was not a number, where base crashed. It now fails with a message naming the value. The CompactifAI and Gemini image-edit non-object body tests accept TypeError or AttributeError, which both base and this branch raise, so they guard against a ValidationError instead of pinning one class. RunwayML's async parse gets the same non-object body test as the sync one. * fix: keep base's error classes, duck-typed checks and fail-closed guardrails the typing pass changed * test: annotate the new regression tests' locals with Final * test: annotate the remaining new test locals with Final	2026-10-24 11:31:09 -07:00
shrey-berri	2590f641f5	feat(proxy): expose wildcard model listings in general settings (#47979)	2026-10-23 12:19:35 -07:00
alberto	f768fb32ef	fix(streaming)!: send the include_usage chunk with an empty choices array (#47109) ... * fix: Send the include_usage chunk with an empty choices array Signed-off-by: Alberto Perdomo <aperdomo@redhat.com> * test: Skip chunks with no choices when reading streams Signed-off-by: Alberto Perdomo <aperdomo@redhat.com> * fix: Strip usage from plain-iterator async stream chunks Signed-off-by: Alberto Perdomo <aperdomo@redhat.com> * test: Expect empty choices on the usage chunk in the gemma tracing test Signed-off-by: Alberto Perdomo <aperdomo@redhat.com> * fix(streaming): send the empty-choices usage chunk only to callers that asked for include_usage The usage chunk now has choices=[] only when the caller set stream_options.include_usage itself. When the OpenAI handler, the proxy's always_include_stream_usage setting, or the harness added include_usage on the caller's behalf, the chunk keeps its legacy single empty-delta choice, so clients that read choices[0] on every chunk keep working. The proxy and harness mark their injection with the private _litellm_stream_usage_injected flag, and the proxy scrubs a client-sent copy. Plain-iterator streams keep usage on their chunk when usage was not requested, and the LLM Shield trailing flush ignores chunks with no choices. * fix(proxy): mark the default stream usage injection so hooks keep the legacy usage chunk shape The proxy's default strip-mode injection adds include_usage for clients that did not ask and drops the usage chunk before it reaches them, but streaming iterator hooks still saw it with choices=[]. Marking that injection with _litellm_stream_usage_injected keeps the old single empty choice for them. * fix(router): treat deployment and router-default stream_options as injected, and drop the client flag on the queue route * refactor(router): move the stream usage injection check into router_utils * fix(router): read stream_options from pydantic deployment params when marking injected usage Update the mantle, openinference, and text-completion integration tests to expect the include_usage chunk with empty choices. * fix(proxy): keep the stream usage injection marker out of stored request bodies * test(streaming): cover who asked for stream usage on chat, queue, deployment and spend log paths * test: type the queue flag test, pin the logging start time, and build the deployment usage config without mutation --------- Signed-off-by: Alberto Perdomo <aperdomo@redhat.com> Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-10-21 02:26:05 +00:00
Yassin Kortam	f9d9466f2b	fix(ui_sso): seal the litellm_cp_return_to cookie so edge firewalls stop blocking the MCP OAuth sign-in (#47595) ... Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-10-20 14:17:13 -07:00
devin-ai-integration[bot]	b55024658c	fix(proxy)!: stop update_cache from resurrecting evicted auth cache entries (#46997) ... * fix(proxy): stop update_cache from resurrecting evicted auth cache entries Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(proxy): update update_cache pins for synchronous write-back Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(proxy): wrap long assert in update_cache tag test Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(proxy): cover refill race and pin TTL with injected clock Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(proxy): count global proxy spend for every request with a user, cached on this worker or not * fix(proxy): add global proxy spend with an atomic Redis increment that never re-creates a reset counter The global proxy spend bump read the total, awaited, and wrote old + cost back from a background task, so a budget reset that deleted the key in between got the pre-reset total written back, and workers overwrote each other's totals. RedisCache.async_increment_if_present runs INCRBYFLOAT only when the key exists, keeping its TTL, and this worker's copy is then set from the shared total with no await between its read and write. The bump now runs whenever a proxy max_budget is set, matching the DB writer, and the armed post-call read and the batch-read declaration it used are removed. * fix(proxy): report the global spend Redis increment to service metrics and request traces The global proxy spend increment now emits Redis service success and failure events under the spend_counters target and passes the request span through, so it shows as redis.incr spend_counters like the write it replaced. The unit tests run the real RedisCache increment against fakeredis instead of mocks, and a new test checks the span lands under the request span on success and on a Redis failure. * test(proxy): keep the spend counter Redis fake off the network * test(proxy): count increment_if_present as a shared-cache call in the key-family guard * fix(proxy): keep the newer global spend total when Redis replies arrive out of order --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-10-20 17:09:53 +00:00
joshua-berri	02199b670b	fix(proxy): share the billing logger across callback registries (#47510) ... * fix(logging): deduplicate equivalent completed success callbacks * fix(proxy): share the billing logger across callback registries * test(proxy): use fixed timestamps for billing regressions --------- Co-authored-by: Joshua Valluru <326636767+joshua-berri@users.noreply.github.com>	2026-10-20 08:21:49 -07:00
Mateo Wang	d698106ed7	fix(router)!: schedule default_priority requests once and reject a malformed priority (#47498) ... * fix(router): schedule default_priority requests once and reject a non-integer priority A router configured with default_priority failed every request that carried no priority with a 500 naming a missing 'priority' argument, and an explicit priority re-entered acompletion and hit the same error. acompletion now resolves the priority once (the request's integer, else the default, with 0 kept as a real priority) and schedules through _schedule_factory directly, while schedule_acompletion delegates to acompletion so the two paths share one queue pass. A non-integer priority answers 400 naming the param, or is dropped under drop_params, instead of leaking to the provider or crashing the scheduler. * fix(router): honor router-level drop_params and schedule prompt-management models under a priority * fix(router): drop a non-integer priority when the model's deployments set drop_params * fix(router): resolve the drop_params deployment scope for team routes and deployment ids * fix(router): read the request and router drop_params flags without a loop-assigned Final * refactor(router): type the prompt-management wrapper's return and mock the provider in its test * perf(router): consult drop_params for the priority only when it is not an integer * test(integration): assert the new priority contract for non-integer and zero priorities * docs(router): describe default_priority as the priority every request without one takes * fix(router): parse default_priority once and apply it to text completions and the queue route A digit string default is taken and any other non-integer default is ignored with a warning instead of failing every request. atext_completion and /queue/chat/completions resolve the priority the way acompletion does: a request without one takes the default, an integer queues once, and a non-integer answers 400 or takes the default under drop_params. Integration cells cover every default-priority surface on an owned two-instance rig. * fix(router): type the priority dispatch so the lint budgets hold * fix(router): read priority like the scheduler did, require a priority on the queue route, and keep litellm's drop_params order * fix(router): drop a priority named in additional_drop_params and report the queued priority on streamed queue responses * fix(router): serve deployment-id requests without waiting behind the queue and ignore a boolean default_priority A request whose model is a deployment id now counts that deployment as healthy in the scheduler, since routing by id already ignores cooldown. Before, the scheduler found no healthy deployment for an id, so the request waited behind every entry ahead of it. A boolean default_priority (true, or YAML off) is ignored at startup with a warning, like any other value that is not a whole number. The scheduler integration tests now share worker state for the defaulted pair, cool a group through /v1/responses so no stale in-memory queue is left behind, and use "high" instead of "1" for the non-integer cases. The concurrent burst test no longer asserts that the queue drains, matching the explicit-priority burst test, since the scheduler's read-modify-write queue can race across workers. * test(router): keep the queue drain check as a BUG-skipped integration test * fix(router): queue default_priority chat requests once instead of failing them * test(router): open with the default_priority regression tests alone * fix(router): honor deployment drop_params for a malformed priority on a request routed by provider model name	2026-10-20 08:00:57 +00:00
Mateo Wang	8a2782e41c	feat(router): route batch creates to mode batch deployments and expose supported_endpoints on /model_group/info (#47376) ... * feat(router): expose supported_endpoints on /model_group/info * feat(router): route batch creates to mode batch deployments in mixed model groups * fix(router): prefer batch deployments after eligibility filters and in group-name credential lookups * fix(router): keep group-name credential lookups on the first deployment and prefer batch deployments before cooldown Model-encoded ids carry only the group name, so preferring the batch deployment there moved ids created before the upgrade onto its credentials. Batch creates now pick batch-capable deployments before health checks and cooldown, routing and group info share one batch-capable check, the cost map's /v1/batch is dropped only for Azure, and /model/info applies the same per-deployment rule * test(router): map _listed_endpoints to its /model/info tests in the router coverage check * fix(router): skip only Azure chat deployments for batch creates and list batch support per caller team Narrow the batch create filter to Azure and azure_ai deployments that are not mode batch, so OpenAI groups keep their chat deployments during cooldown and under tag filters. Mark batch deployments by mode batch only, since stored endpoint lists from older Admin UI saves copy the cost map. Leave /model/info unchanged so the Admin UI never saves a computed endpoint list, and pass the caller's team into /model_group/info. * test(proxy): let the model hub router fakes take the caller's team_id * fix(router): gate batch-create routing on deployments the caller's team may use and list only /v1/batch for Azure batch deployments * fix(router): skip a malformed model_info.supported_endpoints instead of failing group info and chat * fix(router): apply the batch create filter after cooldown, managed files, and tags The batch filter ran before health checks and cooldown, so a batch create on a mixed Azure group answered with a router 429 or 401 when cooldown, tags, or a managed file's targets left no batch deployment, where main sends the create on and returns Azure's 400. The filter now runs right before order filtering, after every eligibility filter, so those cases answer as main does and creates still skip Azure chat deployments whenever a batch deployment is left * test(router): annotate the batch routing test assignments with Final	2026-10-20 07:01:12 +00:00
Mateo Wang	424687ce09	fix(proxy): build the OpenAPI schema off the event loop so a cold /openapi.json no longer stalls the worker (#47352) ... * test(proxy): keep answering requests while a cold /openapi.json builds * fix(proxy): build the OpenAPI schema off the event loop so a cold /openapi.json no longer stalls the worker * perf(proxy): reuse the rendered /openapi.json body until the schema changes * fix(proxy): scope the openapi build's pydantic patch to the build The schema build swapped a process-wide pydantic hook, which live requests saw once the build moved to a thread, and overlapping builds could leave it installed. The hook now checks a context variable set only inside the build. The proxy test counts liveliness answers instead of timing them. * test(proxy): hold the cold openapi build on an event instead of pacing on a clock * fix(proxy): serve a replaced app.openapi like FastAPI's own docs route * test(proxy): build the cold-docs test on the proxy module's current app * test(proxy): freeze the model that holds the cold docs build	2026-10-20 04:31:37 +00:00
hamzashah-dev	9b4f44135d	fix(proxy): refresh cached team on /team/block and /team/unblock (#35643) ... * fix(proxy): refresh cached team on /team/block and /team/unblock Both endpoints wrote `blocked` to Postgres and returned without touching the management-object cache. Auth resolves teams through `team_id:{team_id}` before reading the DB and `common_checks` enforces `blocked` off that cached object, so a team whose traffic had already warmed the cache kept serving requests after being blocked, and kept getting rejected after being unblocked, until the entry expired. `_refresh_cached_team` already exists for exactly this and its docstring says every endpoint mutating `litellm_teamtable` must call it; these two were the only mutating endpoints that didn't. The update now also asks for `object_permission` so the refreshed cache entry doesn't drop the relation, matching `team_model_add` and `team_model_delete`. * fix(proxy): keep a team block from being undone by in-flight spend writes update_cache read the cached team when a request finished and wrote that whole copy back in a background task, so a team blocked or invalidated while requests were in flight came back unblocked on every busy worker until its 60 s cache entry expired. The team spend now goes onto the team held in this worker's memory at write time, and a missing entry is left missing for the next auth read to load. Block and unblock responses keep litellm_model_table and object_permission null, as before this PR, while the cache still gets both relations. * test(proxy): type the team spend cache test parameters * test(proxy): hold a team block under in-flight team spend and keep team aliases routing after block and unblock --------- Co-authored-by: Hamza Shah <hamzasepal@gmail.com> Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-10-19 22:50:33 +00:00
devin-ai-integration[bot]	c80f1c2ec8	fix(router)!: count native Responses WebSocket turns toward deployment limits and budgets (#45727) ... * fix(responses): write one spend-log row per native WebSocket turn Each response.create turn on a native Responses WebSocket connection now logs its own call as it finishes, with its own usage and the resp_ id the client saw as request_id. A failed turn is booked alone and no longer flips the other turns. * fix(responses): bill the usage a failed native WebSocket turn reports * fix(proxy): pin the limiter slot owner to the proxy call id and refund unsettled reservations on disconnect The v3 limiter stashed whichever litellm_call_id the request body carried as the slot owner, so a client-supplied id on a pass-through body left the proxy's own success and failure events unable to release the slot. The owner is now the proxy's logging call id. A WebSocket connection whose turns log under their own ids never settles the TPM, ITPM, or OTPM estimate the connection reserved, so the disconnect release now refunds any reservation no callback settled, once. * fix(proxy): bill failed WebSocket turns and refund only the reserved window on disconnect A failed turn's reported usage now counts toward the key's TPM and the project ITPM/OTPM scopes the same way a success does, and closing the socket hands back only the estimate still unused in the window that holds it, so a window that already rolled is left alone. An id-less error event is booked against the newest pending turn, the one the backend rejected at parse time, and a turn that nests its input under response logs its user message. A connection's one budget reservation goes to the first turn that settles it, every later turn charging its cost directly. * test(proxy): expect the settling call id on a pass-through budget reservation * fix(proxy): charge successful Responses WebSocket turns to TPM and ITPM/OTPM A native Responses WebSocket turn reaches the rate limiter's success callback with its raw event list as the response and no combined_usage_object, so the turn's tokens never reached the key's TPM window or the project's input and output token limits. Read the usage from the turn's events when the call type is a Responses WebSocket turn. * fix(proxy): release a rejected pass-through's slot and count each websocket turn once toward project quotas The limiter's failure hook looked up the parallel slot by the request body's litellm_call_id, while the slot belongs to the proxy's logging call id, so a pass-through request a pre-call hook rejected held its slot for an hour when its body carried an id. Both now resolve the id the same way. Each response.create frame already charges its estimate to project ITPM/OTPM, so the turn's own usage no longer adds a second count. The cost callback's failure hook read the body's litellm_call_id through a strict model and raised on a non-string value, which dropped the failure row. It now reads a string id or none. * fix(proxy): keep per-turn WebSocket limits, request bodies and collector reservations right Free a rejected request's parallel slot through the request it runs in, and skip native Responses WebSocket connections, whose disconnect frees the slot. Count WebSocket turns toward the connection's tag TPM. In collector mode, claim the shared budget reservation before the spend event is built, so only the first turn settles it. Store each turn's own frame as its request body, and send an id-less error to the oldest waiting turn unless an earlier turn is streaming. * fix(proxy): store a failed WebSocket turn's own request and count tag TPM only for calls that carry the tag * fix(responses): log the client's turn frame, keep the payload id and reservation shape for non-WebSocket calls * fix(proxy): keep the legacy limiter's WebSocket slot until the socket closes * fix(router): count native Responses WebSocket turns toward deployment limits and budgets * fix(router): re-check deployment budget and RPM on each Responses WebSocket turn * test(router): list the WebSocket turn admission method as covered through its partial * fix(responses): give a refused WebSocket turn's error frame a 429 status so Codex ends the turn * fix(router): leave ITPM/OTPM to socket open and record a refused WebSocket turn as the socket's failure * fix(proxy): log the routed model group on a WebSocket turn's failure row * fix(router): count WebSocket turn tokens toward TPM and re-run routing strategy checks on each turn * test(integration): cover per-turn admission on Responses WebSockets Adds integration cells for budget, RPM, TPM, a mid-socket TPM edit, usage-based-routing-v2, the project quota frame, and the open-time failure frame on the native and managed relays, plus HTTP controls for TPM counting and the failure row model_group * fix(responses): count WebSocket turn tokens once and end a socket whose turn check fails A native Responses WebSocket turn is now logged as an async call, so a proxy with a sync success callback such as the response cache no longer runs each CustomLogger's sync success hook on top of the async one. That second run counted every turn's tokens twice toward deployment TPM. A turn whose admission check raises an unexpected error now gets a 500 error frame, the socket closes with 1011, and the error becomes the socket's failure, on both the native and the managed relay, instead of the socket dropping with no frame and no failure row. A refused turn's failure row now carries the request's tags from the router's metadata, the same way it already carries the model group. * test(integration): cover a crashed WebSocket turn check and failure-row tags A turn whose admission check raises gets a 500 server_error frame, the socket closes with 1011 and its one failure row carries the error, on both relays. The refused turn's failure row keeps the caller's tags, and HTTP failure rows on chat, responses and messages keep them too. * fix(responses): keep a WebSocket open after a crashed turn check and keep only string tags on failure rows A turn whose admission check raises an unexpected error still gets a 500 error frame, but the socket now stays open, like a refused turn. Closing it cut any turn still streaming on the same socket, which a client that sends its next response.create early would lose. A refused turn's failure row now takes the router's tags only when every tag is a string, so the fallback never copies a caller's unvalidated tags onto the row. * test(integration): cover a turn still streaming when the next WebSocket turn's check crashes A native turn still streaming when the next turn's admission check raises now has its own cell. It must complete and be billed at the turn's price. The crashed-check cell also keeps the fault set for one more turn, so a relay that stopped checking turns after a crash goes red. * fix(responses): run per-turn guardrails before the deployment's turn check on WebSockets A turn after the first now runs the project quota check, then the proxy's pre_call guardrails, then the deployment's turn admission, on both the native and the managed relay, the order HTTP calls use. A turn a guardrail blocks gets the guardrail's error even past the budget and no longer counts toward the deployment's rpm or runs its pre-call checks. * fix(responses): bill a WebSocket turn's guardrail cost when the deployment refuses it A turn the per-turn guardrail passes and the deployment then refuses now writes its own failed spend row with that turn's guardrail information, on both relays, the way a guardrail-blocked frame and an HTTP refusal already do * fix(responses): bill a refused WebSocket turn on one row only A refused turn whose guardrails recorded a cost is already billed on its own row, so it no longer also becomes the connection's close-time failure row * fix(responses): bill a refused WebSocket turn's guardrail cost on the socket's failure row The turns a deployment refuses after their guardrails passed now add their guardrail records to the socket's close-time failure row instead of a row of their own, so that row's failure hooks, model group and call type cover them * fix(responses): bill a WebSocket's refused turns when the session limit closes it * fix(responses): bill refused WebSocket turns when the session limit cancels the relay * fix(responses): free the WebSocket slot at the session limit without waiting on the failure row * fix(responses): keep the session limit's second cancel so a relay that swallows the first frees the slot * fix(responses): free the WebSocket slot before the close-time failure row and drain pending rows at shutdown * fix(proxy): commit spend before waiting on pending logging tasks at shutdown and again after * fix(responses): start the close-time failure row before the slot release so a late cancel cannot drop it * fix(responses): build the close-time failure row inside its task so a bad guardrail record cannot skip the crash close * fix(proxy): always commit spend again after the shutdown wait for pending logging tasks --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-10-19 15:35:20 +00:00
Mateo Wang	5012083b72	feat(vertex_ai): let operators drop chosen anthropic-beta values on the Vertex AI pass-through routes (#47185) ... * feat(vertex_ai): let operators drop chosen anthropic-beta values on the Vertex AI pass-through routes * fix(vertex_ai): drop configured betas sent through x-pass-anthropic-beta and only fail beta requests on a malformed setting The drop list now runs on the final upstream headers, after x-pass- expansion, through an upstream_headers_filter on pass_through_request, so x-pass-anthropic-beta cannot carry a dropped value to Vertex. A malformed setting now answers 500 only for requests that carry anthropic-beta or x-pass-anthropic-beta, so Gemini and discovery calls keep working. * test(vertex_ai): accept the new upstream_headers_filter argument in the existing Vertex route tests * fix(vertex_ai): refuse a malformed drop-betas list at boot and match list entries written with spaces * fix(vertex_ai): split drop-betas list entries on commas like the header	2026-10-18 12:51:27 +00:00
Mateo Wang	68a8b12ab3	fix(proxy)!: end a chat completions stream with an error event when a guardrail blocks mid-stream (#46953) ... * fix(proxy): end a chat completions stream with an error event when a guardrail blocks mid-stream A streaming guardrail that raised an HTTPException after the first chunk re-raised it inside the SSE generator, so the 200 body was cut off with no finish_reason, no error event and no [DONE]. Build the error frame through sse_error_payload instead, keeping the status, message and structured detail, and encode values JSON cannot represent as strings so the frame always goes out. * fix(proxy): keep early guardrail blocks on their HTTP status and end mid-stream errors with [DONE] A guardrail block before the first chunk is raised again, so its response stays the 400 event-stream body it is on main. A mid-stream error event now carries the failure hook's replacement error when one is returned, is followed by data: [DONE] on OpenAI-format streams, and leaves the Gemini route's raise unchanged. Error details JSON cannot encode are written as strings instead of breaking the event. * fix(proxy): keep a per-chunk guardrail block on the first chunk on its event-stream 400 * fix(proxy): treat a keepalive ping as a started stream so a later guardrail block ends in-band * fix(proxy): encode non-JSON guardrail details in early stream error frames An early guardrail block whose detail holds a value JSON cannot encode sent 400 headers and then cut the body. The error frame now writes such a value as its string form, the same as the mid-stream event. The keepalive test now runs on a virtual clock instead of real sleeps. * fix(proxy): show an in-band stream error event as an error in the CLI, Python client, and dashboard playgrounds ChatClient.completions_stream raises StreamError on a data frame carrying an error, so litellm-proxy chat prints the block reason and keeps the partial reply out of history. The prompt playground and the Interactions playground read the same frame through a shared streamErrorMessage helper. Adds the audit's integration cells for chat stream error events, and makes the endpoint tests' failure-hook mocks return None so they check the real error payload. * fix(proxy): print stream error text verbatim in the CLI and read split error events in the prompt playground The CLI printed the server's error message as Rich markup, so a closing tag like [/pii] in a guardrail message crashed lite chat. The prompt playground now buffers partial lines across reads, ChatClient raises only on a non-empty error, StreamError redacts keys like the other client exceptions, and StreamError is exported from litellm.proxy.client * fix(proxy): print CLI stream errors without Rich highlighting or emoji codes * test(proxy): cover the CLI's plain-text 400 error print * test(proxy): annotate the new client stream test locals as Final * fix(proxy): send the failure hook replacement on Responses streams and before the first chunk * fix(proxy): end a Responses stream with response.failed when it fails on a built-in exception * fix(proxy): keep a failure hook replacement before the first chunk in-band so bare stream routes still answer it A non-HTTP failure whose hook returns an HTTP error was raised before the first chunk, which create_response turned into the hook's status but which a bare StreamingResponse route such as /queue/chat/completions answered as a 200 with zero bytes. Only the original HTTP errors raise now. The replacement goes out as the stream's error event, which create_response still answers with the hook's status as a JSON error. Adds audit round 2's integration cells for failure hook replacements on chat and Responses streams, built-in exceptions on Responses streams, and the Python client. * test(proxy): cover a queue stream whose failure hook replaces a failure before the first chunk The audit cell posts to /queue/chat/completions with a callback that raises before the first chunk and a failure hook that answers 403. It checks for a 200 event stream whose only event is the replacement, then one [DONE], with the callback's own message absent	2026-10-17 18:00:12 -07:00
Mateo Wang	75faaeab64	feat(count_tokens)!: configurable token_count_framing and token_count_model, count source in the response (#46777) ... * feat(count_tokens): configurable token_count_framing and token_count_fallback_model, count source in the response Count endpoints now say where each number came from in litellm_count_source (provider, fallback_model, local_estimate) and name the fallback model in litellm_count_fallback_model. When the provider answers 400 or 404 for a model it cannot count, the proxy counts the same request with token_count_fallback_model. token_count_framing adds a per-model constant to local estimates. Both settings ship in the cost map for Bedrock Claude Sonnet/Haiku 5.5, Bedrock Kimi K3, and Databricks Kimi K3 and DeepSeek V4 Flash, and can be overridden in model_info. * test(count_tokens): classify the new cost map keys in the schema generator and expect count_source in exact-shape tests * test(count_tokens): e2e matrix against real inference and integration wire tests for count source, fallback and framing * fix(count_tokens): keep the provider error when local counting is disabled, name the fallback error, keep settings in cooldown Moves the cost map lookup off the event loop * ci(cost map): bound token_count_framing to non-negative and token_count_fallback_model to non-empty * fix(count_tokens): keep a failed fallback's own status, ship fallbacks only where they count, add system.ai framing A fallback model that answers 404 returns the requested model's error unchanged, and any other fallback failure answers with its own status and message. The eu, apac, au and jp Sonnet and Haiku 5.5 entries no longer ship the Opus 5.5 fallback, which AWS serves only in us-east-1. A configured empty token_count_fallback_model turns the shipped fallback off. Config framing accepts whole-number floats and rejects negatives. The Databricks system.ai Kimi K3 and DeepSeek V4 Flash names carry the same framing as their serving endpoint names, and the e2e matrix runs on Bedrock Haiku 5.5 too. * test(databricks): list the system.ai Kimi K3 and DeepSeek V4 Flash names with their published DBU rates * fix(count_tokens): skip a token count fallback model for an hour after it answers 404 on that deployment A us. or global. Sonnet 5.5 deployment outside us-east-1 paid two extra Bedrock calls and about 200 ms on every count, since Opus 5.5 cannot count there either. The Kimi K3 e2e also runs the large system prompt shape now. * test(count_tokens): use the shared integration port allocator in the fallback wire test * fix(cost map): keep the Databricks system.ai entries to framing only, so their spend stays as today * test(count_tokens): cover the fallback 404 skip, inference profile base_model, shipped framing, and deployment-scoped token count settings Adds integration cells for a fallback model that answers 404, an application inference profile ARN deployment whose base_model carries the shipped fallback, and the shipped Kimi K3 framing with token_count_framing 0 turning it off. Adds unit tests that keep token count settings off the shared backend cost map key and that store a token count setting on PATCH only when the operator changed it * fix(count_tokens): keep the counter's own 400 and the provider's error when a framed estimate cannot read the body A 400 the OpenAI or Gemini counter returns before calling the provider is the caller's answer, so it no longer tries the fallback model or a framed estimate. When the framed local estimate cannot read a content block, the client gets the provider's error instead of a 500. Adds /audit round 2's integration tests for the fallback leg's converted body, the Anthropic envelope on fallback failures, and both framing cases. * fix(count_tokens): answer the provider's error whenever a framed estimate raises A tool schema the local estimate cannot format, such as a boolean subschema or an array with no items, raised AttributeError or KeyError and answered a 500 on a framed deployment whose provider had already answered 400. Any estimate failure now answers the provider's own error. * fix(count_tokens): load the custom tokenizer inside the guarded framed estimate A framed deployment whose custom_tokenizer could not load answered a 500 or 503 where the provider had already answered 400, because the tokenizer was selected outside the estimate's error handling. Selecting it inside answers the provider's error, logs a warning with the estimate's error, and still raises the Rust extension's fork-guard errors. * fix(count_tokens): look up framing under the deployment's custom_llm_provider * fix(count_tokens): share one tokenizer download per worker and bound the wait after a provider error A framed deployment whose custom_tokenizer is not cached locally made every count wait out the Hugging Face hub's whole retry cycle when the hub could not be reached, and a burst of those loads held the default thread limiter that every count's cost map lookup also needs. Counts on a worker now share one download per minute, a failed download is reused for the rest of that minute, and after a provider error a count waits at most 5 s before answering the provider's error. The swallowed-estimate warning logs only the error type. * fix(count_tokens): keep one tokenizer download per key running and load the local route per request * test(count_tokens): type the scripted counter and wait on events instead of wall time where possible * fix(count_tokens): run framed tokenizer loads on their own capped threads and answer a tokenizer panic with the provider error Framed loads used anyio's default 40 threads, so a hub that drops packets let about 40 hanging loads stall every count on the worker. Loads now run on a per-worker pool of 4, and a count that finds 4 loads running answers the provider's error at once. Any finished load is reused for a minute after it finishes, so a load that fell back to tiktoken is not retried on every count. A pyo3 PanicException from the tokenizer, while loading or counting, now answers the provider's error instead of a plain 500 * feat(proxy): count tokens with token_count_model in its pinned region token_count_model {model, region} replaces token_count_fallback_model. Bedrock Sonnet and Haiku 5.5 count with Opus 5.5 in us-east-1, or Melbourne for au. profiles, and a failure names the region to enable * fix(count_tokens): use the renamed count model constant in the e2e test and list token_count_model with the object keys * fix(proxy): keep the upstream error envelope when adding the region hint * test(proxy): annotate the token count setting echo test * ci(cost map): require non-blank token_count_model strings and test the region rules * fix(count_tokens): answer the token_count_model's error even when framing is set * fix(count_tokens): keep the provider error for Google contents instead of a framed estimate * fix(count_tokens): estimate Google contents the route translated to messages * fix(count_tokens): keep the provider error for Google contents with non-text parts * test(proxy): drive tokenizer wait and cost map tests by events and a skippable loop clock * fix(count_tokens): check Google text-only parts without a KeysView comparison basedpyright flags * feat(proxy): count Bedrock Kimi K3 tokens exactly with its pinned tokenizer * test(bedrock): keep the Kimi K3 image check in-process * fix(count_tokens): join the region hint to an upstream message that already ends with a period Also pins that a blank token_count_model in config falls through to the cost map default * fix(count_tokens): count Gemini tool-call turns and Responses tools, and stop tokenizer retries from stalling counts The Google countTokens route estimates contents with functionCall, functionResponse and thoughtSignature parts again, and still answers the provider error for media, thought and unknown parts The Kimi K3 exact count renders Responses API and Anthropic tools through the chat tool translation the Vertex count already used, now shared from the prompt factory, and falls back to the estimate when the Converse body cannot carry a tool A failed Kimi K3 tokenizer load is retried in the background without making counts wait, and a load still running after the wait logs a warning * fix(count_tokens): count Kimi K3 tools the way Bedrock inference sends them A flat Responses API function or custom tool becomes the chat tool the Responses route sends to inference, and every other tool goes to the Converse body as is, so hosted tools Bedrock drops are left out of the count instead of being counted or turning it into an estimate. * test(count_tokens): annotate new token counter test locals with Final * fix(count_tokens): keep shipped count regions inside the deployment's geography A plain Bedrock Sonnet or Haiku 5.5 deployment in a non-US region no longer sends its count prompt to us-east-1. Global count models and configured token_count_model keep their pinned region. Also expect litellm_count_source in the Vertex Llama count test main added * fix(count_tokens): resolve count settings off the event loop	2026-10-17 17:11:27 -07:00
Kerry Lu	1552b5b69f	fix(proxy): warn at boot when a config environment_variables key replaces a value already set in the process environment (#47165) ... Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-10-17 20:52:01 +00:00
Mateo Wang	610443519a	fix(proxy): unset DB environment variables removed from the row on every worker by row revision (#46709) ... * fix(proxy): unset DB environment variables removed from the row on every worker by row revision * fix(proxy): order the environment variables revision migration after main's and set its lock timeout * fix(proxy): snapshot the environment at the first DB sync even when no environment_variables row exists yet * fix(proxy): close gaps in DB environment variable reconciliation Grant the revision sequence to every role so a writer role with only table grants can still save environment_variables. Unset SSO keys that only the environment_variables row stored on the next SSO sync, while keeping keys the SSO settings table holds. Skip a revision-less row once a revision has been applied, so a stale cached row cannot bring a removed key back. Keep plaintext and undecryptable values when rotating the master key instead of dropping them from the row. * fix(proxy): keep env var revisions monotonic across DB copies and restores Stamp at least the old revision plus one on update so a copied row keeps rising, re-read a stale cached row from the writer and apply it when the database itself went back, unset dropped SSO keys when sso_settings are not loaded from the DB, and keep env values master key rotation cannot decrypt exactly as stored * fix(proxy): skip an env var row read before an overlapping sync that already applied a newer one * test(proxy): cover a read replica back on an older row and an SSO client removed from DB environment_variables * fix(proxy): keep config reads up when the writer is down, find the revision sequence by table schema, read the writer before rewriting env vars, and add the trigger under db push * test(proxy): model the writer database in the retry policy config update fake * fix(proxy): recheck the environment_variables row revision right before applying it A sync that started later could read a stale environment_variables row before an earlier sync applied a newer one, then pass the read-order check and apply the stale row, bringing back keys the newer row removed. The behind-the-applied-revision check now runs at apply time, re-reading the writer when the row is behind, and a writer row read while another sync applied a newer revision is dropped. The config sync tests now inject a fake database client and use the real get_config_param and its cache instead of replacing get_config_param. * test(proxy): keep encryption real in the SSO replica-lag test * test(proxy): inject a fake Prisma client in the env var replica-lag test * fix(proxy): keep the applied environment variables when the writer database is out of connections * test(proxy): cover a writer database out of connections in the env var integration tests	2026-10-17 03:51:05 +00:00
Kerry Lu	339fd90836	fix(proxy): streaming responses hand the event loop back every 50 ms instead of holding it for the whole buffered stream (#46925) ... * fix(proxy): streaming responses hand the event loop back every 50 ms instead of holding it for the whole buffered stream * fix(proxy): yield to the event loop on every streamed chunk, no hold budget or clock * docs(proxy): say why the streaming loop yields on every chunk * fix(proxy): streaming loop yields once it has held the event loop 50 ms, clock read inline * docs(proxy): streaming loop comment names the 50 ms hold * Revert "docs(proxy): streaming loop comment names the 50 ms hold" This reverts commit cbf1371873. * refactor(proxy): streaming loop hold budget is _STREAM_LOOP_HOLD_MS = 50 * Revert "refactor(proxy): streaming loop hold budget is _STREAM_LOOP_HOLD_MS = 50" This reverts commit 356454def6. * docs(proxy): streaming loop comment notes the yield is a floor, not a count of releases --------- Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-10-16 11:33:01 -07:00
devin-ai-integration[bot]	e45f2737d3	fix(proxy)!: delete callbacks configured under callbacks or failure_callback (#42740) ... * fix(proxy): delete callbacks configured under callbacks or failure_callback Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(e2e): assert on configured callback rows and use a callback absent from CI configs Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(e2e): follow logging suite conventions for callback delete tests Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(proxy): flatten remaining callbacks with a helper generator Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(proxy): type callback delete helpers to satisfy basedpyright budget Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(proxy): save callback config without mutating the loaded dict Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(proxy): delete callbacks from every litellm_settings list GET shows POST /config/callback/delete only searched success_callback while GET /get/config/callbacks also lists failure_callback and callbacks, so the Admin UI Delete returned 404 for those rows Remove the name from every list that holds it, matching entries case-insensitively, and skip lists the config file owns when the database owns another copy so the save is not refused Replace the e2e test with an integration test: the shared e2e stack config owns litellm_settings.callbacks, so it cannot exercise that list * test(proxy): cover config-owned success_callback on callback delete and wait on runtime state Add the mirror mixed-ownership case where the config file sets success_callback and the database owns failure_callback, which main refused with a 400, plus a string-valued list case Poll GET listings and /active/callbacks instead of reading them once, so a request served by a worker that has not synced yet cannot fail the test, and reuse the existing callback list key map in the route * test(proxy): annotate callback delete test helper locals with Final * fix(proxy): save callback delete through the config it read so concurrent settings edits are kept --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: yucheng-berri <293926326+yucheng-berri@users.noreply.github.com>	2026-10-16 09:22:22 -07:00
Mateo Wang	2dc6aafa6b	fix(proxy)!: count prompt-only token_counter requests with the provider when call_endpoint is set (#46839) ... * fix(proxy): count prompt-only token_counter requests with the provider when call_endpoint is set * fix(proxy): keep the local count for a prompt-only token_counter request when the provider count fails * fix(proxy): log a prompt-only token count fallback as a one-line warning * fix(proxy): send a prompt-only token_counter request to the provider only on Gemini and Vertex AI * fix(proxy): count a token_counter prompt sent next to an empty messages or contents list * fix(proxy): count a prompt next to messages on the local token count instead of answering 500 * fix(proxy): answer 400 for a token count prompt next to a conversation unless drop_params is set With drop_params the conversation is counted without the prompt. On the local count, a prompt sent with system or tools is counted as a user message with them, like the messages form. * test(proxy): count tools with a prompt in the token counter fallback test * test(proxy): cover prompt counting on /utils/token_counter in the Gemini count wire tests * test(proxy): name the token counter fallback test by what it checks	2026-10-16 10:54:42 +00:00
Tufail Akram	84eeebd509	fix(proxy): track and drain management audit log writes on shutdown (#43594) ... * fix(proxy): track and drain management audit log writes on shutdown * fix(proxy): track hook spawns and defer registration in audit log drain * fix: satisfy LIT006 on audit log task registry by documenting the required cast * fix(proxy): satisfy LIT006 and LIT010 in audit task registry * test(proxy): drive the audit drain tests with events instead of wall-clock waits * fix(proxy): drain audit writes beside the shutdown flushes and track only premium writes Run the audit drain concurrently with the gateway and request-error flushes so the two waits overlap inside the teardown budget, register the running audit task only after the premium check, drop the four duplicate task-failure loggers, and cover the shutdown order and a real hook spawn with tests * fix(proxy): start hook audit writes before their email and secret manager calls Key and user hooks now spawn their tracked audit write first, so a slow invite email or secret manager call no longer spends the drain budget before the write starts. The endpoints stop tracking the hook tasks, so shutdown never waits on hook I/O when audit logging is off. Adds regression tests for the hook order, the failed block or unblock update, and the shutdown drain without a database. * fix(proxy): track only the audit insert, not the code that awaits it The shutdown drain now waits on each audit row insert itself, so it never waits on a caller's later work and covers every caller without per-endpoint wrappers. The user/new hook still tracks its own task because it reads the user row before the insert * test(proxy): drain the audit registry in the config audit tests The audit insert now runs in its own tracked task, one hop after the caller's task, so a single yield no longer reaches it * fix(proxy): name dropped audit rows in the drain timeout warning Each tracked audit task is named after its action, table, object id and destination, so a shutdown that drops a row says which one. Adds tests that the drain waits for slow audit callbacks and that the warning names the dropped row --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-10-16 08:41:34 +00:00
ryan-crabbe-berri	38f3f8a1a9	fix(proxy): cooldown errors carry 429, Sentry starts at boot without request bodies, tracing failures log at ERROR, exporter counters exported (#46344) ... * fix(router): carry a 429 status on cooldown and no-deployment errors RouterRateLimitError and RouterRateLimitErrorBasic had no status_code, so the proxy answered an all-deployments-cooled-down request as a generic 500 class error and logged the bug report notice for every cooldown. Both now carry status_code 429, which the proxy maps to a rate_limit_error and which the retry policy already treats as retryable * fix(proxy): start Sentry at config load and never attach request bodies Sentry only came up inside set_callbacks on the first LLM call, so boot and background errors were never reported, and general_settings.alerting: [sentry] always raised Missing SENTRY_DSN because the alerting path read a dead litellm.utils pin instead of the live SDK. The proxy now initializes Sentry once while loading the config whenever sentry is a configured callback or alerting target, the alerting and failure handlers read the live instance, and max_request_body_size is pinned to never so prompts and completions stay out of Sentry events * fix(tracing): log Lens failures at ERROR and export row counters to Prometheus A tracing store that fails to start, or a Lens service that rejects a batch, was only logged at WARNING, so operators running at the default ERROR level never saw tracing silently go dark. Both now log at ERROR and name the lens store. The exporter's rows_written and rows_dropped tallies were never exported, so the two counts are now Prometheus counters litellm_lens_export_rows_written_total and litellm_lens_export_rows_dropped_total, registered lazily on the default registry like the other proxy-level counters and charted on the all-metrics Grafana dashboard * test(proxy): drop the litellm import the alerting tests no longer use * fix(proxy): start Sentry once a DSN is set, from any config source Boot-time init used to lock Sentry in even with no SENTRY_DSN, so a DSN loaded later from DB environment_variables was never picked up. initialize_sentry now skips when no DSN is set and re-inits when the DSN changes. alerting_handler calls it too, so alerting: [sentry] set through the DB works. Drops the litellm_settings.callbacks check since that config fails to load with sentry in it * fix(sentry): keep stack frame locals out of events unless PII is opted in Frame locals carried prompts (messages, request bodies, provider options) into every error event. include_local_variables now follows SENTRY_SEND_DEFAULT_PII * test(router): drop the test that only pinned the 429 class attribute * fix(proxy): start Sentry at boot when it is only an input callback * fix(proxy): keep Sentry init and alerting within the strict ruff and basedpyright ceilings --------- Co-authored-by: yuneng-jiang <yuneng@berri.ai>	2026-10-16 00:53:52 +00:00
L	bdd80781f2	fix(proxy): list team access group models in /v1/model/info and keep key model lists narrow (#32599) ... * fix(proxy): list models from team access groups * 修复 PR #32599 的类型检查并补齐模型列表回退测试 * fix(proxy): keep a key's own model list narrow when adding team access group models A key with its own model list was shown every model from its team's access groups on /v1/models, /v1/model/info and /model_group/info, though auth denies those calls. Team access group models now widen a key's list only when the key inherits the team's models. /v1/model/info reuses the shared listing helper without a catch-all, and route-level tests replace the mock-only ones * fix(proxy): skip the list build in the unrestricted model info check and test every listing route The unrestricted check for /v1/model/info only needs to know whether the key and team lists are empty, so it no longer builds and expands the model list a second time. The route test now covers /v1/models and /model_group/info next to both model info routes * fix(proxy): scope model info for personal and agent keys with no model list * fix(proxy): keep config keys' team access groups and own-team team_id scope in model listing --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-10-15 10:23:03 -07:00
devin-ai-integration[bot]	3740d48844	test(cost): add tier 1 claude messages stream twins and recapture drifted usage shapes (#46690) ... * test(cost): add tier 1 claude messages stream twins and recapture drifted usage shapes Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(proxy): freeze the login throttle clock so the retry-after assertion is deterministic Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-10-15 17:11:13 +00:00
Hayden Moulds	de0ed1557e	fix(proxy): resolve model aliases to their target deployment for listing metadata (#41041) ... * fix(proxy): resolve model aliases to their target deployment for listing metadata A model alias (a key's `aliases`, a team's `model_aliases`, or the router's `model_group_alias`) is rewritten at request time and owns no deployment row, so `get_model_listing_info` returns None for it and the alias name is not a cost-map key either. Both metadata candidate sources come up empty and the listing entry carries no `mode`, `max_input_tokens`, or `max_output_tokens`, while the deployment it routes to reports them on the same response. Resolve a listed alias to the deployment it routes to for the metadata lookup, keeping the alias as the response id. Maps are applied in order, each against the result of the last, mirroring how `litellm_pre_call_utils` applies the team map and then the key map, so a chained alias reports the deployment the caller actually reaches rather than an intermediate one. An alias shadowing a same-named deployment follows the alias, because the request rewrite does too. Fixes the model_aliases half of #40553. * fix(proxy): look listed aliases up through the shared alias chain Drop the separate alias resolver and retarget a listed alias in alias_listing_entries and alias_target instead, so /v1/models and /v1/models/{id} report the model /chat/completions rewrites it to, and only when the caller can list that model * fix(proxy): keep deployed listed models' own rows and resolve team alias targets at list and retrieval * fix(proxy): resolve router_settings aliases, keep alias labels, and skip paused targets in model listing * fix(proxy): leave an alias to an undiscoverable model bare in model listing A team alias whose target has discoverable: false showed the hidden model's mode and limits on GET /v1/models and retrieved it by alias. Undiscoverable groups now count as hidden for alias resolution, as on main, while admins still see them. * fix(proxy): leave an alias to a model a listing callback hides bare in model listing * fix(proxy): keep aliases bare for models a listing callback never judged or another team owns --------- Co-authored-by: haydster7 <haydster7@users.noreply.github.com> Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-10-15 00:38:50 -07:00
TheMentalCoding	ca04c4909e	feat(ui): filter active and paused model deployments (#43710) ... * feat(ui): status filter + sort and provider-aware search on Models & Endpoints The models table mixes active and paused deployments with no way to tell them apart beyond the pause toggle, and the drawer only filters by public model name and access group. Backend (/v2/model/info): - new optional `blocked` query param to filter by routing status - new `blocked` sort field (asc = active first); the existing `status` field keeps sorting by config-vs-DB source - `search` now also matches litellm_params.model, so typing a provider or upstream model id (e.g. "nvidia_nim", "openai/gpt-4") finds deployments whose public name doesn't mention them Frontend (Models & Endpoints): - visible Status column with Active/Paused badges, sortable (active first on asc) - Status filter (All/Active/Paused) in the Filters drawer, wired to the new server param through URL state The routing-status filter ignores non-bool sentinels the same way exclude_auto_routers does, so direct calls bypassing FastAPI keep their no-filter behavior. * fix(ui): inverted routing-status filter mapping and compact Status column Selecting "Active" in the Filters drawer sent blocked=true to the server, showing the paused deployments instead of the active ones (and vice versa). The mapping now lives in routingStatusToBlocked() with unit tests, and the Status column is narrower so the row actions stay in view. * fix(ui): pin the Actions column to the right on the models table With the new Status column the table overflows on narrower screens and the pause/resume switches (last column) scroll out of view. Pinning the Actions column keeps them visible; uses the existing DataTable meta.pinned support (same pattern as MemberTable). * refactor: readable db search where-condition and ruff-format clean * fix: apply routing-status filter inside the search flow and regen API types Greptile flagged (P1): with search + a status filter combined, the bounded DB fetch ran before status filtering, so matches of the other status could consume the page budget and leave short pages / stale totals. - extract _matches_routing_status() (sentinel-safe) and apply it to the router-side matches inside _apply_search_filter_to_models - push the status condition into the DB where clause of _fetch_db_models_for_search, so rows of the other status never count against the bounded fetch - endpoint passes blocked through to the search flow; tests cover the combined search+status case and the generated where clause - regenerate ui/litellm-dashboard/src/lib/http/schema.d.ts for the new query param - update useModels.test.ts expectations for the new trailing argument * fix(proxy): keep the flat DB where shape for single-condition searches The upstream CI test test_apply_search_filter_honours_exact_model_name_in_db_query pins the Prisma where shape: exact model_name stays a top-level key, and plain substring search keeps {model_name: {contains, mode}} at the top level. Only genuinely multi-condition queries (status filter and/or router-id exclusion) now wrap in AND. * fix(proxy): avoid reassigning a Final variable in the db where builder basedpyright's reportGeneralTypeIssues budget counts the reassignment of the Final-annotated db_where_condition; express it as a single conditional expression instead. * chore(deps): bump pypdf to 6.19.0 to clear osv-scan pypdf 6.16.2 has 8 High CVEs (GHSA-5jq2-8x83-x246 and aliases) fixed in 6.17.0-6.19.0; bump past all of them * Revert "chore(deps): bump pypdf to 6.19.0 to clear osv-scan" This reverts commit 0e2528123db7ab23304ae20f6a4c0961d0e2d073. * fix(proxy): keep LIT002 budget flat in the model search where-builder Extract the dynamic Prisma where-condition fragments into module-level helpers so every dict starts on a short line, split the litellm_params fallback lookup into a Final local, and annotate suppressions on the construct start lines the checker attributes * chore(tests): repoint unbounded-IN baseline at the extracted where-builder helper The bounded-router-id filter moved from _fetch_db_models_for_search into _not_in_router_ids, so the old baseline key is stale and the new helper site is the finding to record * fix(proxy): avoid searching encrypted deployment model names * refactor(ui): keep model routing status as a filter only * docs(api): regenerate model info sorting description * fix(api): remove remaining paused status sort references --------- Co-authored-by: thementalcoding <312162298+thementalcoding@users.noreply.github.com> Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-10-14 23:19:59 -07:00
Yassin Kortam	ca2f99edac	fix(caching): make Redis Cluster commands wait for a pool slot instead of tripping the circuit breaker (#46488) ... Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-10-14 15:27:00 -07:00
yujonglee	dff8131f14	refactor(rust_bridge): route Python state reads through host modules (#45689) ... * refactor(rust_bridge): group the Python modules Rust imports by name under rust_bridge/host Rust reads Python-owned state (legacy callback registries, model capabilities) through modules it imports by path. Put them in one folder with an AGENTS.md that states the contract, and update the module path constants on the Rust side. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * refactor(rust_bridge): route Python state reads through host accessors * docs(rust_bridge): point test_support at the moved host contract test Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * refactor(rust_bridge): move secret manager, transport and timeout accessors under host Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * chore(greptile): register rust_bridge/host AGENTS.md in files.json Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-10-11 14:10:16 -07:00
devin-ai-integration[bot]	8a103e26ab	test: move routing, caching and callback tests out of local_testing (#45761) ... * test: move routing, caching and callback tests out of local_testing Moves the 37 routing, caching and callback files in tests/local_testing into mirrored tests/unit paths and tests/integration sdk and security groups per the migration manifest, and deletes the 8 rows it marks for deletion * test: cover least-busy routing and tighten migrated router, cache and pass-through tests Add router-level least-busy tests for the three legacy rows that had no destination, move completion fallback and model alias tests to their mirror files, move router batch tests next to the router, and strengthen tests whose mutants survived (specific deployment, batch pair grouping, custom cache key, router cache ttl). Add a unit test for configured pass-through Authorization forwarding. * test: tighten migrated callback, proxy config and cache ttl tests Assert the full StandardLoggingPayload key set and the wire user on callback tests, add an async embedding failure callback test, drive the prometheus redis failure through a real RedisCache, use the real salt key in proxy config tests, pin the redis ttl value and rename fallback test helpers * test: fix semantic cache assertions and order-dependent router and callback tests Count semantic cache embeddings on the route, drain async cache writes before the second lookup, echo the requested model from the callback provider mock so response_cost does not depend on model_cost entries other tests register, and move the two-key authorization assertion to the auth failure test it belongs to * test: cover model_list fan-out and the usage-based routing tpm limit lost in the move * test: wait for the streamed cache entry in redis and drop deleted files from the hosted mock guard * test: restore sync callback, audio redaction and langfuse export coverage, keep redis-compat filter on the moved file * ci: run the remaining router-keyword legacy tests on one node * test: refuse the redis write in-process instead of resolving an invalid host --------- Co-authored-by: yuneng <yuneng@berri.ai>	2026-10-10 09:33:09 -07:00
devin-ai-integration[bot]	141dba2372	fix(proxy): keep the mapped status on assistants and threads route errors (#44994) ... * fix(proxy): keep the mapped status on assistants and threads route errors * test(proxy): type and annotate the assistants and threads route error tests * fix(proxy): redact internal details from assistants and threads route error messages * test(proxy): integration cells for assistants route error mapping * test(proxy): drop malformed JSON cases that never reach the assistants route handler --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-10-09 16:55:15 -07:00
moe-berri	22ad3c5fff	refactor(lens): connect LiteLLM to the independent Lens service (#45529) ... * feat(lens): consume the independent Lens service and shared UI * fix(lens): keep embedded setup stable and vendor UI before builds * chore(lens): pin embedded UI provenance to published Lens source * fix(lens): complete embedded UI extraction across CI builds * docs(lens): record latest main and removal-boundary validation * fix(lens): wire external chart credentials and ingestion modes * test(lens): cover adapter failures and fix extraction CI * docs(lens): refresh bundled chart setup instructions * test(lens): restore gateway adapter test package marker * build(lens): pin clean-cut UI and supported storage chart * test(ui): await guardrail scope menu before checking options * perf(lens): omit unused trace-team lookup from product requests * docs(lens): record final-image adapter permission checks * test(lens): assert delegated investigation authorization * refactor(lens): remove copied trace runtime and preserve spend logging * test(lens): verify independent gateway image upgrades and outages * test(lens): pin the chart qualification database image * fix(lens): update embedded UI metadata filters * test: wait for Lens chart pod identity convergence * fix(e2e): select Lens chart pods by deployment ownership * fix(e2e): select Helm migration ownership for Lens upgrades * test(lens): retain migration Jobs through rollout assertions * docs(helm): require existing database for migration hooks * test(lens): qualify release boundaries with embedded navigation update * docs(lens): record browser qualification for embedded tabs * fix(lens): remove unrelated CI and guardrail test changes * test(lens): give qualification tenants unique key aliases * feat(lens): embed the latest canonical Lens interface * fix(lens): qualify split inference through the gateway * fix(lens): preserve scoped auth and isolate delegated credentials * chore(lens): remove unrelated documentation and lint changes * fix(lens): bound concurrent forwarding buffers through response delivery * fix(lens): preserve OAuth2 dispatch without inference policies * chore(lens): adopt current shared onboarding UI * fix(lens): refresh shared setup UI and review context * fix(lens): bound uploads before gateway authentication * docs(lens): remove extraction evidence from the gateway repo * docs(lens): refresh shared setup prompt for paired releases	2026-10-09 16:17:05 -07:00
devin-ai-integration[bot]	0d17f954c0	feat(credentials): add display_name and make credential_name immutable on PATCH (#43148) ... * feat(credentials): add credential_alias and make credential_name immutable on PATCH Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(ui): query credential rows by role to stay under the no-node-access budget Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(proxy): expect credential_alias in load_credential_list dump Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(ui): drive the credential SearchSelect by placeholder and option roles Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(proxy): pass the decrypted CredentialItem to update_db_credential during master key rotation Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(ui): trim credential_alias in CredentialModal so whitespace-only input clears the alias Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * feat(credentials): replace credential_alias with display_name and reject edits to config credentials credential_name stays the immutable reference key. display_name is a nullable, trimmed, max 255 character label set on POST or PATCH (omit keeps it, null clears it, blank is a 400). Reads return display_name plus an in-memory source tag (db or config), and PATCH or DELETE on a config-defined credential answers 400 instead of a misleading 404 * feat(ui): show credential display names everywhere and lock config credentials The credentials table shows the display name with the credential name beneath it, badges config credentials and disables their edit and delete actions. The edit modal adds an editable display name next to the read-only credential name and only sends it when it changed. Every credential picker and reference (add model, model info, models table, vector store form and info) shows the label while still submitting credential_name * feat(cli): set display names on credentials and show them in the list * refactor(credentials): keep the new credential types inside the lint gate budgets * refactor(cli): print credential command JSON through one helper * fix(credentials): treat an empty credential_name on PATCH as omitted and pin the 404 for vanished rows A blank credential_name never renamed anything, so PATCH accepts it again instead of answering 400. New tests pin a trimmed display_name on PATCH, the repository carrying display_name, and a 404 (not the config-owned 400) for a DB credential another worker already deleted. The hydration helper no longer copies display_name, since none of its callers read it, and the CLI update passes display_name straight through because the exactly-one check already makes it None when clearing. * fix(ui): keep a model's credential when its picker text is emptied and skip the admin-only list for other roles Clearing the search text in the model edit form's credential picker used to submit null, silently detaching the model's credential on save. None is now the only way to clear it. The models table fetches /credentials only for proxy admins, since everyone else got a 403 on each page load and falls back to the raw name anyway. Also fixes a type error in the re-use credential dialog and adds tests for the display name surviving a provider switch, the 255 character limit, the request payload, and the vector store None choice. * fix(client): percent-encode the credential name when updating its display name A name with ? or # was cut short in the URL, so the PATCH landed on a different credential or 404'd. * fix(credentials): answer 405 with Allow: GET for PATCH and DELETE on config credentials A config-defined credential exists (GET returns it) but is read-only through the API, which is what 405 Method Not Allowed means. 400 described a malformed request, and 404 would claim the credential does not exist. * fix(proxy): ignore a display_name set on config credential_list entries so non-string values cannot fail boot --------- Co-authored-by: yuneng <yuneng@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-10-08 23:49:56 -07:00
joshua-berri	04f0ea8c33	feat(mcp): translate input requests and bind continuations (#45464) ... * feat(mcp): translate input requests and bind resumable continuations * fix(mcp): keep continuations bound to their original upstream * fix(mcp): sync advertised protocol API schema * test(mcp): run interaction regressions in GitHub CI * test(mcp): set source paths for interaction proxy processes * test(mcp): verify salt guidance across interaction carriers * fix(mcp): preserve interaction authentication and elicitation policy * fix(mcp): bound preflight bodies and preserve challenge routes * test(mcp): type interaction regression fixtures --------- Co-authored-by: Joshua Valluru <326636767+joshua-berri@users.noreply.github.com>	2026-10-08 20:10:10 -07:00
Mateo Wang	fbbb54c841	fix(proxy): answer a rejected websocket handshake without crashing the HTTP exception handler (#45251) ... * fix(proxy): answer a rejected websocket handshake without crashing the HTTP exception handler * fix(proxy): read the OTLP route check's scope keys with .get and drive the websocket auth tests through the real auth path * test(proxy): build the websocket auth test keys without reading the clock	2026-10-07 22:27:46 -07:00
Mateo Wang	58258409c9	refactor(proxy): expose public names for private proxy helpers (#45170) ... * refactor(proxy): expose public names for private proxy helpers Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(proxy): keep original class names behind public aliases Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(proxy): preserve internal callback filtering Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(proxy): keep _PROXY_ class names for managed files hooks Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(proxy): keep old private names in package exports Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(proxy): preserve recursive auth helper name Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(proxy): align MCP limiter tests with server enforcement Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(proxy): match main's MCP limiter tests Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(proxy): keep old private names bound in importing modules Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(proxy): preserve compatibility imports through strict lint Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(proxy): use exact pyright suppression in password helper test Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * style(proxy): add reasons to compatibility import noqa comments Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(proxy): update IN-list baseline for renamed helpers Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-10-07 18:59:27 -07:00
moe-berri	e9cfba2c17	feat(lens): isolate ingestion and investigations in a Rust service (#45148) ... * feat(lens): isolate trace storage and investigation in a Rust service * fix(lens): include Rust sources in the image build context * feat(lens): wire service setup, scoped delivery receipts and lease attempts * fix(lens): complete service routing and reject stale investigation results * fix(lens): retry key propagation and validate isolated Compose setup * fix(lens): seed through isolated ingestion and preserve upstream queue fixes * chore: sync schema.prisma copies from root * fix(lens): bind nullable due timestamps as text for Prisma * chore(ui): remove stale lint suppressions * fix(lens): address CI failures and review findings * refactor(lens): remove retired Python worker and run evaluations in Rust * fix(lens): reuse control connections and satisfy review checks * test(lens): install and upgrade both Helm charts on Kubernetes * test(lens): run connection reuse coverage as an integration test * fix(ui): upgrade Next.js to 16.3.8 security release * fix(lens): fence stale attempts and preserve reviewed evidence * Revert "fix(ui): upgrade Next.js to 16.3.8 security release" This reverts commit 2f79a51b25. * fix(lens): stop failed investigations and stream history excerpts * test(lens): cover model tool and result contracts * test(lens): fix retired routes and reuse installation build artifacts * test(lens): use portable grep in Helm installation smoke * test(lens): wait for migrations before forwarding Helm services * fix(lens): keep failed evidence reads retryable * fix(lens): preserve sandbox output during process exit --------- Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>	2026-10-07 17:18:58 -07:00
devin-ai-integration[bot]	498a3e1a67	test(proxy): make the stagger-offset and linear-dedup guards independent of runner identity and load (#45031) ... * test(proxy): inject the cleanup job stagger offset so the runtime sync test no longer depends on the runner's host and pid Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(policy_engine): count resolver calls instead of timing them so the linear dedup guard is deterministic under CI load Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(policy_engine): annotate the line-count guard's locals as Final Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-10-07 04:07:36 -07:00
devin-ai-integration[bot]	c3a23fe499	refactor: expose core private helpers under public names (#44871) ... * refactor: expose core private symbols with compatibility aliases Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * types: narrow core migration diagnostics Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix: preserve runtime behavior in core symbol migration Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix: preserve core private usage migration behavior Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix: preserve private value rebinding compatibility Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix: restore optional imports and cover public helpers Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test: exempt router property from call coverage Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test: align recursive detector ignore names Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: mateo <mateo@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-10-07 00:05:27 +00:00
devin-ai-integration[bot]	282d733fb6	feat(auth): deny search tools by default when search_tool_deny_by_default is set (#44490) ... Co-authored-by: mrinal <mrinal@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-10-06 12:50:26 -07:00
devin-ai-integration[bot]	abc543e701	feat(proxy): add opt-in vector_store_deny_by_default for least-privilege vector store access (#44244) ... * feat(proxy): add opt-in vector_store_deny_by_default for standalone virtual keys Adds general_settings.vector_store_deny_by_default (typed bool, default false). When enabled, a virtual key with no team must list the requested vector store in its object_permission.vector_stores; no permission record, null, or an empty list is denied with key_vector_store_access_denied. Omitted or false keeps the existing behavior, including nonempty allowlist enforcement. The master key is unchanged in both modes. Team keys and keyless callers are deferred to later increments of LIT-6035 Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * feat(proxy): require key and team vector store grants for team keys under vector_store_deny_by_default With the flag enabled, a virtual key on a team needs both its own grant and its team's grant for every requested vector store. A missing permission record, an empty list or an unresolved team grants nothing. Dashboard session keys and the master key keep their existing behavior, and flag-off behavior is unchanged Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * feat(proxy): require user or team vector store grants for keyless requests under vector_store_deny_by_default Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(proxy): cover vector_store_deny_by_default through a real proxy with key, team and JWT identities Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * chore(ui): regenerate schema.d.ts for vector_store_deny_by_default Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(auth): cover path and file_search vector store ids under vector_store_deny_by_default Strict mode now reads vector_store_ids and tools[].vector_store_ids from the request body without needing a vector store registry, so /v1/vector_stores/{id}/search and Responses file_search are checked. User grants load through the object permission cache, and the proxy admin user rebuild keeps object_permission_id Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(proxy): broadcast user entitlement cache eviction to every worker Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * refactor(auth): reuse VectorStoreRegistry id extraction for vector_store_deny_by_default Strict mode now uses get_vector_store_ids_to_run on an empty registry when none is loaded, instead of a parallel set of request-shape helpers, and vector_store_access_check documents the policy Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): add strict vector store audit cells for routes, SDKs, workers and concurrency Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): consolidate vector_store_deny_by_default coverage to core cases Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(proxy): reject invalid vector_store_deny_by_default at config load and return 400 for malformed vector_store_ids Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(auth): read vector store key and team grants through the object permission cache Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(auth): return typed permission rows in request flow vector store tests Co-authored-by: mrinal <mrinal@berri.ai> Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * refactor(auth): validate vector store ids and tools as immutable sequences Co-authored-by: mrinal <mrinal@berri.ai> Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: mrinal <mrinal@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-10-06 12:50:25 -07:00