It asserted no provider name is a substring of any other provider's default api_base, so any new provider whose name sits inside an existing hostname (sail vs parasail) broke main without a bug in our code. The litellm_proxy default api_base fix it originally guarded is covered by the explicit api_base tests in the same file
Co-authored-by: kerry <kerry@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Kwargs LiteLLM code introduces for its own use were only kept out of provider
bodies if someone also listed them in all_litellm_params. Undeclared ones went
into extra_body or optional_params, reached the provider, and the provider
rejected the request. is_litellm_owned_kwarg in types/utils.py now defines
LiteLLM-owned once: a registered name, or any name starting with
INTERNAL_KWARG_PREFIX from litellm/constants.py. Every filter that builds
provider params from kwargs uses it: chat completion, transcription,
embedding, image generation and edit, search and video, ElevenLabs text to
speech, and the Bedrock batch mapper. The two untyped shared filters now take
Mapping[str, object]
The stream_chunk_size wire test becomes test_internal_params_wire.py. It also
sends an undeclared _litellm_ kwarg and asserts that no _litellm_ key reaches
any of the six provider bodies, while extra_body passthrough keeps working
Refs LIT-8318, LIT-8319
* docs(pr-template): add the backport-stable label only for a P0 regression
* docs(pr-template): keep a narrow security regression eligible for backport-stable
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* feat(proxy): add maximum_daily_tag_spend_retention_period cleanup setting
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* chore(ui): regenerate schema.d.ts for new retention setting
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(proxy): rebase daily tag spend retention onto the run-budgeted cleanup job
Reworks the cleanup on top of the refactored SpendLogCleanup: the daily tag spend table is pruned through the shared batched delete with a text cutoff on the indexed ISO date column, the setting is picked up by /config/update and the scheduler registration, and an integration test proves rows older than the period are pruned while the cutoff day and unset retention are left alone
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): schedule the cleanup job when a retention db row lands before the side effects run
A config reload applies the db row to the SettingsStore before _update_general_settings snapshots the previous retention values, so the before/after compare saw no change and a retention period first set through /config/update never scheduled the cleanup job. Also reschedule when the job is missing but a retention period is set
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): accept list-valued top-level keys in the base integration proxy config
The shared tests/integration/proxy_config.yaml now carries list-valued top-level keys, so the retention config helper validates only the mapping it merges into. Also drops a SQL-shape assertion from the unit test in favor of the behavioral cutoff-day check
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): cover runtime update, invalid value, independent horizons and worker loss for daily tag spend retention
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): restore the shared retention setting, capture seeded days once and kill a listening worker
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): retry a failed cleanup schedule only when its settings change
_apply_retention_settings rescheduled whenever retention was set and no job existed, so an unparseable cleanup cron was retried on every config reload. Remember the last attempted retention, cron and interval tuple and retry only when it differs
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): give daily tag spend retention tests a 240s timeout
Each node boots a proxy and waits for a whole-minute cleanup cron tick, so the global 90s pytest-timeout can expire during teardown on a slow runner, as integration-accounting did on pipeline 90302
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): reschedule cleanup when only the cron or interval changes at runtime
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): reschedule cleanup when the first db sync changes only the cron or interval
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(ui): add text input for String general settings so retention periods can be set from the Admin UI
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): record a cleanup schedule attempt only after it did not raise
Records _last_cleanup_schedule_attempt after _reschedule_spend_log_cleanup_job returns, so a transient add_job error is retried on the next config sync while an invalid cron, which is caught and logged inside the reschedule, is still attempted once per settings value
Also adds --num_workers 2 to the dev proxy command in AGENTS.md as requested on the PR
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* docs: revert unrelated AGENTS.md dev command change
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* Revert "docs: revert unrelated AGENTS.md dev command change"
This reverts commit 047706a623.
* Revert "fix(proxy): record a cleanup schedule attempt only after it did not raise"
This reverts commit 678f7c72b4.
* Revert "feat(ui): add text input for String general settings so retention periods can be set from the Admin UI"
This reverts commit 24e49d71d7.
* fix(proxy): record a cleanup schedule attempt only after it did not raise
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): validate the cleanup schedule before swapping the job and leave startup registration to the startup block
_reschedule_spend_log_cleanup_job builds the new trigger first and only touches the live job once it parsed, so an invalid cron or interval (including a non string value) keeps the previous schedule running instead of removing it. An error raised while rescheduling is logged and retried on the next sync, so it no longer stops the rest of the general settings sync. _apply_retention_settings skips the job-missing path while the scheduler is still stopped, so the startup block is the only registration before start and the cross-replica stagger it applies to pending jobs survives
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): skip cleanup rescheduling while the scheduler is stopped and retry a failed replacement
The stopped-scheduler guard only covered the missing-job path, so the first DB sync (which runs before the startup block) still registered the cleanup job whenever the DB schedule differed from yaml, and startup then replaced it. Every runtime path now defers to the startup block while the scheduler is stopped.
A raised add_job that was replacing a live job was never retried because the live job kept wants_job == has_job; the sync now remembers the failure and retries on the next sync until the schedule is applied.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(proxy): drop redundant docstring on _spend_log_cleanup_trigger
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): schedule DB-only retention at boot and log overflowing cleanup intervals once
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(proxy): drop explanatory comment from startup cleanup block
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): reject a non-string cleanup cron at startup and drop legacy covers markers
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): reschedule spend log cleanup when the reload path already applied a DB cron or interval edit
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(proxy): assert cleanup scheduling on a real paused scheduler instead of mock call counts
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: yucheng <yucheng@berri.ai>
* fix(s3_v2): drop terminal upload failures, bound retries per flush and enforce the queue cap at enqueue
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(s3_v2): keep retrying credential-rotation 403s, only AccessDenied-style errors are terminal
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(env_keys): exclude DEFAULT_S3_MAX_FLUSH_ATTEMPTS as an internal tuning var
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(s3_v2): retry every 5xx, warn on first queue overflow, validate the flush budget
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(s3_v2): read the queue cap defensively so un-initialized loggers still enqueue
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(s3_v2): drop the getattr in _enqueue and tighten the retry tests
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(s3_v2): keep the constructor flush budget when the callback override is invalid
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(s3_v2): adapt per-object upload concurrency to sink latency and throttling
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(s3_v2): tidy adaptive limiter
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* perf(s3_v2): wake one waiter per released upload slot
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(s3_v2): make the enqueue queue cap configurable with s3_max_queue_size
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(s3_v2): audit cells for cache hits, coded 403 and callback modes
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(s3_v2): retry bucket-wide failures by default, age-budget requeues and make terminal drops and adaptive concurrency opt-in
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(s3_v2): suppress the missing-waiter ValueError explicitly in the adaptive limiter
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(s3_v2): count oldest events trimmed after a failed flush as callback failures
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* style(s3_v2): move the mutable-ok marker onto the list literal it suppresses
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(s3_v2): report post-flush overflow drops once and grow adaptive concurrency above the floor before asserting back-off
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(s3_v2): fail the SlowDown back-off test when the measured window sees no PUTs
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(s3_v2): restore base retry defaults, opt-in age budget, no enqueue cap, back off outside the limiter slot
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(s3_v2): hoist the default no-op upload slot to a module constant
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(s3_v2): drop unused mutable-ok suppressions on queue appends
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(s3_v2): keep the retry queue oldest-first and prioritise fresh events at upload time
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* style(s3_v2): keep the mutable-ok marker on the queue list literal
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(s3_v2): build request bodies inside the upload slot and keep the sync retry set at base parity
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(s3_v2): drop wall-clock sleeps from the unit tests
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(s3_v2): rebuild the request body inside the slot on every retry attempt
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(s3_v2): anchor the backoff window on the first observed failure and tighten shard assertions
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(s3_v2): default the upload slot to the logger limiter so monkeypatched doubles keep working
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(s3_v2): clear ambient AWS env credentials so the rotating profile signs the sync retry test
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(s3_v2): shrink the linear send-batch perf test to 2k/8k elements
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(s3_v2): fix stale batch sizes in the perf test assert message
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(s3_v2): keep async in-call retries on the base 403/500/503 set
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(s3_v2): hold the upload slot across retries, restore the bool upload contract, and fail safe on bool config
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(s3_v2): mark dropped uploads by element identity so a shared key cannot mask a retryable sibling
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(s3_v2): make the per-flush drop lookup constant time
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(s3_v2): take the upload slot in the caller like base, build the body once per attempt loop
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(s3_v2): match base retry, logging and hook behaviour unless the new options are opted in
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(s3_v2): assert the wire key in the init-bypassed sync upload test
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(s3_v2): assert the signed headers and wire key in the init-bypassed sync upload test
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(s3_v2): default the upload limiter at class level instead of reading it with getattr
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(s3_v2): drop the duplicate annotations that redeclare the class-level counters
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(s3_v2): drop terminal-failed uploads by default and bound retry age to one hour
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(s3_v2): fall back to the configured retry age on invalid values
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(s3_v2): drive retry-age tests from a fixed clock
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(s3_v2): audit cells for retry-age opt-out and 429 single-put parity
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: yucheng <yucheng@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(guardrails): scan retrieved vector store chunks with the request's pre-call guardrails
Vector store retrieval runs inside acompletion after the proxy's pre-call
guardrails have already seen the request, so a retrieved chunk carrying an
injection reached the prompt unscanned. Each retrieved context message now
goes through every pre-call guardrail the request is subject to before it
is injected: a block raises the same 400 the guardrail gives for user text,
a masking guardrail rewrites the context, and a guardrail that fails while
scanning fails the request instead of injecting the chunk unscanned
* fix(guardrails): return a guardrail block unmapped from exception_type so the Responses API surfaces the guardrail's own 400
* fix(guardrails): build the deployment hooks' identity from stamped metadata only
Top-level user_api_key_* fields in a request body are client controlled, so the pre-call, chunk scan, and post-call deployment hooks now take UserAPIKeyAuth from the metadata the proxy stamped, and the chunk scanner returns or raises on every branch.
* fix(guardrails): block route verdicts on retrieved chunks, keep guardrail verdicts out of router retries and fallbacks, and scan chunks against the client's request
* fix(guardrails): keep the merged guardrail list when scanning chunks against the client's request
The scan request laid the client's kept body over the deployment kwargs, so a client
that sent its own top-level guardrails list shadowed the merged metadata.guardrails
list and a key or team guardrail skipped the chunk scan. The kwargs now win and the
keys the proxy relocates into metadata are dropped from the body's contribution.
* fix(guardrails): strip the deployment's guardrail keys from the chunk scan request so merged team guardrails still run
* test(guardrails): type the vector store scan test doubles
* test(guardrails): type the scan double's request data
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* test(logging): drain the logging worker after each logging callback test so no later test inherits its events
* test(logging): run the drain canary in a child interpreter so xdist can never split it
* test(logging): type the drain fixture's ordering parameter and return
* test(logging): record the canary's runs through a queue instead of a mutable probe
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* ci: warn on SQL IN lists with no written bound
Postgres caps a prepared statement at 32,767 bind parameters and a
membership filter binds one per value, so an IN list built from table
data breaks once the table outgrows the cap. That is how the budget reset
job froze every due budget (LIT-7535, #40564).
check_unbounded_in_lists.py reports every Prisma "in" / "not_in" filter
whose value has no fixed size and every raw SQL literal that splices a
list in after "IN (", unless the line carries "# bounded-ok: <reason>".
It only warns for now: the output is the inventory for RCA action item
AI-1, and it exits 0.
* ci: decide a constant IN list by its module binding, not its casing
An ALL_CAPS name imported or filled at runtime is as unbounded as any
other, so a name now passes only when the module binds it once to a
value of fixed size. Adds Final to the locals a loop does not forbid.
* ci: only a frozen module value makes an IN list constant
A module list bound once could still grow through append or extend, so
a name now counts as fixed only when it is bound to a tuple, frozenset
or constant. Trims the module docstring to what a reader needs.
* ci: chunk Prisma IN lists with a shared helper and fail on new unbounded ones
Add litellm.repositories.bounded_in: find_many_in, count_in, update_many_in
and delete_many_in split a deduplicated value list into 5,000-value chunks,
AND each chunk with the caller's where, run them in order (a transaction
handle works) and combine the results. Writes take a required atomicity
argument, and a where that already filters the chunked field is refused.
check_unbounded_in_lists.py now fails CI on any finding missing from
unbounded_in_baseline.txt and on any stale baseline entry, so the baseline
only shrinks. Entries are keyed by path, enclosing scope, kind, field and
occurrence, not line numbers. The helper module is exempt, a constant
spread into a frozen tuple counts as fixed, and messages point at the
helper for "in" and at an array parameter for "not_in" and raw SQL.
A real-Postgres integration test shows a raw 40,000-value filter rejected
for too many bind variables while the helpers handle it.
* refactor: rename bounded_in to chunked_in and let callers pick a chunk size
The helper module is litellm.repositories.chunked_in, and its unit and
integration tests, the checker's exemption path and its finding messages
follow the new name. The `# bounded-ok` marker is unchanged.
find_many_in, count_in, update_many_in and delete_many_in take a
keyword-only chunk_size, defaulting to IN_LIST_CHUNK_SIZE (5,000). A value
below 1 or above MAX_IN_LIST_CHUNK_SIZE (30,000) raises ValueError before
any query, which leaves the rest of the filter headroom under Postgres's
32,767 bind-parameter cap.
* refactor: flatten chunked_in's stacked comprehensions with chain.from_iterable
LIT014 (#42650) caps a comprehension at one for and one if clause. The four nested walks in the helper now chain their iterables instead, with the same order and results.
* refactor: recover user details with find_many_in, sending chunks as lists
_details_for_user_ids reads users through find_many_in instead of a raw
"in" filter, so its lookup stays under the bind-parameter cap for any
number of recovered keys. Up to 5,000 ids it still sends one find_many
with the same where dict, and a PrismaError from any chunk is still
logged and treated as no details.
The helper now sends each chunk as a list, so a chunked filter equals
the dict a hand-written call would send and a migrated call site's
existing assertions keep passing.
The site's baseline entry is gone.
* ci: skip functional TypedDict field maps in the unbounded IN list check
The dict passed as the field map of TypedDict("Name", {...}), or as its fields= keyword, names fields: an "in" or "notIn" key there is a type, not a filter. Only that dict is skipped, for TypedDict, typing.TypedDict and typing_extensions.TypedDict; a filter nested in a field value or passed to any other call is still reported. The two types/proxy/management_endpoints/team_endpoints.py entries leave the baseline, which is now 156.
* fix: refuse an update_many_in whose data writes the chunked field
Chunks run one after another, so an update that sets the chunked field can move a row into a later chunk, which updates it again and counts it twice: values ["old", "new"] with chunk_size=1 and data={"id": "new"} does exactly that. update_many_in now raises ChunkedFieldWriteError before any query when data has the chunked field as a top-level key, in any form, including Prisma operators such as {"set": ...}.
* docs: cut the unbounded IN list checker's docstring to what it flags and how to clear it
It now says what is reported, the three ways to clear a finding, and how the baseline and --update-baseline work, in 11 lines. The per-shape detail lives in the tests.
* ci: key an unbounded IN list finding by its filtered expression too
A baseline key of path, scope, kind, field and occurrence let a PR delete
a baselined filter and add a different unbounded one on the same field in
the same function, and the new one took over the old key. The key now
also carries the filtered expression's source, whitespace-normalized
(the Prisma value, or a raw-SQL `IN (...)` slot), so that swap reads as
one new and one stale entry and fails the run. The same expression
re-added in the same function is still the same finding.
Every baseline entry is rewritten in the new form; the 156 findings are
unchanged, and only occurrence indexes renumber where one field had
several different expressions.
Register Sail (providers.json, LlmProviders.SAIL, OpenAI-compatible lists,
ProviderConfigManager) for chat, Responses and /v1/messages, and add its 12
models to both cost maps with asap, balanced and flex price columns.
Sail picks speed and price with metadata.completion_window and rejects
service_tier, so the Sail chat and Responses configs translate the tier:
default and priority to asap, flex to flex, balanced to balanced, auto to no
window. Billing prices the window that was sent. A tier Sail has no window
for, or a window or tier set where billing cannot see it (request metadata,
extra_body), is a 400 unless drop_params is set.
Add balanced to ServiceTier and its _balanced price columns to the model
info types, the Rust catalog and the dashboard schema. A transform_extra_body
hook on the chat and Responses base configs, which returns extra_body
unchanged by default, lets Sail keep the window when a caller also sends
extra_body.metadata. Sail is listed in the Add Model form and model picker.
Co-authored-by: shrey kharbanda <shrey@berri.ai>
* fix(guardrails): block private destinations in custom code http_request and bound guardrail execution time
* fix(guardrails): keep startup fail-closed on a custom code compile error and report a load timeout on the test endpoint
A compile failure is no longer a ValueError, so a config-file custom code guardrail that does not compile
stops the proxy at startup as it did before, while POST /guardrails catches it by name and still rolls back.
The admin test endpoint reports a module-level timeout as an execution timeout instead of a compile error,
a caller-supplied Host header is stripped from http_* requests while validation is on, and GET keeps the
shared client's connect timeout.
* test(guardrails): cover the http_request methods, header passthrough and cancellation paths
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(responses): run stream failure and success hooks on the iterating loop instead of blocking it
A dropped provider stream on native /v1/responses ran the failure logging through run_async_function from inside the async iterator, which parks the event loop thread on a helper-thread future until every failure callback returns, and never returns when a callback waits on state only that loop can advance. With a running loop the failure handlers (and the completed-stream success deployment hook) are now scheduled as tasks on it, the way chat streaming already does; the sync iterator keeps its blocking path
* fix(responses): await stream failure and success logging on the iterating loop before propagating
Keep the merge-base hook set for the native Responses stream: async_failure_handler plus the executor-thread failure_handler on failure, and the post-call success deployment hook on completion. Inside a running loop the async handler is scheduled as a task on that loop and the async iterator awaits it before re-raising, so the loop is never blocked on a foreign-loop future and the failure is attributed before the router's fallback wrapper re-enters the same logging object. The sync iterator inside a running loop keeps the task fire-and-forget with a strong reference.
* fix(responses): submit the sync failure handler only after the async one finishes
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* feat(proxy): add fail_closed_rate_limit_enforcement to reject requests with 503 while Redis rate limit counters are unreachable
* fix(proxy): reject fail-closed rate limit checks before logging the in-memory fallback and pin the boot warning in the lifespan
* fix(proxy): coerce the fail-closed flag, fail closed on read-only checks, and refund partial cluster increments
* fix(proxy): window-guard rate limit refunds and catch the fail-closed rejection by type
* fix(proxy): read the compaction rate-limit gate's limiter from the proxy hook registry
* fix(proxy): count the pending request in read-only rate-limit checks and keep the compaction gate off the caller's parallel slot
The compaction polyfill's summary-model gate, once it ran against the real v3 limiter, showed two behaviors nobody had chosen. The read-only check compared the stored counter with the same `>` the increment path uses, but a read-only check decides a request that has not been counted yet, so a summary model exactly at its rpm limit still went out. The read-only path now adds the pending increment of 1 before comparing; the increment path is unchanged.
The gate also passed the key's max_parallel_requests gauge through, and the read-only gauge count includes the caller's own in-flight slot, so a key with max_parallel_requests: 1 never compacted. The gate now drops that gauge from its descriptors, since the summary call runs inside a request the limiter already admitted.
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* test(integration): pin the team-admin status-code matrix across every management door
Every endpoint that admits a team admin today is called as a proxy admin, an admin of the
target team, a plain member, an admin of another team and a teamless user, and the current
status code is asserted per actor. The matrix is the parity check for collapsing the five
team-admin helpers into one shared gate and for the later default-off permission flip.
* test(integration): pin the permission-enabled team-admin doors in the gate matrix
Adds three doors that run with team_admin_editable_team_fields granting max_budget, projects and member_key_budgets, so the enabled path is pinned alongside the default-off one. Hoists the ui_settings toggle from test_warmed_policy into the shared client so both files use one helper
* test(integration): grant each permitted door only the permission it needs
A door now names its single grant instead of every permission at once, so a gate that checks the wrong permission for a route turns that door red
* test(integration): rewrite the team-admin matrix rows as request plus expected codes
Each row now names the route it calls and the code each caller gets, and creates the member, key, model, callback or invitation it acts on through plain helpers on the shared team. Drops the Need, Target, World and Door types and the prepare step that seeded fixtures by enum.
* fix(otel): detach post-response service spans by request phase, name redis spans by operation
Service spans logged from the post-response phase (success callbacks, the response-cache write) now root their own trace linked to the request span even while the server span is still recording, instead of only when they happen to end after it. Redis service spans are named `redis <operation>`; the litellm call chain that issued them moves to the `litellm.service.caller` attribute via a typed `ServiceLoggerPayload.caller` field.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(otel): keep the service caller on failure and legacy spans, test the production phase dispatch sites
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(otel): mark anthropic messages stream cache write as post-response phase
The /v1/messages streaming cache writer awaits async_add_cache inline
instead of going through create_cache_write_task, so its redis span
stayed parented under the request trace. Wrap the write in
post_response_phase so it detaches like the chat completions write.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(anthropic): write the Messages stream cache in a background task after handoff
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: yassin <yassin@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(ci): stop five stale or flaky CI reds and retry CyberArk policy-load conflicts
The Langfuse redaction unit test exports to a local OTLP capture instead of
polling Langfuse Cloud through a recorded lookup. The passthrough worker-kill
test only requires spend rows for requests the surviving worker served. The
spend-routes sweep treats the intentional /spend/capture_rate 503 as expected.
CyberArk retries a 409 policy load in Python, Rust and the e2e Conjur helper
instead of reading it as "variable exists". The integration egress guard now
matches the script's own cgroup, so it no longer blocks the CircleCI agent,
which runs as the same user.
* fix(ci): keep the policy-load backoff typed as float
* fix(ci): retry CyberArk policy loads without blocking the event loop and tighten the worker-kill and Langfuse tests
* fix(secrets): load CyberArk policy one request at a time per manager
* test(secrets): pin that non-conflict CyberArk policy failures are not retried
* test(unit): run tests/unit with only an allowlisted host environment
CircleCI's unit job inherits every project env var, so real provider keys,
REDIS_HOST, DATABASE_URL and AWS or Azure credentials reached tests that
assume none are set. Locally, litellm's import-time load_dotenv did the same
from any .env up the tree. The unit conftest now drops every variable outside
a small allowlist and disables dotenv before litellm is imported.
* test(e2e): name a failed search and the stuck batch status instead of misattributing them
The websearch session test read an empty web_search_tool_result_error block as a
successful search, so a failing search tool surfaced as a session billing bug.
The batch cancellation timeout now reports the last status the proxy returned.
* fix(ci): scrub the host environment per unit test instead of for the whole pytest process
GHA shards run tests/unit next to other suites in one process, so the import-time
scrub deleted MCP_TEST_PEER_PYTHON before tests/mcp_tests read it and the MCP
upstream fell back to the SDK2 interpreter. The two websearch tests that called
OpenAI and Perplexity live are removed: tests/unit no longer sees their keys.
* fix(ci): scrub only the host variables present before litellm is imported
The per-test scrub also deleted TIKTOKEN_CACHE_DIR, which litellm sets at import to
its bundled encodings, so tokenizer paths tried to download them and hit the
socket guard. The prisma setup test now passes its own database URL instead of
reading one another test leaked into the process environment.
* fix(ci): stop the order-dependent unit reds and settle logging tasks on their own queue
LoggingWorker marked a task done on whichever queue was current when the callback
finished, so a callback that outlived an event-loop change raised "task_done()
called too many times" or undercounted the new loop's queue. It now settles the
queue the task came from.
The rest are test isolation fixes for failures that only appeared when another
file ran first on the same xdist worker: a replaced user_api_key_cache, breaker
metrics unregistered by prometheus tests, semantic_router's health-check filter on
uvicorn.access, logging tasks carried over from bedrock tests, a Router-written
model_cost entry, and a stray post captured by the langflow test. The token
counter check now asserts bounded chunking instead of wall-clock time.
* test(e2e/ui): wait for the logout redirect before visiting a protected page
Logout revokes the session server-side before clearing cookies and navigating, so an immediate page.goto either ran with the cookie still set or was aborted by the logout redirect (net::ERR_ABORTED).
* test(unit): restore the prometheus metrics config per test and settle logs carried from earlier tests in the a2a cost tests
* test(router): pin the router clock in the usage counter tests so a minute rollover cannot empty the read
* test(e2e/ui): wait for logout to clear the token cookie instead of for a login redirect
* test(integration/mcp): answer the model-info probe another test's proxy sends to the model double
* refactor: clean up fresh tech debt from 2026-09-24 (stacked comprehensions, getattr, bare dict)
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* chore: drop the budget ratchet from the PR branch, the default-branch automation owns it
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(rust): prepare inference and auth foundations
* fix(rust): keep textract operations parsing from kebab-case model names
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* done
* refactor(types): derive Anthropic beta string conversions with Strum
* fix(anthropic): report missing max_tokens as a missing field
* refactor(rust): type Anthropic messages headers and auth after the Python layout
Delete anthropic/messages/headers.rs. Its OAuth handling, credential ladder
and beta merging move to anthropic/common_utils.rs where Python keeps them
(optionally_handle_anthropic_oauth, get_auth_header, _merge_beta_headers),
and the feature beta injection becomes update_headers_with_anthropic_beta on
the messages config, as in Python. The BaseAnthropicMessagesConfig impl is
unchanged apart from the bodies of validate_environment and request_headers
Beta values are now the AnthropicBeta enum and BetaSet, which sort, dedupe
and comma-join by construction. Request params gain typed speed, tools and
context_management through Recognized, so the beta logic matches on enums
instead of string-comparing JSON. OauthToken parses the sk-ant-oat token once
and the chat config shares that detection instead of its own copy
Case-insensitive header helpers move next to has_header in litellm-http.
One deliberate divergence: a Bearer-prefixed OAuth key configured through
api_key or ANTHROPIC_API_KEY is sent with a single Bearer scheme, where
Python would emit "Bearer Bearer"
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* done
* fix(rust): repair test compilation and clippy failures
resolve auth before building the outbound request in prepare tests, give the host hook tests their own error type, and drop the disallowed reqwest client and err().expect() from core tests
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Yujong Lee <yujong@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
* test(integration): run the Langfuse DB-callback test on its own scratch database
The test from #43282 wrote success_callback=langfuse and the LANGFUSE_* env
into the shared integration LiteLLM_Config. The suite's long-running gateway
reloads that table and only ever adds callbacks, so it kept exporting to the
test's closed Langfuse fake for the rest of the shard even after the rows were
restored. The owned proxy now gets a scratch database, which also removes the
snapshot/restore code. scratch_database moves into _support/database.py so
test_cache_and_quota and this test share one copy, and the stock-config guard
now checks the callback settings instead of the raw YAML text.
* test(integration): include failure_callback in the stock-config Langfuse guard
* refactor(rust): prepare inference and auth foundations
* fix(rust): keep textract operations parsing from kebab-case model names
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Yujong Lee <yujong@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test: move key-gated tests/test_litellm SDK tests into tests/llm_translation and drop empty folders
* test: make token counter and health check unit tests run offline
* ci: point unit shards, rust path filter, Makefile and docs at tests/unit
* docs: fix stale test_litellm run paths in moved llm_translation tests
* fix: correct databricks e2e sys.path depth and contributing example path
---------
Co-authored-by: yuneng <yuneng@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(e2e): accept the otel cost write as a linked root trace
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(e2e): window and fail-closed the linked otel trace read-back
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(e2e): poll the otel read-back without recursion
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: yuneng <yuneng@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): port langfuse callbacks-in-db coverage to the local harness
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): assert the langfuse db rows after the exported span
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: yuneng <yuneng@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(callbacks-legacy-python): traverse and release the retained headers dict
LegacyLogging keeps the headers dict it hands to pre_call and post_call, but
its traverse never reported that edge to the collector and close never dropped
it. A cycle a callback builds through that dict could not be collected, and a
closed call kept the dict alive until the driver dropped the whole adapter.
Visit and clear headers like body, with regression tests for both
* refactor(callbacks-legacy-python): move the test support module into its own file
---------
Co-authored-by: Yujong Lee <yujong@berri.ai>
* feat(proxy): email alerts at configured percentages of a team member budget
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(alerting): label team member budget crossings as team member budget
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(auth): cover the team member alert dispatch from _check_team_member_budget
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(email): drop the emoji from the team member budget alert template
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): ignore team member alert thresholds outside 1 to 100 on both the backend and the dashboard
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): bound team member alert threshold key length before int parsing
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): drop the legacy covers marker from the team member alert test
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(team): reject malformed team_member_max_budget_alert_emails on team writes
Thresholds outside 1-100, non-list recipients, and invalid emails now return 422 on
/team/new, /team/update and PATCH /team/{id} instead of being stored and silently
ignored. The value is stored canonically. Read-side LiteLLM_TeamTable is unchanged,
and the PATCH body stays a raw merge patch so a null threshold still deletes it.
* fix(auth): enforce and alert on team member budgets only in common_checks
The builder re-checked the team member budget inline before common_checks ran the same
check, so one request that crossed a team_member_max_budget_alert_emails threshold
dispatched two alerts. Drop the inline check; common_checks is the single authorization
point and already covers per-member rows, the team default member budget, zero-cost
skips and the cross-pod spend counter. Its 422 message now uses the TeamMember=user:team
form the builder and budget reservation already returned.
* Revert "fix(team): reject malformed team_member_max_budget_alert_emails on team writes"
This reverts commit 703e754b46.
* fix(alerting): keep BaseBudgetAlertType.get_event_message zero-arg
Requiring user_info broke existing callers and out-of-tree subclasses. The team member
label now comes from SlackAlerting.budget_alerts, so the interface and its Readme are
unchanged from main.
* fix(mcp): keep team member budget enforcement on the MCP OAuth auth dependency
The MCP OAuth dependency stops at _user_api_key_auth_builder and never reaches common_checks, so removing the builder's inline member budget check would have let over-budget members through there. Enforce it explicitly for that caller.
* fix(auth): keep main's team member budget enforcement, alert once per request
Restore the builder's team member budget check and 422 message exactly as on main and drop the MCP-only gate. The builder sends the member alert only on the request it rejects; common_checks sends it for requests that get past the builder, so no request alerts twice.
* test(integration): read team member alert deliveries without a shared accumulator
* test(integration): match team member alert deliveries by subject so other alerts cannot race the count
* refactor(proxy): build the team member alert threshold config without mutable collections
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(proxy): collapse the alert recipient isinstance checks into one call
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): read the SMTP sink through lock-guarded snapshots and assert the exact deliveries
---------
Co-authored-by: ryan <ryan@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test: load the embedding base image from a committed 100x100 PNG instead of downloading it
* test: move the volcengine embedding test into tests/unit
* test: check gpt2 and r50k_base tokenizer parity against committed tiktoken reference files
* test: check hub tokenizer selection against an in-memory Hugging Face hub
* test: serve image URLs from respx in the gemini tool-result and format-param tests
* ci: drop the emptied legacy core-utils test path
* test: cover the cohere and anthropic tokenizer paths in the hub tokenizer test
* test: fetch every format-param image through respx and check its bytes reach the request
* test: drop the gpt2 and r50k_base parity tests, which no litellm path uses
* test: drop comments that restate assertions in the format-param test
test_update_config_success_callback_normalization replaced
proxy_server.proxy_logging_obj with a MagicMock and never restored it.
Since the proxy unit tests joined tests/unit (#42903), 14 JWT mapping,
end-user and MCP tests on the same xdist worker awaited that mock and
failed. The test now uses monkeypatch.
test_prometheus_logging_callbacks set verbose_logger to DEBUG and
litellm.set_verbose at import, so every worker in the unit job ran with
DEBUG on. That broke caplog equality in the JEV classifier test, the
vertex streaming memory ratio, and four event-loop lag checks. The
module-level setup is removed; nothing in the file depended on it.
#43081 removed the OCR harness modules but left them in the
importability parametrize list.
test_get_model_info_bedrock_region reassigned litellm.model_cost and set
LITELLM_LOCAL_MODEL_COST_MAP without restoring either, and never cleared
the get_model_info caches, so it failed whenever an earlier test had
looked up the regional model. It now uses monkeypatch and invalidates
the caches; the local_testing isolation fixture also invalidates them
after restoring model_cost.
The Windows job hit CircleCI's 10 minute no-output limit while cargo
compiles the Rust crates inside uv sync and uv build. Those two steps now
allow 30 minutes of silence.
* refactor(messages): take the provider client from the injected HTTP pool
The messages route kept its own process-wide reqwest client, so it ignored
ssl_verify, CA bundles, client certs, proxies and every other setting that
litellm-http resolves. The machine now takes the HttpClientPool and the
call's HttpClientConfig, as OCR does, and the bridge passes its shared pool.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* refactor(http): hand out an owned Client and move chat, audio and OIDC onto the pool
HttpClientPool now returns litellm_http::Client, a newtype only crates/http
can build, so every provider client carries the resolved TLS, proxy and
timeout settings. Chat completions and audio transcription drop their
process-wide reqwest clients and take the pool and call config like
messages; their 600s ceiling moves to the request. OidcResolver takes its
client instead of building one, and the bridge hands it the pooled one.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* refactor(secrets): build Google, Azure and CyberArk manager clients from the pool
The native secret managers built bare reqwest clients, so they ignored the
host's TLS and proxy settings. load_native_manager now takes the pool and
the host config and hands each manager a pooled client.
CyberArk's CYBERARK_SSL_VERIFY and CYBERARK_CLIENT_CERT/KEY become an
override on the host config instead of a hand-built client. To express a
certificate and key in separate files, HttpClientConfig::client_certificate
is now a ClientIdentity that is either one PEM or a split pair.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* chore(clippy): only crates/http may build a reqwest client
Fence reqwest::Client, ClientBuilder and the TLS builder methods with
disallowed-types and disallowed-methods so new code takes a
litellm_http::Client from the pool. crates/http is exempt as the one place
clients are built, and testkit as a dev-only installer. Tests move to
litellm_http::Client::plain_for_test or a pooled client.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix(secrets-cyberark): keep verifying certificates when the host disables it
Python hands CyberArk its own ssl_verify, which wins over the global
setting, so CYBERARK_SSL_VERIFY unset or true still verifies even when the
host sets ssl_verify false. The pooled client copied the host's Disabled
and would send the API key unverified; fall back to the built-in roots
instead.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix(python-bridge): treat a missing litellm package as no host HTTP settings
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Yujong Lee <yujong@berri.ai>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(router): hold Responses lifecycle events until output so a pre-output fallback announces one response
* fix(router): narrow the responses wrapper close guards to Exception and test the hold helpers directly
* test(router): type the responses fallback test helpers
* fix(router): replay the held lifecycle events when the fallback stream fails before its first event
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
The legacy callback crate carried two rewrites that have nothing to do with the
Logging contract: litellm_credential_name inheritance and the max_budget and
num_retries_per_request checks. Any later callback host would need them
unchanged, which is the smell the crate's AGENTS.md now names. They are now a
Preflight the driver in litellm-host-python runs on the keyword view begin
returned, before the host projects from it, supplied by python-bridge and passed
through run_legacy_call. The call order is unchanged (setup, deployment hook,
credentials, limits) and a rejection still fails the call as a host failure, so
the failure callbacks run as before. The preflight rewrites the adapter's own
copy in place, so no extra dict copy and no new lifecycle method
Co-authored-by: Yujong Lee <yujong@berri.ai>
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
