The config stub shared one list object between save_config and get_config, so the
DB overlay handed the endpoint back the very list it had just appended to and both
tests passed with the product fix reverted. Store the settings as JSON the way the
litellm_config row does, and check the duplicate guard against a list that only
ever existed in the DB.
The monitor reads the queue on its first pass, before it ever waits on a
request, so dropping a request made while spend_log_flush_requested is still
None delays nothing. Reordering the loop to wait first would turn that drop
into a real delay for the Responses chaining flow, and now fails this test.
`POST /v1/agents/{id}/make_public` appended the agent id to
`litellm.public_agent_groups` and only then called `get_config()`, which
re-applies the DB's `litellm_settings` over the module globals and threw the
append away. The config it saved was therefore a no-op: the endpoint answered
200 with an empty `public_agent_groups`, the agent never reached
`GET /public/agent_hub`, and re-publishing never hit the "already public" 400.
Read the config first, derive the new list from the refreshed globals, save it,
then update the global
Also fixes the e2e model hub spec, which is flaky for a second reason: the
"Make Models Public" modal preselects the groups that are already public, so a
blind click on "Select All" cleared them and left "Next" disabled for the full
15s action timeout. Check the box instead of toggling it, and wait for "Next"
to be enabled before clicking
Covers the reintroduction of a second module-level cache for the guardrail
translation mappings: remapping the loader between two pre-call hooks must
change which handler runs, and the module must expose no assignable map of
its own.
Streaming /v1/messages against a model served through the chat-completions
bridge (every non-Anthropic provider other than OpenAI) minted its msg_ id
inside the stream wrapper, so the spend row landed under the provider's own
completion id and the caller could not find the call by the only id it saw.
The wrapper now mints the id once in its constructor and hands it to the
logging object, the same way the Responses-API bridge does.
`PrismaClient.spend_log_flush_requested` was an `asyncio.Event` built at
import time, so it bound to whichever event loop first awaited it and every
later loop got `RuntimeError: ... is bound to a different event loop` out of
`_wait_for_spend_log_flush_request`. The queue monitor's blanket `except
Exception` swallowed that into its error logger, so the flush silently never
happened and the row sat in the worker's queue until the next poll.
The monitor now creates its own Event inside the loop that awaits it and
hands it to the client, and `request_spend_log_flush` signals through the
client instead of the class. A request that arrives before the monitor is
running is dropped and loses nothing, because the monitor reads the queue on
its first pass before it ever waits.
In CI this showed up as the proxy-endpoints shard flaking on
test_monitor_spend_logs_queue_flushes_as_soon_as_one_is_requested whenever
--dist=loadscope put the health-endpoint tests, which boot a proxy TestClient
and start a monitor, on the same worker ahead of the spend-log tests.
TestStreamingScanDedup restored the reduced module-level translation
mapping on teardown via monkeypatch, so under --dist=loadscope the
worker that ran only that class carried the reduced mapping into the
streaming block test modules. Tag routing tests now assert the eligible
deployment set directly instead of sampling ten random picks. The
liveliness latency check measures steady-state polls after a warm-up
request rather than the first request through a fresh app.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
The documented DISABLE_AIOHTTP_TRANSPORT env var already selects the httpx transport, so the extra module-global write was redundant. Types the monkeypatch fixture while here.
The unified guardrail cached the endpoint translation mappings in its own
module global on top of the loader's cache in litellm/llms. Tests wrote to
that second copy directly, so a teardown that restored a stale snapshot left
a test double installed for every later test on the same xdist worker, and
proxy-endpoints went red on whichever guardrail streaming test happened to
land after it.
Read through load_guardrail_translation_mappings() at each call site and give
the tests one seam to patch, so pytest owns every restore.
Failing closed at the sensitive-data masker's depth of 10 turned an ordinary
nested tool JSON schema into REDACTED leaves, because a list level costs two
depth. The walk now bounds on DEFAULT_MAX_RECURSE_DEPTH, which no real payload
reaches, and the masker's own limit is left alone.
A streaming /v1/messages call against a non-Anthropic model is served an SSE
message_start frame carrying a msg_ id the adapter mints locally, since the
Responses API upstream only issues a resp_ id. That value never left the
adapter, so the spend row was keyed on the bridged response id and
GET /spend/logs?request_id=msg_... came back empty.
The adapter now hands the id it minted to the logging object, and the
/v1/messages logging path keys the row on it.
The "Final returned optional params" line printed whatever the caller nested inside
extra_body, so a credential tucked in there reached stdout in plaintext one line after
the request line that already redacts it.
The call site now runs redact_credentials_in_payload behind a guard reading both of
print_verbose's consumers, litellm.set_verbose and the LiteLLM logger's DEBUG level, so
the line prints in exactly the cases it did before and the walk costs nothing when
nothing would read it.
The proxy's exception tails defaulted `type` and `param` to the four-character
string "None", which is neither a known OpenAI error type nor the JSON null the
nullable `param` field is typed as, so a client's error handler matched nothing
and fell into its generic branch.
Lifts the helpers PR #39521 added for the unified LLM endpoints into
litellm/proxy/common_utils/openai_error_payload.py and calls them from the file,
rerank, image, realtime, anthropic, and pass-through route families, plus the
shared handle_exception_on_proxy handler that the management, batches,
fine-tuning, credential, SCIM, guardrail, and customer routes funnel through.
The remaining families (proxy_server, auth, health, spend tracking, and
management endpoints) follow in separate PRs so each slice stays QA'able on a
live proxy.
Background streaming emits event types with no typed model, which arrive
as GenericEvent holding a plain dict. Only typed events had their nested
response id rewritten, so those frames advertised the raw internal id
while their siblings advertised the encrypted one. The raw shape skips
the ownership check, so any other key could retrieve or cancel that
response.
Rewrite the advertised id wherever an event carries one, whichever shape
it arrives in, so a future event type cannot reopen this.
redact_credentials_in_payload only recursed into mappings, so a
credential-named key one level inside a list or tuple, the shape
extra_body and metadata routinely carry, still reached stdout under
set_verbose. Rebuild sequences element by element too, keeping the
container's own type so the printed repr is unchanged apart from the
secret.
`litellm.set_verbose = True` printed the caller's kwargs verbatim to stdout, so
`api_key` and its siblings landed in terminals and container log drains in
plaintext while the same statement's logger emission was already redacted.
Mask the kwargs at the source with a shared helper in
`litellm_core_utils/sensitive_data_masker.py`, reusing the existing
`SensitiveDataMasker` key classification and the `REDACTED` marker
`secret_redaction.py` already owns, so both debug surfaces agree.
The OCI wrapper overrides chunk_creator wholesale, so it never recorded the finish reason or marked the terminal chunk as sent. The shared end-of-stream finalizer then appended a synthetic chunk whose finish_reason was always stop, which downgraded a tool_calls completion for any client that reads the finish reason off the last chunk.
A blocked guardrail (and any other HTTP error the proxy converts) came back
with "type": "None" and "param": "None", because the converters passed the
string "None" as the getattr default instead of None. OpenAI types error.type
as a required string and error.param as nullable, so type now falls back to
the type its status code stands for and param serializes as JSON null.
Covers the non-streaming body, the SSE error frame, the client-disconnect
frame, and the unclassified-exception path, so every unified LLM endpoint and
the anthropic endpoints return the same shape.
litellm_settings keys are set on the litellm module with no allowlist, so a typo
in vector_store_search_failure_mode reached assert_never and turned every
vector-store request into a 500. Validate the configured value and fall back to
the permissive default with a warning naming the supported modes.
AmazonMoonshotConfig.transform_request called
_get_boto_credentials_from_optional_params purely for its side effect of
popping the aws_* keys off optional_params, then threw the result away. On
a box whose default AWS profile uses login_session without botocore[crt],
that call raises, so a bearer-token bedrock/invoke/moonshot.* deployment
still 500s with MissingDependencyException even after the rest of this
branch skips the chain.
It now filters the aws_* keys into a local dict the way the Qwen, OpenAI
and Claude 3 invoke transformations already do, so no credentials are
resolved and the caller's optional_params keeps the keys sign_request
reads afterwards.
/v1/messages against a non-Anthropic model answers with the Responses id,
but the spend row was built from a fresh ModelResponse, so it landed on a
chatcmpl- uuid nobody can look up. Carry that id through the same way the
Anthropic branch now does, and make the passthrough spend assertions fail
on an empty lookup instead of skipping past it.
A vector store search that fails is swallowed by the pre-call hook, so the
request goes to the model with an un-augmented prompt and the caller gets a
200 answering from the model's own knowledge with no way to tell the
knowledge base was skipped.
Failed searches now ride the same channel their successes already use: a
vector_store_search_failures entry on provider_specific_fields naming the
store id, provider, and error. That is additive and always on. For callers
who would rather fail than answer ungrounded, litellm_settings
vector_store_search_failure_mode: error raises VectorStoreSearchError (400)
instead; the default stays annotate, today's permissive behavior.
The hook's outer catch-all also now names the requested vector store ids in
its log line, and only wraps the augmentation itself, so the fail-closed
raise is not swallowed by it.
The /v1/images/edits handler binds the documented image[] and mask[] aliases
into their canonical parameters, then re-reads the multipart body, so the raw
bracketed keys rode along to the provider next to the values already built from
them. OpenAI rejected both: image[] as "Invalid type for 'image[0]'" and mask[]
as "Invalid parameter: 'mask'". Drop both aliases from what gets forwarded.
Review turned up two real problems in the TTS path.
Router.aspeech forwarded voice=None whenever the caller omitted it, which overwrote a
voice set in the deployment's litellm_params, so a configured fallback voice was
ignored on voice-less requests. It now leaves the key alone when no voice is passed.
get_complete_url also fell back to MISTRAL_API_BASE, but speech() always receives a
non-null api_base from get_llm_provider, whose mistral branch only reads
MISTRAL_AZURE_API_BASE and otherwise hardcodes the public host. That branch could
never run, and its unit test asserted a behavior the real path does not have. The
working override is api_base on the deployment, now pinned by an end-to-end test
Address the two Greptile P2 notes: construct the chunk through the shared
creator instead of mutating its choices afterwards, and drop the decorative
divider comment from the new tests.
POST /v1/messages returns an Anthropic-shaped body whose `id` is the only
request id the caller ever sees, but the spend row was written with a
`chatcmpl-<uuid>` (non-streaming) or the bare `litellm_call_id` (streaming and
the /anthropic/v1/messages passthrough), so
GET /spend/logs?request_id=msg_... returned [].
The logging conversion now carries the provider's response id through:
_handle_anthropic_messages_response_logging seeds the ModelResponse it builds
with the Anthropic id, and the passthrough logging handler prefers the id it
read off the response body or the message_start chunk over litellm_call_id.
get_spend_logs_id already prefers response_obj["id"], so the spend row and
standard_logging_object["id"] now both carry the id the client holds.
Every field of a multipart form arrives as a string, so `n` reached the
provider as "2" and Bedrock Nova Canvas rejected the request with
"expected type: Number, found: String". Restore the type the request
schema declares at the boundary where the form is parsed, driven by the
schema's own type hints so the helper covers any int- or float-typed
field on any multipart endpoint.
Resolve type-discipline-budget.json by taking the lower limit per rule so no
ceiling ratchets back up.
Staging's 66a3d24b3f left a duplicate embedding_executor parameter in the
Bedrock KB fake handler, which makes ruff fail on the whole tests tree. Drop
the duplicate here so this branch compiles; #39502 makes the same change on
staging.
Resolves two conflicts:
- tests/test_litellm/vector_stores/test_main.py: staging moved search() to a
RouterVectorStoreEmbeddingExecutor while this branch parametrized the same
test over query; keep both the executor assertions and the parametrize.
- tests/logging_callback_tests/test_bedrock_knowledgebase_hook.py: staging
carries a duplicate embedding_executor kwarg that makes the file a
SyntaxError; drop the trailing duplicate.
