Milvus REST and Azure AI Search still embedded the query through the SDK, so
a bare Router alias as litellm_embedding_model kept failing after the executor
landed for Valkey. Both now share BaseQueryEmbeddingVectorStoreConfig, which
embeds through the injected executor, drops the empty litellm_embedding_config
requirement, and awaits aembedding on the async path.
The Router executor falls back to the SDK for models the Router does not
serve, so inline provider configs such as azure/text-embedding-3-large with
their own credentials keep working through the proxy.
Tests fake OpenAI and Milvus at the HTTP boundary with respx instead of
patching litellm.embedding.
Carries a mutable-ok suppression on the router session rewrite for the
tightened LIT002 budget, since the realtime callees deep-copy and
JSON-dump the session, and captures the realtime session kwargs through
an async mock in the router tests instead of an untyped dict.
The async client cache is keyed per event loop, and pymongo's AsyncMongoClient
holds a reference to the loop it was built on, so an entry for a closed loop
kept that client and its sockets alive for the life of the process. A script
that calls asyncio.run once per search fills the cache to its cap this way and
then stops caching entirely. Measured live against Atlas over 40 loops: 32
pinned clients and 212 open descriptors before, 1 cached client and no
monotonic descriptor growth after.
parseDynamicAgentForForm recovered a credential field's value from a
stored model string by splitting both the model_template and the model
on "/" and matching by array index. That breaks for any placeholder
value that itself contains "/", such as a Bedrock AgentCore runtime ARN
resource path (runtime/<runtime-id>), silently dropping everything
after the first slash when populating the edit form. Saving without
touching the field then persisted the truncated ARN.
Replace the index-matching split with a non-mutating template parse
(split on the placeholder pattern, escape and rejoin the literal
segments into a regex) so a placeholder captures everything it needs
regardless of embedded slashes. Also add a lightweight ARN-shape
validator for the AgentCore runtime ARN field, guarded against a
malformed pattern string, so a truncated value is rejected client-side
before it reaches the backend.
Resolves LIT-6737
A host-only api_base or MISTRAL_API_BASE (the documented form, https://api.mistral.ai) built
https://api.mistral.ai/audio/speech and 404ed. Match the chat and OCR configs by appending /v1
when the configured base does not already end with it.
litellm.exception_type passes only litellm's own exception types through
untouched, so the NotImplementedError the search-only refusal raised reached
the caller as APIConnectionError. The proxy served that as a 500 with a
traceback in the body for what is a plain client mistake. Raising
BadRequestError gives the caller a 400 and the message on its own.
Two replicas racing prisma migrate deploy can deadlock, and the loser dies
mid CREATE INDEX CONCURRENTLY, leaving the index INVALID. The retried
migration's IF NOT EXISTS then skips it, so the planner never uses it.
After migrations succeed, look for INVALID indexes on LiteLLM tables and
have one replica (advisory try-lock) REINDEX INDEX CONCURRENTLY each of
them, dropping _ccnew/_ccold leftovers of an interrupted rebuild instead.
The repair never blocks startup: a failed rebuild is logged and retried
on the next boot. Also encode DATABASE_URL query values with quote instead
of quote_plus so options=-c%20... reaches psycopg intact.
Sort the chat-tool keys once and number duplicates with groupby instead of
rescanning every preceding key per position, so the guardrail merge stays
O(n log n) on client-supplied tool lists. Drop the comment that restated the
unsupported-tool warning in the Responses-to-chat transformation.
tests/test_litellm/llms/mongodb imports pymongo's exception classes to check the
error translation against the real hierarchy, and the shard that runs it
(tests/test_litellm/llms, per test-unit.yml) synced --extra google, proxy,
semantic-router and saml but not mongodb, so 24 of 109 tests would have errored
with ModuleNotFoundError on the first CI run. CircleCI hid this because it syncs
--all-groups --all-extras.
uv export --frozen ... --extra saml -> no pymongo
uv export --frozen ... --extra saml --extra mongodb -> pymongo==4.17.0
Also close the two gaps a mutation run found in the suite: nothing asserted that
a short request timeout shortens server selection as well as connect, and the
existing code 13 case carried "not authorized", which the message markers match
too, so it could not tell whether the code was still being checked. 28 of 28
mutants now die.
update_in_memory_litellm_params validated mode into GuardrailEventHooks members while __init__ stores the plain strings LitellmParams.mode carries, so readers that stringify event_hook (akto, straiker) saw different values on the serving worker than on re-initialized workers. Presidio forced post_call assignments go through the same shape, and Straiker recomputes configured_modes on every update
* fix(proxy): share per-model budget counters across replicas through the spend counter cache
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(proxy): keep the shared fake Redis store immutable
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: yassin <yassin@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Native AgentCore A2A always sent either a fresh generated runtime session id
or the single configured runtimeSessionId, so related turns lost context and
unrelated callers shared one AgentCore microVM. The runtime session id is now
params.message.contextId scoped to the calling key hash, then runtimeSessionId,
then generated, and is length-validated (33-256) before the header is signed.
Invalid ids surface as JSON-RPC -32602 / HTTP 400 instead of a 500.
Co-authored-by: yassin <yassin@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
The type-discipline and test-quality gates blamed the branch for 4 LIT001, 12
LIT002 and 5 TQ008 violations. Rather than suppress them:
- the $vectorSearch and $project stages are MappingProxyType and the query
vector a tuple, verified against live Atlas to encode identically. The outer
pipeline stays a list because pymongo's common.validate_list raises
"pipeline must be a list, not <class 'tuple'>", which a unit test now pins.
- the client caches are Final[dict[...]] and _client_kwargs returns a
MappingProxyType.
- _field_value recurses over the dotted path instead of rebinding a local.
- _client_key declared Final locals in one branch and reassigned them in the
others, so it is split into an early-returning _timeout_ms.
- the injected callables carry explicit Final[Callable[...]] annotations, which
stops pyright resolving self.embedding_fn against litellm.embedding's
overloads.
- get_sync_client and get_async_client take an optional client_class, so the
cache tests inject a recording double instead of patching the importer, and
can assert the connection string and timeouts the client was built with.
SensitiveDataMasker is public SDK surface, so extra_sensitive_patterns moves to
the end of the signature: in slot two it silently reinterpreted an existing
caller's positional override set as extra sensitive patterns.
When the headroom_retrieve tool is exposed to a client that runs its own
tool-execution loop (the LiteLLM MCP gateway path), the client executes the
retrieve call and sends the recovered original content back as a tool result
on the next turn. The guardrail then compressed that row again, and because
CCR is content-addressed it collapsed back to the exact same hash it was just
retrieved from. The model never saw the expansion and the agent looped.
Hold tool-result rows that carry headroom_retrieve output back from the
compression service, the same way the live turn and trailing tool exchange are
already protected, so the expansion survives. Retrieve calls are matched by the
direct headroom_retrieve name and the mcp__<server>__headroom_retrieve gateway
name. Because a long gateway name is truncated past 64 chars in the
OpenAI-translated view the guardrail scans, the pairing also falls back to the
tool-call id read from the request's own untranslated messages, which is never
truncated.
Fixes#38558
