AWS Bedrock pricing publishes a separate 1-hour prompt-cache write rate for
Claude 4.5 / 4.6 / 4.7 (1.6x the 5-minute rate). Without
`cache_creation_input_token_cost_above_1hr`, cost tracking for 1-hour-TTL
prompt caching on Bedrock falls back to the 5-minute rate and undercounts
spend by ~60%.
Adds the field to the spot-checked Global and US-region entries:
- anthropic.claude-opus-4-7 (Global $10.00 / MTok)
- anthropic.claude-opus-4-6-v1 (Global $10.00 / MTok)
- anthropic.claude-opus-4-5-... (Global $10.00 / MTok)
- anthropic.claude-sonnet-4-6 (Global $6.00 / MTok)
- anthropic.claude-sonnet-4-5-... (Global $6.00 / MTok regular,
$12.00 / MTok long-context >200K)
- anthropic.claude-haiku-4-5-... (Global $2.00 / MTok)
- global.anthropic.* mirrors of the above
- us.anthropic.* mirrors at the US +10% premium
Also updates the long-context (>200K) variants of Sonnet 4.5 with
`cache_creation_input_token_cost_above_1hr_above_200k_tokens`.
The mirrored entries in `litellm/model_prices_and_context_window_backup.json`
are updated in lockstep.
EU / AU / APAC / JP / us-gov regional variants are out of scope for this
change pending separate verification against AWS Bedrock pricing for those
regions.
Adds tests/test_litellm/test_bedrock_anthropic_1hr_cache_pricing.py to lock
in the expected values and the 1.6x ratio invariant.
Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
Two issues from the previous push's review:
1. **Greptile P1**: ``get_vector_store_info`` had the same catch-all
``except Exception`` pattern as ``update_vector_store``, so the
HTTPException(403/404) raised by both the in-memory access check and
the new ``_fetch_and_authorize_vector_store`` helper was rewritten as
500. Mirror the ``except HTTPException: raise`` guard from
``update_vector_store``.
2. **code-quality CI** (``tests/code_coverage_tests/recursive_detector.py``)
flagged ``_redact_sensitive_litellm_params`` as an unallowlisted
recursive function. Match the convention of other allowlisted
helpers ("max depth set"): bound recursion at depth 10 (well above
any plausible nesting level for real ``litellm_params`` payloads),
return the redaction sentinel on overflow, and add the function
name to ``IGNORE_FUNCTIONS``.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Three issues surfaced in review of the previous commit:
1. **Veria — Medium**: ``litellm_params`` carries a nested
``litellm_embedding_config`` dict (auto-resolved from the model
registry on create / update) which itself holds ``api_key`` /
``aws_*`` / ``vertex_credentials``. The previous redactor only
inspected top-level keys, so the nested values passed through
unredacted. Recurse into nested dicts.
2. **Greptile — P2**: when ``litellm_params`` is a JSON-serialized
string (the in-memory registry occasionally stores it that way), the
previous redactor silently no-op'd via the ``isinstance(..., dict)``
guard and echoed the raw payload back. Now: parse, redact, re-serialize.
If the string is not valid JSON, replace it with the redaction
sentinel rather than echo it.
3. **mypy** flagged ``_redact_sensitive_litellm_params``'s
``Optional[Dict[str, Any]]`` signature as incompatible with the
``object``-typed call site. Widened to ``Any -> Any`` to reflect the
actual contract (the function now handles dict / str / None / other).
Also fixes a related test regression in
``test_remove_sensitive_info_from_deployment_with_excluded_keys``: the
``"credentials"`` plural addition to ``SensitiveDataMasker`` defaults
caused the first call (without ``excluded_keys``) to mutate the input
dict's ``litellm_credentials_name`` to a masked value. The second call
(with ``excluded_keys``) then saw the already-masked value rather than
the original. Construct fresh input for each call.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
/simplify pass:
- ``update_vector_store`` (newly added) and ``get_vector_store_info``'s
DB-fallback path duplicated the same shape: ``find_unique`` →
``model_dump`` → ``LiteLLM_ManagedVectorStore(**)`` →
``_check_vector_store_access`` → raise 404/403. Extract into
``_fetch_and_authorize_vector_store`` so the pattern lives in one
place; future endpoints that need the same gate get it via one call.
- The ``except HTTPException: raise`` guard added in the prior commit is
retained — the helper raises HTTPException(403/404) and the catch-all
``except Exception`` would otherwise rewrite them as 500.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Two architectural extensions to the credential-redaction in the previous
commit:
1. ``/vector_store/update`` had two gaps:
- No per-store access control. Any authenticated principal that
passed the premium-feature gate could mutate *any* vector store,
including stores belonging to other teams.
- The response returned the full DB row including ``litellm_params``,
so the caller could read another team's persisted provider
credentials by submitting a no-op metadata change.
Mirror the access-control check ``/vector_store/info`` already
performs (``_check_vector_store_access`` against the existing row),
redact ``litellm_params`` in the response, and add an
``except HTTPException: raise`` guard so the 403/404 responses don't
get rewritten as 500 by the catch-all.
2. ``SensitiveDataMasker``'s default ``sensitive_patterns`` set used
segment-exact matching, so ``credential`` matched ``vertex_credential``
but not ``vertex_credentials`` (the actual Vertex field name). The
previous commit worked around this with a per-call extension; this
commit puts the plural in the upstream defaults so every caller
(Redis config dump, MCP debug headers, cache routes, ...) gets the
correct behavior. The local override in
``vector_store_endpoints/management_endpoints.py`` is removed.
Also updates ``test_excluded_keys_exact_match`` which relied on
``credentials`` *not* being a sensitive pattern to demonstrate
case-sensitive ``excluded_keys`` matching. The intent of the test
(case-sensitive match) is preserved; the assertion now reflects that
when ``excluded_keys`` fails to apply (wrong case), the field falls
through to standard pattern-based masking instead of being passed
through unchanged.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
/simplify pass:
- Remove the single-call-site ``_redact_vector_store`` wrapper. Inline
the two-line redaction at its only caller in ``list_vector_stores``;
``get_vector_store_info`` was already calling the inner helper directly.
- Inherit ``SensitiveDataMasker``'s default sensitive-key set instead of
duplicating the 12-element list, then add only the plural
``credentials`` extension. Won't drift if upstream defaults change.
- Trim the over-explained docstring on ``_redact_sensitive_litellm_params``
to a one-paragraph summary; the WHY (credential-leakage class) belongs
in the commit message, not in every consumer's IDE tooltip.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
``LiteLLM_ManagedVectorStore.litellm_params`` carries the upstream provider
credential — OpenAI ``api_key``, AWS ``aws_access_key_id`` /
``aws_secret_access_key``, GCP ``vertex_credentials``, etc. ``GET
/vector_store/list`` and ``POST /vector_store/info`` return these
verbatim to any authenticated principal. Because both routes are in
``openai_routes``, ``RouteChecks.is_llm_api_route`` short-circuits the
standard role gate, so even read-only users and narrowly-scoped keys can
read every stored credential.
Replace credential-bearing values with the ``REDACTED_BY_LITELM``
sentinel in both responses while preserving non-secret keys
(``api_base``, ``region``, ``model``, ``api_version``) so callers can
still see *which* upstream is configured. Detection reuses
``SensitiveDataMasker.is_sensitive_key`` with the default heuristics
plus the plural ``credentials`` pattern (covers Vertex's
``vertex_credentials`` field, which the singular ``credential`` pattern
misses on segment-exact matching).
Applied at:
- ``list_vector_stores`` (``GET /vector_store/list``,
``GET /v1/vector_store/list``)
- ``get_vector_store_info`` (``POST /vector_store/info``), both the
in-memory-registry path and the prisma-DB fallback
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
CI surfaced two issues from the previous commit:
1. ``general_settings`` and ``master_key`` were still imported at the top
of ``get_logging_payload`` but had no remaining users after the
master-key hash-detection blocks were removed. Drop the import.
2. ``tests/proxy_unit_tests/test_user_api_key_auth.py::test_x_litellm_api_key``
and ``tests/proxy_unit_tests/test_key_generate_prisma.py::test_master_key_hashing``
asserted ``valid_token.token == hash_token(master_key)`` — the
pre-alias behavior. The new contract is
``valid_token.token == LITELLM_PROXY_MASTER_KEY_ALIAS`` (and !=
``hash_token(master_key)``), since the master key (and its hash)
must not propagate to the verification-token column or any other
downstream consumer.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Two related changes to how the master-key auth path interacts with
downstream consumers of UserAPIKeyAuth.api_key:
1. The master-key auth branch in user_api_key_auth.py now sets
`valid_token.api_key` to a stable alias
(`LITELLM_PROXY_MASTER_KEY_ALIAS = "litellm_proxy_master_key"`) instead
of the raw master key. Downstream consumers — spend logging,
Prometheus metrics, audit trails, rate limiting, cost tracking — now
receive the alias instead of the master key (which they would
previously hash and propagate). Neither the raw master key nor its
hash flows past the auth layer.
2. `_is_master_key` in spend_tracking_utils.py is reduced to a strict
raw-only constant-time comparison. The hashed form is no longer
considered equivalent.
Side effects:
- The two hash-detection blocks in `get_logging_payload` are removed.
They were re-detecting the master key per spend-log write to swap in
the alias; that detection happens once at the auth layer now.
- The `disable_adding_master_key_hash_to_db` general setting becomes a
no-op. Operators can remove it from their config; existing config is
still accepted.
- Operator dashboards that filter Prometheus metrics by the master-key
hash will need to switch to the `api_key="litellm_proxy_master_key"`
label.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Store search tool allowlists only on object permissions, wire auth/management/UI flows to object_permission.search_tools, and remove legacy team-metadata search credential code and tests.
Made-with: Cursor
Greptile review on #26225 (P2): the docstring said "Called when disconnect()
fails", and the SIGTERM warning log read "after failed disconnect", but
both were stale — `_kill_engine_process` is now invoked on every routine
reconnect (via the unified `recreate_prisma_client` path), not as a
disconnect-failure recovery branch. The misleading wording would have
produced confusing log lines on every reconnect cycle in production.
Update the docstring to explain the actual reason (avoiding the blocking
`disconnect()` event-loop freeze) and reword the SIGTERM warning to "during
reconnect" so it matches reality.
No behavior change; logs only.
Greptile review on #26756 (P2): if `attempt_db_reconnect` itself raises
(e.g. lock cancellation, timer error, unexpected internal failure), the
original `httpx.ReadError` / transport error was lost — `failure_handler`
and `db_exceptions` alerts then logged the reconnect exception instead of
the actual DB transport problem, masking the root cause.
Wrap the reconnect call in a try/except. On reconnect failure, re-raise
the *original* `first_exc` and chain the reconnect error as `__cause__`
so it remains visible for debuggability without becoming the primary
exception observers see.
Adds `test_call_with_db_reconnect_retry_preserves_original_error_when_reconnect_raises`
asserting (a) the propagated exception is the original transport error
and (b) the reconnect exception is attached as `__cause__`.
Two related fixes layered on top of the existing reconnect plumbing:
1. Restore reconnect-and-retry on `PrismaClient.get_generic_data` (issue
#25143). 1.83.x lost the transport-reconnect-and-retry-once branch that
1.82.6 had on this method, so transient `httpx.ReadError` flaps now
surface immediately as `db_exceptions` alerts. `_update_config_from_db`
fans out four concurrent `get_generic_data` reads, so a single transport
blip used to mark four alerts and a stale config window.
Adds `call_with_db_reconnect_retry` to `litellm/proxy/db/exception_handler.py`
— a single canonical "try DB read, on transport error reconnect once and
retry once" wrapper. Mirrors the inline pattern in
`auth_checks._fetch_key_object_from_db_with_reconnect` so we have one
implementation rather than three drifting copies, and gives future read
paths a clean opt-in.
2. Fix the `_engine_confirmed_dead` flag-reset bug in
`_run_reconnect_cycle`. The flag was cleared before `_do_heavy_reconnect()`
ran, so any failure inside the heavy reconnect (timeout, missing
DATABASE_URL, recreate failure) left the flag False — and the next
attempt could silently demote to the lightweight path even though the
engine was genuinely dead. Move the reset into the success branch so the
flag stays True across heavy-reconnect failures and the next attempt
re-enters the heavy branch.
Tests:
- `tests/test_litellm/proxy/db/test_exception_handler_reconnect_retry.py`
(new) — 9 tests covering the helper's contract: happy path, retry on
transport error, no retry on data-layer errors, propagation when reconnect
fails, propagation after second transport error, `hasattr` guard for
partial mocks, fresh-coroutine-per-call invariant, explicit timeout
override, default timeouts read off the prisma_client.
- `tests/test_litellm/proxy/db/test_prisma_self_heal.py` — adds:
- `test_get_generic_data_retries_on_transport_error_for_config_table`
- `test_get_generic_data_propagates_when_reconnect_fails`
- `test_engine_confirmed_dead_persists_across_failed_heavy_reconnect`
(regression test for the flag-reset bug).
All 16 self-heal tests + 9 helper tests + 535 auth/exception-handler tests
pass locally.
- server.py: drop the redundant server_id append in
_get_filtered_mcp_servers_from_mcp_server_names. iter_known_server_prefixes
already yields server_id unconditionally, so the manual append (and its
misleading comment) was a no-op duplicate.
- utils.py: rewrite the SHORT_MCP_TOOL_PREFIX docstring to accurately
describe the collision behaviour. The previous wording said collisions
were 'cosmetic only', but a natural-hash collision IS a routing-correctness
issue, which is precisely why we already added _assign_unique_short_prefix
to rehash deterministically. The new comment cross-references that path.
- utils.py: restrict the first character of the short prefix to [A-Za-z]
via a 52-char alphabet for position 0 only. The remaining two positions
still use the full base62 alphabet. This keeps prefixes valid identifiers
on every backend and gives 52*62*62 = 199_888 distinct prefixes (still
comfortably more than any realistic deployment).
- tests: add coverage proving the first character of the prefix is always
alphabetic across many server_ids and rehash attempts.
Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
The test was flaking on unrelated asyncio ERROR records (e.g. "Unclosed
client session" from background tasks in other tests). Restrict the
assertion to records emitted by LiteLLM loggers so the test only fails
on errors actually produced by the code under test.
Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
Two MCP servers can natural-hash to the same three-character base62
prefix. With 62**3 = 238_328 slots the birthday bound is ~488 servers
for 50% collision probability, so a single proxy hosting more than
~100 MCP servers has a non-trivial chance of seeing a collision in
practice — and a collision means tool names from two different servers
share a routing key, causing silent mis-routing.
Mitigation:
- compute_short_server_prefix(server_id, attempt=N) folds an attempt
counter into the SHA-256 seed, so rehashes are deterministic and
produce a fresh three-char prefix space per attempt.
- New MCPServer.short_prefix field caches the resolved (post-dedup)
prefix on the model so it stays stable across the process lifetime.
- MCPServerManager._assign_unique_short_prefix walks attempts 0..N
until it finds a prefix not already used by another server in the
combined registry. Logs an INFO line when a rehash happens so
operators have a breadcrumb if it ever does.
- Wired into every registration path: load_servers_from_config,
add_server, update_server, reload_servers_from_database. The
database reload path also carries the previously-resolved prefix
forward so reloads don't churn it.
- get_server_prefix prefers the cached short_prefix when set, so the
resolved value (not the raw natural hash) is used everywhere.
- iter_known_server_prefixes yields the cached short_prefix too, so
reverse-lookup tolerance covers the rehashed form.
No-op when LITELLM_USE_SHORT_MCP_TOOL_PREFIX is disabled — the field
stays None and behaviour is unchanged.
Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
* feat: add AIHubMix provider to providers.json
* fix: add aihubmix to provider_endpoints_support.json for CI check
---------
Co-authored-by: yuneng-jiang <yuneng@berri.ai>
`prerelease: false` was hardcoded, so dispatching create-release with
`1.84.0rc1`, `1.84.0.dev42`, or legacy `v1.83.13-nightly` would publish
them as stable releases on the GitHub Releases page. Derive the flag
from the tag instead.
The detector matches `rc`, `.dev`, `nightly`, `alpha`, `beta`. PEP 440
post-releases (`1.84.0.post1`) and legacy `-stable[.patch.N]` are
stable maintenance releases per PEP 440, so they intentionally do not
match.
The tag validator required a leading `v`, so dispatching create-release
with `1.84.0` (or `1.84.0rc1`, `1.84.0.dev42`, `1.84.0.post1`) failed
even though those are the new naming convention. Make the leading `v`
optional in both create-release.yml and create-release-branch.yml so
both legacy (`v1.83.10-stable`, `v1.83.14.rc.1`, `v1.82.3.dev.9`,
`v1.82.3-stable.patch.4`, `v1.83.13-nightly`) and new PEP 440 forms are
accepted during the transition. Refresh the input descriptions to show
the new examples.
Adds LITELLM_USE_SHORT_MCP_TOOL_PREFIX. When enabled, tool / prompt /
resource / resource-template names emitted from MCP servers are prefixed
with a deterministic three-character base62 ID derived from the server's
server_id (SHA-256 → base62) instead of the (potentially long)
alias / server_name. This keeps namespaced tool names well under the
60-character upper bound enforced by some model APIs while still letting
us distinguish MCP-routed tools from local tools.
Behavioural notes:
- Default off — when the env var is unset, the long-prefix behaviour
is unchanged. The plan is to flip the default in a future release
and remove the gate after a deprecation window.
- Prefix derivation is deterministic, so it is stable across processes,
workers and restarts without any persistence layer.
- Reverse-lookup is tolerant: _create_prefixed_tools registers every
known prefix form (alias / server_name / server_id / short ID) in
the routing map and _get_mcp_server_from_tool_name resolves any of
them. Old clients holding cached long-prefixed names continue to
route correctly even after the flag is enabled.
- _get_allowed_mcp_servers_from_mcp_server_names accepts the short
prefix in /mcp/{server_name}-style URLs.
- The OpenAPI tool-listing path now filters by the active server
prefix instead of server.name so spec-backed servers benefit too.
Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
