A string above TOKEN_COUNTER_MAX_EXACT_CHARS was counted from its first cap characters and scaled, so a string whose start tokenizes unlike its end got a skewed count, and that count reaches fallback billing when the provider sends no usage. The estimate now tokenizes 16 evenly spaced samples that together total the cap and scales their sum by the string's length, keeping the same bound on work while tracking the whole string
AWS prices Marengo 2.7 and 3.0 text and image embeddings per request, never per
token, and their responses carry no token count. The old transform estimated
prompt tokens from the vector length, which billed a text request at 128 tokens
times the per-token rate (0.00896 instead of 0.00007). Marengo responses now
report zero tokens with query_count and image_count derived from the request
batch, and all six Marengo cost-map entries price per request (with the video
and audio per-second and per-image rates on the base entries). query_count is a
new prompt_tokens_details field wired to input_cost_per_query in the cost
calculator.
* fix(proxy): log disable_budget_reservation notice once at config load
The disabled-budget-reservation reminder fired as a WARNING inside request
authentication, so every authenticated request on a proxy that deliberately
set the flag produced one warning line. The notice now runs once per worker
when general_settings loads, at INFO, and the request path only skips the
reservation. Reservation skipping and read-time budget checks are unchanged
* fix(proxy): keep budget notice sentinel with constants
* fix(proxy): expose shared budget notice state
Add cost map entries for azure_ai/gpt-chat-latest, codex-mini, whisper,
model-router, cohere-command-a, grok-4-20-reasoning, and
grok-4-20-non-reasoning, priced from the live Azure AI Foundry and Azure
OpenAI pricing pages and the Azure Retail Prices API.
Skip the model router flat fee when the response model is the router
entry itself, since the generic cost already priced that fee. Before,
azure_ai/model_router charged it twice.
Resolves LIT-3157
Both proxy token counting endpoints already count in a worker thread, but the
HuggingFace tokenizer's encode holds the GIL for the whole call, so a 600k-token
count on a Claude model still froze the event loop for up to 0.8 s and every
other request with it. Count through encode_batch_fast, which releases the GIL,
and tokenize at most TOKEN_COUNTER_MAX_EXACT_CHARS characters of any one string
(default 4,000,000), scaling the exact count of that prefix by the string's
length above it so the largest payloads stay bounded.
A post_call pipeline now releases the original stream instead of refusing the
request on every shape it has no handler for: a background request, a pipeline
guardrail without the unified apply_guardrail interface, a route with no
endpoint translation, a buffered stream no translation resolves, and a rewrite
the translation cannot write back (tool-call edits, text edits on translations
without write-back, n>1 chat, an unended Anthropic stream, a Responses dump
with no event envelope). Each case logs a warning naming the policy and
guardrail. Real blocks and writable text masks are unchanged.
The revision an operator checks is now the git blob id of the exact bytes the process
loaded, the same id git rev-parse <commit>:model_prices_and_context_window.json prints,
so it is always present, never goes stale between bot writes, and needs no stamp in the
JSON that every PR touching the file would have to regenerate. The _metadata block, the
generated_at field, the schema and guard changes, and the bot stamping are dropped
* fix(bedrock): keep x-amzn-RequestId on chat error responses
Bedrock chat error paths built BedrockError from only a status code and a
message, so the provider response headers were gone before exception mapping
ran and the proxy had nothing to forward. AWS support needs x-amzn-RequestId
to investigate a server-side error.
- converse and invoke chat handlers pass the real headers and response when
they turn an httpx.HTTPStatusError into a BedrockError, and read the body
through error_response_text so a streamed body nobody read does not throw
- every bedrock chat get_error_class honors the headers it is already handed:
invoke, moonshot, bedrock-hosted openai, agentcore and the invoke agent
- BedrockError carries those headers into the response it synthesizes when a
caller has headers but no response, skipping values httpx cannot carry
- the bedrock 500 mapping forwards the provider response like its 4xx and 503
siblings instead of fabricating a blank one
The proxy now returns llm_provider-x-amzn-requestid on Bedrock chat errors.
* fix(bedrock): keep request-id on text-classified errors
The context-window and image branches of _map_bedrock_exception built their
litellm exception without the provider response, so a Bedrock 400 classified
by its body text lost x-amzn-RequestId while the sibling branches kept it.
Also narrows the new BedrockError types and trims its docstrings.
* chore(bedrock): drop the docstrings on the new error helpers
* fix(bedrock): keep request-id on every error path that has one
The ticket's root cause is that every BedrockError raise site under
litellm/llms/bedrock/ was built from status and message alone. The first
commits covered the chat and invoke handlers; this covers the rest.
Embeddings, rerank, image generation, image edit, count tokens, search and
the transformation layers now hand on the provider response or its headers,
and both bedrock_mantle configs return a BedrockError instead of the
OpenAI error that drops them.
Two blockers surfaced while verifying the streaming path. The trailing
`except Exception` in make_call and make_sync_call swallowed the BedrockError
raised a few lines above, relabelling a provider status as a 500, and the
non-200 branch read an unread streamed body, which throws.
The raise sites left alone have no provider response to carry: timeouts,
credential and config errors, and mid-stream event frames.
* fix(bedrock): forward provider headers from the count tokens route
The count tokens route converts BedrockError into an HTTPException, and dropped
the headers the handler had just kept, so that route still lost the request id.
get_response_headers now takes a Mapping so an httpx.Headers can be handed to it
without a copy.
* fix(bedrock): classify every bedrock surface through BedrockError
Eleven bedrock configs still inherited a provider-agnostic get_error_class
that builds a blank response, so the request id was gone before the proxy
read it. Claude platform, bedrock anthropic-messages, both image edit
configs, passthrough, realtime, vector stores and agentcore search now
return BedrockError, and a parametrized audit drives all 36 configs.
* fix(proxy): keep provider headers on the httpx status error branch
_handle_llm_api_exception forwards safe_headers on every branch except the
httpx.HTTPStatusError one, which the bedrock passthrough route reaches, so
the request id was dropped before the client saw the response.
* fix(bedrock): keep the request id on the timeout mappings
Timeout takes no response argument, so the three bedrock timeout branches
dropped the provider headers even when the upstream answered 408 or 504
with an x-amzn-RequestId. They now ride on the exception, already
llm_provider-prefixed, which is the form the proxy emits.
* fix(bedrock): keep the provider response on mapped timeouts
The previous round attached llm_provider-prefixed headers directly to the
Timeout. That shadowed the raw upstream headers for _get_response_headers,
so router cooldown and fallback cooldown stopped honouring retry-after on
bedrock 408/504 replies.
Give Timeout an optional response instead, the way every other mapped
bedrock exception already carries one. Retry logic reads the raw
retry-after off the response, and the proxy prefixes those headers on the
way out, so clients still see llm_provider-x-amzn-requestid.
* chore(bedrock): drop the explanatory comment on Timeout.response
The drop_params validator collapsed every string it did not recognize to None. A pre-fix DB row holds the flag as ciphertext, so a partial PATCH rebuilt the deployment without it and dropped the key from the stored row, and /model/new turned an os.environ/ reference into nothing before the loader could resolve it. The validator now returns the raw value when it is not a boolean flag, the field admits strings the way timeout already does, and the flag set follows pydantic's lax bool parsing instead of a hand-rolled true/false pair
The cost map JSON now carries a top-level `_metadata` block with `generated_at` and `source_revision`, written by the two bot writers only when model data changed. The loader pops it before the map becomes `litellm.model_cost`, records it next to the fetch ETag, and `/reload/model_cost_map`, `/model/cost_map/source`, and the reload schedule status return it. The Price Data Reload card shows the stamp, the ETag, and when the pod loaded the map. The schema and the cost map guard treat `_metadata` as a non-model root key
The rate fields reported base cost-map prices while the cost lines were billed at the token tier, off-peak window and regional multipliers the calculator picks for the request, so a line did not always equal tokens times its reported rate. get_billed_token_rates now resolves the rates once, the token-type breakdown and the endpoint both read from it, and a tiered-model test asserts every line equals its token count times the rate reported next to it
Claude-Session: https://claude.ai/code/session_011Tn3657NkV6ojLqewL64Kb
Take staging's test_bedrock_knowledgebase_hook.py, which drops the duplicate
embedding_executor parameter that turned the lint check red, and make the two
cross-module helpers this branch added public (raise_denied_scoped_mcp_access
and routes_through_gateway) so the private-usage budget stays at its base count
Resolve the conflicts in utils.py, types/router.py, and the tests, and collapse the 56 per-provider isinstance(drop_params, bool) gates to bool(drop_params) now that get_optional_params normalizes the flag once at the top
Encrypt secret maps at the shared persistence boundary, preserve plaintext API/runtime views, and extend rotation and migration scanning to legacy rows.
Co-authored-by: Claude Code <noreply@anthropic.com>
POST /cost/estimate now accepts cache_read_input_tokens,
cache_creation_input_tokens and reasoning_tokens, bills them at the
model's cache and reasoning rates, and reports each share per request,
per day and per month next to the rates it used.
Custom-priced deployments also get cache and reasoning lines in the cost
breakdown now, so the estimate and the spend logs reconcile with their
totals instead of showing zero for those tokens.
Requested by a customer (Pylon #7365).
Claude-Session: https://claude.ai/code/session_011Tn3657NkV6ojLqewL64Kb
