Gap in this PR's own fix, found by auditing the other call paths that reach
_record_input_audio_sample_rate.
The method read the rate only from the GA shape (audio.input.format.rate), on
the stated reasoning that "the beta shape's input_audio_format is a bare codec
name with no rate". The bare codec name does carry a rate. LiteLLM's own type
stub says so:
"The format of input audio. Options are pcm16, g711_ulaw, or g711_alaw. For
pcm16, input audio must be 16-bit PCM at a 24kHz sample rate, single channel
(mono), and little-endian byte order."
OpenAIRealtimeSession.input_audio_format, litellm/types/llms/openai.py:1898
And RealTimeStreaming._remap_beta_session_to_ga already expands it to exactly
that, mapping "pcm16" to {"type": "audio/pcm", "rate": 24000} via
_AUDIO_FORMAT_MAP (realtime_streaming.py:219). That remap runs on every
client session.update when the backend is in GA mode
(realtime_streaming.py:1501), before the payload reaches this config.
So the declared rate for one client payload depended on the OpenAI-Beta header,
which says nothing about sample rates:
no header -> remap runs -> audio/pcm;rate=24000
header set -> remap skipped -> audio/pcm;rate=16000
Same bytes, two labels, and the server resamples against whichever it gets.
The rate is now also read from the flat beta codec name, so both routes agree
at 24000 for pcm16. A rate stated outright in the GA shape still wins; the
name-implied one is only the fallback. Only pcm16 is mapped, because
get_audio_mime_type labels every append as pcm16 regardless, so a rate lifted
from a g711 name would describe bytes with a codec they are not in.
The existing coverage could not catch this: it built the GA dict by hand and
fed it straight to the config, sharing the same assumption as the code under
test. The new test drives the real _remap_beta_session_to_ga instead and
asserts both routes land on the same rate. Verified against the pre-fix source:
assert 'audio/pcm;rate=24000' == 'audio/pcm;rate=16000'
3 tests added, the two new beta ones verified to fail before this change. The
beta case is removed from the "keeps the native default" parametrize list,
where it asserted the 16kHz the remapped path never produces, and replaced
with g711_ulaw.
tests/test_litellm/llms/{gemini,vertex_ai}/realtime plus
tests/test_litellm/litellm_core_utils/test_realtime_streaming.py: 224 passed.
Both from the automated review on #40617, both real.
Vertex never recorded the declared rate. VertexAIRealtimeConfig handles
session.update in its own transform_realtime_request and returns without
reaching the parent's _handle_session_update, so the recording call added in the
previous commit never ran on that path and a Vertex client's declaration was
silently discarded. Recorded in the Vertex branch too, before the
first-setup/subsequent-setup split, matching where the parent records it.
A later declaration could reprice audio already sent. The estimate accumulated
raw bytes and divided by the rate current at consume time, so a client could
stream at 16kHz and then declare 24kHz before the estimate was read, billing two
thirds of what it actually sent while the backend processed all of it. Audio is
now converted to seconds at append time, at the rate in force when the chunk
arrived, so a mid-stream redeclaration cannot reach backwards. Mixed-rate
sessions bill each chunk at its own rate.
Also bounded the accepted rate to 8000-48000 Hz. The value is client-controlled
and now feeds the spend estimate directly; an unclamped declaration of 100 MHz
would bill a long session as a few milliseconds. Out-of-range declarations are
ignored with a warning and the native default stands.
6 tests added or extended, each verified to fail without these changes.
tests/test_litellm/llms/{gemini,vertex_ai}/realtime: 103 passed.
Fixes#40563.
Both GeminiRealtimeConfig and VertexAIRealtimeConfig hardcoded the pcm16 input
MIME type to audio/pcm;rate=24000. 24kHz is the Live API's *output* rate. Its
documented native *input* rate is 16kHz, and the MIME rate is the only channel
the caller has for telling the server what it is actually sending:
"Audio output always uses a sample rate of 24kHz. Input audio is natively
16kHz ... To convey the sample rate of input audio, set the MIME type of each
audio-containing Blob to a value like audio/pcm;rate=16000."
https://ai.google.dev/gemini-api/docs/live-api/capabilities
Because the server resamples against whatever the MIME type claims, a client
streaming correct 16kHz PCM16 had it relabeled as 24kHz, which corrupts it
server-side and degrades transcription with no error anywhere.
Three changes:
1. The rate now comes from what the client declared. session.update carries it
in the GA shape at audio.input.format.rate, so that value is recorded and
used for every subsequent blob. The rate-less beta shape (input_audio_format
is a bare codec name), a missing or malformed rate, and a bool (an int
subclass, so excluded explicitly) all leave the default alone.
2. That default is now 16000, the documented native input rate, instead of the
output rate.
3. VertexAIRealtimeConfig's byte-identical copy of get_audio_mime_type is
deleted so it inherits the parent. The duplicate is why patching the parent
alone had no effect on the Vertex path, which is the trap the report calls
out; a test now asserts the override stays gone.
The billed audio duration reads the same rate, so the label and the duration
estimate cannot disagree. PCM16_INPUT_AUDIO_BYTES_PER_SECOND (48000, that is
24kHz x 2 bytes) is replaced by the declared rate x PCM16_BYTES_PER_SAMPLE.
This does move the estimate for a transcribe-live caller who declares no rate:
the same byte count is now billed as 1.5x the duration, because 16kHz audio
takes 1.5x as long to send as the 24kHz the old constant assumed. The two
existing estimate tests are updated for that, and a new test covers a caller
that declares 24kHz and still bills at the old numbers.
14 tests added or updated, each verified to fail against unpatched sources.
tests/test_litellm/llms/{gemini,vertex_ai}/realtime: 97 passed.
* fix(params): carry stream_chunk_size through litellm_params instead of provider params
* test(integration): fence stream_chunk_size out of every provider request body
* test(bedrock): type parametrized stream chunk test params
* test(integration): drop the contracts manifest resurrected by the main merge
* test(bedrock): type the stream_chunk_size test helpers
* test(params): finish AGENTS.md typing pass on stream_chunk_size tests
* test(integration): drop the covers marker from the stream_chunk_size wire test
---------
Co-authored-by: shrey kharbanda <shreshth@berri.ai>
* fix(bedrock): send json_schema as a forced tool on Claude Opus 4.7 and 4.8 Converse
Bedrock rejects outputConfig.textFormat on Opus 4.7 and 4.8 with
"output_config.format: Extra inputs are not permitted", and the AWS
model cards list structured outputs as not supported for both, so
their cost-map entries no longer claim supports_native_structured_output
and json_schema requests fall back to the json_tool_call tool.
Fixes#27846
* test(bedrock): assert Opus 4.7 and 4.8 inline the schema on Invoke, move the native case to Sonnet 4.6
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* test: point CircleCI-only suites at models still in the cost map
#42435 removed cost map entries past their deprecation date and #42437 added
litellm_uisettings to the config-synced tables, but both only updated
tests/test_litellm. The CircleCI-only suites (local_testing, llm_translation,
logging_callback_tests, litellm_utils_tests, unit) kept using the removed
models or the old table list and went red on main.
Each test keeps its assertions and swaps the removed model for a current one
with the same provider and capabilities. The fireworks tests pick a vision
model from the cost map because #34941 set supports_vision false on
minimax-m3, and the vertex image provider test injects the image model set
because #42435 removed every vertex_ai-image-models entry.
* test(vertex_ai): register the image model through add_known_models in the provider test
* chore(cost-map): remove models past their deprecation date
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(cost-calc): drop the empty parametrize left behind by the gemini web search removal
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(cost-map): drop merge base block left by conflict resolution
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(cost-calc): drop gemini image cost tests pinned on removed model
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: kerry <kerry@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(cost): bill batch prompts above 272K at OpenAI's long-context batch tier
* fix(cost): mirror batch long-context keys on custom pricing params
Register the two *_above_272k_tokens_batches keys on CustomPricingLiteLLMParams so a per-deployment override stays out of the shared backend key, add them to the inline model-info schema and alias-count tests, and build LiteLLM_Params and GenericLiteLLMParams through model_validate at the two dict-splat call sites so basedpyright's reportArgumentType budget ratchets down instead of blocking the new fields.
* fix(cost): add the gpt-5.5-pro batch long-context tier and ignore malformed batch tier keys
* fix(cost): bill cached batch tokens at OpenAI's cached batch rate
Adds cache_read_input_token_cost_batches and
cache_read_input_token_cost_above_272k_tokens_batches for the tiered
OpenAI entries at half the standard cached rate, bills cached batch
tokens at that rate per output line, and parses string-valued batch
rates in deployment-level model_info.
* fix(cost): bill batch cache writes at the batch cache-write rate and carry published batch rates for one-sided deployments
OpenAI's Batch table prices cache writes for gpt-6-astra, gpt-5.6, gpt-5.6-sol, gpt-5.6-terra and gpt-5.6-luna at half the standard cache-write rate, so the cost map gains cache_creation_input_token_cost_batches and its above_272k tier for those entries and batch cost pulls written tokens out of the input bucket at that rate; models without the key keep billing writes at the batch input rate.
A deployment declaring only one side of its batch pricing now carries every published batch rate of the other side (tier, cached, cache write), its own keys win, and a lone tier, cached or cache-write batch key counts as declared pricing instead of being ignored.
* fix(cost): select the batch long-context tier from any batch tier key
A deployment that declares its own flat standard input rate keeps every
published batch rate of the output direction, including the 272K output
tier, but the tier was only ever selected when an input tier key was also
present. Detect the crossed tier from any of the four batch tier keys so
the carried output, cache-read, and cache-write tiers bill at their tier
rate above 272K tokens.
* chore(proxy): keep the OpenAPI snapshot as CI generates it
* fix(cost): pick each batch price component's tier from its own keys
The batch rate picker crossed one threshold for every component, so a
deployment declaring only an output tier also moved its input, cached, and
cache-write rates to that cutoff. Each component now crosses its own
*_above_<N>k_tokens_batches keys and falls back to its flat key.
The JSON schema is regenerated with the generator as it is on main:
cost-map-guard renders the PR's cost map with the base branch's generator,
so the descriptions for the new batch cache keys move to a follow-up.
* chore(proxy): restore the lazy OpenAPI snapshot to what CI's Python 3.12 generates
The merge commit carried a snapshot regenerated on a Python 3.14 venv, which dedents
docstrings at compile time, so one description line differed from the file CI regenerates
on 3.12 and the schema.d.ts sync check went red. The snapshot is byte-identical to main again
* test(unit): make bedrock collector and secret scan timing tests deterministic
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(unit): count interpreter calls instead of wall clock in the secret scan scaling test
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(unit): profile the secret scan with cProfile, restore the outer profiler and tighten the scaling bound
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: kerry <kerry@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(cost): honor per-second custom pricing on chat completions for every provider
* test(cost): assert a per-second priced deployment bills instead of staying at $0
The zero-cost diagnostic tests from #42345 used a per-second-only entry as their
misconfigured fixture, which this branch now bills. Switch that fixture to a
per-query-only entry, which is still selected as the deployment's own pricing and
still prices chat usage at $0, and add a per-second test asserting the call
duration is billed with no diagnostic
Also let a caller's explicit total_time outrank the logging window in
completion_cost, so the SDK precedence stays stamped response, caller, logging
* test(response_metadata): move the per-second pricing regression into the mapped tests/unit file
* fix(cost_calculator): keep media-mode per-second rates off the wall-clock path
A video, transcription, speech, or realtime entry's per-second rates price media seconds, which
their dedicated cost paths bill from the media itself. The generic per-second branch now skips
those modes, so a video status poll on a per-second video model bills nothing instead of the
seconds the poll took to answer.
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* docs(rust): plan Python interop foundation
* fix(rust): preserve Python settings coercion at the native boundary
* chore(rust): drop interop planning note
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(rust): resolve OCR provider secrets through an async SecretSource before transformation
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(rust): project the Python secret manager into the bridge and resolve OCR secrets through it
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(rust): drop premium_user from the secret manager snapshot
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(rust-bridge): read the private key management globals once in the settings snapshot
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(rust): bound the bridge secret manager state cache to the active snapshot
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(rust): inline coercion unit tests
* fix(rust): preserve Python secret manager bindings
* refactor(rust-bridge): let settings projectors own their contract specs
Each settings group now declares its SettingSpec rows next to the projector
that reads them, and the manifest test derives python_settings.json from those
tables instead of a hand-copied duplicate. Field carries (group, name) instead
of a dotted path, and coercion gains the dict-item reader plus the Redis
Boolean, certificate-requirement, non-empty string, and numeric adapters that
the cache configuration projection adopts next.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* refactor(rust-bridge): capture the secret manager binding in one settings read
The secret_manager accessor now carries the live client and settings objects,
so the bridge classifies the binding from a single snapshot instead of
re-reading litellm globals. The unreachable native arm and the service alias
go away, the binding-to-state mapping moves next to the snapshot, and the
Python callback precomputes its key_manager name.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* refactor(rust-bridge): execute typed settings field declarations
* refactor(rust-bridge): compare cache backends by identity behind one exact trait
cache-response gains an object-safe ExactResponseCache so every exact-match
backend sits behind one pointer; WriteBuffer flushes through it. The bridge's
NativeResponseCache shrinks from nine variants and fifteen per-backend
accessors to an exact service plus the three semantic backends, and facade
mismatch detection compares BackendIdentity values instead of matching on
each backend type. Request projections move next to NativeRequest.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* refactor(rust-bridge): drive both Python-embedded semantic caches through one execution
Redis-semantic and Valkey-semantic operations now share one SemanticExecution
body: await the Python embedder, seed the task-local vector, run the native
backend, repeat per batch entry. Valkey drops its with_embedder path in favor
of the same seeded embedder, and each backend keeps its own embedding-failure
policy. PythonEmbedder exposes one call shape. Redis-semantic thresholds are
compared at the backend's f32 width, which un-breaks the redis-stack parity
tests that a 0.8 facade threshold failed before this branch.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* wip
* feat(rust-bridge): complete response cache runtime surface
* fix(rust-bridge): preserve secret manager callback exceptions
* refactor(rust-bridge): unify route cache and secret rollout catalog
---------
Co-authored-by: Yujong Lee <yujong@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
* fix(bedrock): sign batch S3 requests with s3_access_key_id and s3_secret_access_key
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(bedrock): keep S3 signer test additions scoped to new cases
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(bedrock): drop e2e suite changes from the S3 signing fix
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(bedrock): build S3 credentials directly from the s3_* pair so ambient AWS_* env never mixes in
Restores the split-identity e2e coverage
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: yucheng <yucheng@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Cherry-pick of merge commit b3882d8e43 (PRs #39321, #39562, #40107), which landed on litellm_internal_staging instead of main.
Co-authored-by: ojensen-berri <ojensen@berri.ai>
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
