_route_streaming_logging_to_handler is called via asyncio.create_task()
after streaming chunks are already delivered to the client. Any
unhandled exception in this logging task propagates as an unhandled
asyncio task exception, polluting error logs.
Wraps the method body in try/except to log errors without propagating.
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
The OpenAI SDK raw_response.parse() can return a plain str instead
of a Pydantic model when Azure returns a non-JSON content type (e.g.,
HTML error page, proxy error). Calling .model_dump() on the str then
raises AttributeError.
Adds isinstance(response, str) checks before all 4 model_dump() call
sites in the Azure chat completion and embedding paths.
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Spend logs are written asynchronously in batches, so a DB lookup for a
just-completed response's spend log races and returns zero rows. Replace the
DB-only fallback with an in-memory session store (_session_history) that is
populated after each response.completed event and consulted at the start of
the next response.create. This makes same-connection multi-turn reliable
without any timing dependency on the DB write queue.
Made-with: Cursor
Also suppress pre-existing pyright type error on model_response.close() call
which is guarded by hasattr at runtime.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Also fix pyrightconfig.json to use the conda venv for type checking, and remove
redundant inline import of ResponsesAPIRequestUtils that was confusing pyright.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
The response.output_item.done handler for function_call type was emitting
finish_reason='tool_calls' and a duplicate tool_call delta. This caused
premature stream termination after the first tool call in multi-tool
scenarios — downstream wrappers (e.g. AnthropicStreamWrapper) would close
the stream before subsequent tool calls arrived.
The response.completed event already inspects the response output list and
emits finish_reason='tool_calls' when function_call items are present, so
output_item.done does not need to (and must not) do so.
This mirrors the existing fix for message-type output_item.done (#17246).
Updated test_function_call_done_emits_is_finished (renamed) to assert
finish_reason=None and no duplicate delta. Updated test_text_plus_tool_calls_sequence
to match. Added test_multi_tool_call_stream_no_premature_finish which exercises
a synthetic 2-tool-call stream and verifies no premature termination.
Renamed test_build_custom_pricing_entry_kwargs_take_precedence_over_model_info
to test_build_custom_pricing_entry_setdefault_does_not_override_existing.
The original test claimed to verify kwargs precedence over model_info but
had no overlapping keys between the two sources. CustomPricingLiteLLMParams
fields and the model_info metadata keys (mode, supports_prompt_caching,
max_tokens) do not currently overlap. Updated the test to verify that
model_info fields are correctly merged, and added an explicit setdefault
assertion demonstrating that pre-existing keys would not be overwritten.
Co-authored-by: openhands <openhands@all-hands.dev>
Previously, register_model() was called with only input_cost_per_token,
output_cost_per_token, and litellm_provider. This dropped ~40+ other
pricing fields from CustomPricingLiteLLMParams (cache_read_input_token_cost,
cache_creation_input_token_cost, output_cost_per_reasoning_token, etc.)
as well as model_info metadata (mode, supports_prompt_caching, max_tokens).
For DB-sourced custom-priced models, the first request after a pod restart
would register a partial entry in litellm.model_cost, causing cost
calculations to miss cache token discounts and other extended pricing
until the entry was later enriched by deployment_callback_on_success
mutating the lru_cache.
Changes:
- Add _build_custom_pricing_entry() helper that iterates over all
CustomPricingLiteLLMParams.model_fields and merges model_info metadata
- Replace hardcoded 3-field dicts in both completion() and embedding()
with the new helper
- Add 7 tests covering field collection, model_info merging, precedence,
None skipping, and end-to-end register_model behavior
Co-authored-by: openhands <openhands@all-hands.dev>
- Parse VIDEO modality in promptTokensDetails → prompt_tokens_details.video_tokens
- Parse VIDEO modality in candidatesTokensDetails → completion_tokens_details.video_tokens
- Parse VIDEO modality in cacheTokensDetails and subtract from prompt video tokens
- Add video_tokens field to PromptTokensDetailsWrapper and CompletionTokensDetailsWrapper
- Fix implicit caching text-token fallback to not fire when cacheTokensDetails is present
- Add 4 unit tests covering: prompt video tokens, response video tokens,
auto-calculated text fallback with video, and explicit video cache subtraction
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Covers:
- Region + modelId correctly extracted for ap-northeast-1, us-east-1, us-west-2
- No region in path leaves modelId and optional_params unchanged
- Cross-region inference prefixes (us., eu., ap.) are not treated as region segments
- Explicitly set aws_region_name is not overridden by region in model path
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
When a user passes model="bedrock/ap-northeast-1/moonshotai.kimi-k2.5", get_llm_provider
strips the "bedrock/" prefix and passes "ap-northeast-1/moonshotai.kimi-k2.5" to the
converse handler. Two bugs occurred:
1. modelId was encoded as "ap-northeast-1%2Fmoonshotai.kimi-k2.5" (region included),
which AWS rejects as "not a valid model identifier"
2. The region ap-northeast-1 was never extracted, so the request went to the wrong
default region instead
Fix: after stripping routing prefixes in converse_handler.py completion(), check if the
remaining path starts with a known AWS region and strip it from modelId, injecting it
into optional_params so _get_aws_region_name picks it up.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Extract the repeated usage-combining block from both
_completion_streaming_iterator and _acompletion_streaming_iterator into a
shared static helper method _combine_fallback_usage. This brings both
functions under the PLR0915 50-statement limit, removing the noqa
suppressions.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add `# noqa: PLR0915` suppression to match the async twin
`_acompletion_streaming_iterator` which already carries the same
suppression. The function's complexity is inherent (nested class,
generator with fallback logic, cleanup code).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The observatory test workflow failed because the "Verify tunnel
connectivity" step used a single curl with no retries. Cloudflare quick
tunnels need time for DNS propagation, and the first lookup can return
NXDOMAIN (curl exit 6). Replace with a retry loop (10 attempts, 5s
apart) matching the pattern already used in the health check step.
Also add `# noqa: PLR0915` to `_completion_streaming_iterator` in
router.py, matching the suppression already on its async twin.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
actions/checkout treats short commit hashes as branch names, causing
fetch failures. The checkout only needs the config file from the
repo, so use the default branch instead of a specific ref.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The secrets context is not available in step-level if: conditions,
causing the workflow file to fail validation. Move the conditional
check into the shell script instead.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Pass AZURE_API_KEY, AZURE_API_BASE, OBSERVATORY_URL,
OBSERVATORY_API_KEY, and REQUEST_ID through step-level env
blocks so they are never interpolated directly into shell scripts.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Validate inputs.tag matches vX.Y.Z format to prevent script
injection via workflow_dispatch
- Pass tag via env var instead of direct interpolation in shell
- Add cleanup step to kill cloudflared and remove docker container
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Add timeout-minutes: 30 to prevent runaway jobs
- Build /run-test payload with jq --arg to safely escape
TUNNEL_URL and LITELLM_MASTER_KEY values
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Fail early if request_id is missing or null from the /run-test
response instead of polling /run-status/null for 15 minutes.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Avoids shell quoting issues with single quotes in JSON and
multi-line output truncation when using GITHUB_OUTPUT.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Add permissions block (contents: read) per GitHub security scan
- Poll /run-status/{request_id} instead of global /queue-status
to avoid race conditions with concurrent test runs
- Add result verification step that fails the workflow if tests
did not pass or the run errored
- Fix auth header to use X-LiteLLM-Observatory-API-Key
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- New reusable workflow that spins up a LiteLLM container from the
release image, exposes it via cloudflared tunnel, and triggers
test runs on the Railway-hosted observatory
- Integrates into ghcr_deploy.yml for RC and stable releases
- Can also be triggered manually via workflow_dispatch
- Add placeholder litellm_config.yaml for observatory test models
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The transformation.py file was using FEATHERLESS_API_KEY (missing _AI_)
while the rest of the codebase (get_llm_provider_logic.py, utils.py)
correctly uses FEATHERLESS_AI_API_KEY. This caused 401 auth errors when
the user set FEATHERLESS_AI_API_KEY as documented.
Now checks FEATHERLESS_AI_API_KEY first (canonical name) with fallback
to FEATHERLESS_API_KEY (legacy compatibility). Same fix applied to
FEATHERLESS_AI_API_BASE.
Refs: #22490
- Add @log_db_metrics decorator to get_jwt_key_mapping_object for
consistent DB latency/error tracking with other helpers
- Move virtual key mapping lookup before auth_builder() to avoid
unnecessary team/user/org DB queries when mapping resolves
- JWT is decoded early; auth_builder only runs when no mapping found
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Persist description field on create (was silently dropped)
- Remove phantom key_alias from JWTKeyMappingResponse (not in schema)
- Populate created_by/updated_by audit fields from authenticated user
- Pass actual jwt_valid_token in admin path instead of empty dict
- Restore hash_token on create and fix duplicate try block
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add page/size query params with take/skip to prevent unbounded queries.
Returns paginated response with total_count, current_page, total_pages.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
