Fixes#26083
vertex_ai/google/gemma-4-26b-a4b-it-maas previously fell through to the
NON_GEMINI route. Per owtaylor's plan on #26083: add the google/gemma-
prefix to PartnerModelPrefixes so is_vertex_partner_model picks it up
and should_use_openai_handler routes it to the OpenAI-compatible
/endpoints/openapi/chat/completions URL. No gemma-detection exclusion
needed (the "gemma/" check uses a slash, which google/gemma-... doesn't
match). No OpenAIGPTConfig subclass needed — works with the base handler.
* [Test] Add Azure async chat completion timeout test. WIP
* Capture TTFT for /v1/messages streaming responses
The pass-through streaming path for /v1/messages (Anthropic, Bedrock,
Vertex AI, Azure AI, Minimax) logged completion_start_time only after
the entire stream finished. async_success_handler then fell back to
end_time, making TTFT equal to total duration or null in the UI and
Prometheus.
Record the timestamp of the first chunk in async_sse_wrapper and
propagate it to model_call_details before the logging handler runs,
so gen_ai.response.time_to_first_token reflects the real first-chunk
latency.
Fixes#25598
* [Refactor] Implement timeout resolution logic in completion function
add fetch ``request_timeout`` from litellm_settings
* remove stale test case
* remove extra print statement
* default request timeout value in constants to 600s to match timeout defaults handled in the proxy
* fix request timeout if using default value from constants.py
* update code structure, test cases
* only override if the global timeout sets timeout to 6000s
* update code structure, move hard coded values to const and make the reslve function readable by moving fallback logic to a seperate function
* modify default timeout values, replacing hard coded ones with default values defined
---------
Co-authored-by: harish876 <harishgokul01@gmail.com>
Co-authored-by: Joaquin Hui Gomez <joaquinhuigomez@users.noreply.github.com>
Unit Tests: Proxy DB Operations / proxy-db (auth-checks, tests/proxy_unit_tests/test_auth_checks.py tests/proxy_unit_tests/test_user_api_key_auth.py, 20, 8) (push) Waiting to run
Unit Tests: Proxy DB Operations / proxy-db (remaining, tests/proxy_unit_tests --ignore=tests/proxy_unit_tests/test_key_generate_prisma.py --ignore=tests/proxy_unit_tests/test_auth_checks.py --ignore=tests/proxy_unit_tests/test_user_api_key_auth.py, 30, 8) (push) Waiting to run
* fix: make PodLockManager.release_lock atomic compare-and-delete
Re-lands #21226 (reverted in #21469).
release_lock() previously did GET + compare + DEL in separate calls,
leaving a window where another pod could reacquire the lock between
the GET and DEL, causing a stale owner to delete a live lock.
Fix: use a Redis Lua script for atomic compare-and-delete. Script
registration is cached per PodLockManager instance. Falls back to
the old GET+DEL path for cache backends that don't expose
async_register_script.
Original revert was due to e2e tests running in CI without Redis.
Those tests now carry @pytest.mark.skip(reason="Requires Redis connection.")
so this re-land is safe.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* fix: add Lua fallback on execution error + test coverage gaps
Address Greptile review feedback on #24466:
1. Wrap Lua script execution in try/except — if Redis clears loaded
scripts (restart) or scripting is disabled, fall back to GET+DEL
rather than letting the exception propagate and leave the lock held
until TTL. Reset cached script handle so the next call re-registers.
2. Add test_release_lock_lua_path_emits_released_event — verifies
_emit_released_lock_event is called when Lua path returns 1.
3. Add test_release_lock_falls_back_to_get_del_when_lua_execution_fails
— verifies the fallback path is taken and script handle is reset.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
---------
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
Current fix includes
- Updates test case
- Optimized query with docstring. The change leverages deduplication and sorting logic from SQL
- Added a bench script to differentiate peak memory usage before and after
Vertex AI rejects requests containing both search tools (googleSearch,
enterpriseWebSearch, urlContext) and function declarations with error:
'Multiple tools are supported only when they are all search tools.'
When _merge_tools_from_deployment() combines deployment-level search
tools with user-request function tools (e.g. via MCP), the mixed tool
list causes a 400 error. This fix detects the conflict in _map_function()
and drops search tools, keeping function declarations.
Non-search tools like code_execution and computerUse are preserved.
Fixes#23337
* fix#25506
* address greptile review feedback
* [Test] UI - Models: Add E2E tests for Add Model flow
Add E2E tests covering:
- Test connection with bad credentials shows failure modal
- Adding a specific model and verifying it appears in All Models table
- Adding a wildcard route and verifying it appears in All Models table
- Verifying model dropdown shows provider-specific models (existing test updated)
Added data-testid attributes to UI components to support stable test selectors.
Tests verified passing 3/3 consecutive runs with zero flakiness.
* address greptile review feedback (greploop iteration 1)
Add cleanup helper to delete models created during tests, preventing
stale data accumulation across repeated test runs.
* fix CI: replace data-testid selectors with text/role-based selectors
The data-testid attributes added to React components are not present
in the CI-built UI output. Switch to using getByRole and getByText
selectors which work with the rendered DOM regardless of build cache.
* remove unnecessary cleanup helper
The database is freshly seeded on every test run via seed.sql,
so per-test cleanup is not needed.
---------
Co-authored-by: Yuneng Jiang <yuneng@berri.ai>
Co-authored-by: Krrish Dholakia <krrish+github@berri.ai>
* fix: emit input_json_delta for tool args bundled in first streaming chunk
Some providers (xAI, Gemini) include tool_call function arguments in the
same streaming chunk as the function name/id. The AnthropicStreamWrapper
was discarding the trigger chunk entirely when starting a new content
block, which silently dropped the input_json_delta carrying tool
arguments. This caused tool_use blocks to arrive with empty input {}.
Now queue the processed_chunk after content_block_start when it carries
non-empty input_json_delta data. Backward compatible: providers that send
empty arguments in the first chunk (OpenAI-style) are unaffected since
the condition checks for truthy partial_json.
* test: add tests for input_json_delta emission on bundled tool args
Covers the fix for providers (xAI, Gemini) that bundle tool_call
arguments in the same streaming chunk as the function name/id.
Verifies the AnthropicStreamWrapper emits input_json_delta after
content_block_start, and that empty-arg chunks (OpenAI-style) are
unaffected.
* style: apply Black formatting to streaming_iterator.py
* fix: mirror input_json_delta fix to sync __next__ and add sync tests
* test: make no_extra_delta tests assert explicitly instead of passing silently
Extend existing test modules with coverage for the instructions merge
logic, upstream cache, ContextVar-based injection, and client-side
capture — following each file's established patterns.
Made-with: Cursor
Block cross-team key update/regenerate operations by raising when the caller is not a member of the target key's team, and add unit coverage for deny/allow team membership paths.
Made-with: Cursor
- Omit messages whose list content is empty after stripping thinking blocks
- Retry only on HTTP 400 plus invalid-signature body match
- Return response inline from retry loop; drop unreachable None guard
- Tests: thinking-only turn dropped, non-400 no retry
Made-with: Cursor
