Commit graph

3 commits

Author SHA1 Message Date
yuneng-jiang
21ab2cb87b
test: update stale integration and guardrail e2e tests to intended behavior (#48665)
* test(integration): update stale expectations to intended provider behavior

Each expectation moved because a merged PR changed behavior on purpose.
- chatgpt string input is wrapped as one user message (#41522)
- Gemini 2.5 tool media sits beside function_response (#48428)
- sampling knobs move to Gemini 3.5 Flash, which keeps them (#47645)
- a body base_model is stripped without allow_client_pricing_override (#47639)
- ai_decide answers without usage return usage null (#48261)
- azure_ai forwards mapped client anthropic-beta values (#48033)
- the OpenCode scripted upstream mints a fresh id per request

* test(e2e-ui): type the guardrail name before deleting it

GuardrailDeleteModal requires the exact name before Delete enables since
#45893. All three guardrail delete flows now go through one helper.

* test(observability): accept any sampled traceresponse flags and expect a 400 for a non-object body

OpenTelemetry 1.45 (#48055) sets the W3C random trace id flag on new traces,
so the gateway's traceresponse now ends in -03. The producer copies the server
span's own trace flags, so the tests now accept any two-digit flags with the
sampled bit set instead of pinning -01

#48438 made a non-object JSON body a 400 "Invalid JSON payload: expected an
object" on purpose, so the [1] body case is renamed non_object_body and
expects that 400

* test(observability): stop waiting on OpenTelemetry's export log line in the langtrace outage test

The outage test waited for "Exception while exporting Span batch" in the proxy
log, which is OpenTelemetry's wording and no longer appears with 1.45. The test
keeps asserting what the gateway guarantees, callers unaffected during the
outage, a live gateway, and every span after recovery delivered once with no
outage span duplicated

* test(guardrails): count the Responses disconnect spend rows #47611 added

Since #47611 a streamed /v1/responses disconnect writes its own spend row, so
the concurrent disconnect test saw 20 rows where it counted 10 by model group.
It now reads each row's call type and expects ten acompletion rows with the
outage verdict and ten aresponses rows with success. /v1/messages disconnects
still write no row on main, so they are left out of the count rather than
pinned as absent

The responses-httpx case of the single disconnect test no longer needs its
LIT-8603 skip and passes

* test(observability): size K4's timeout from the harness boot budget

K4 boots three two-worker proxies in a row. On CircleCI each boot takes 43 to
58 seconds, so a passing run took 179.97s against its fixed 180s cap and 6 of
30 runs hit pytest-timeout. The cap and the restart gate wait now come from
readiness_seconds and graceful_stop_seconds, like other multi-boot tests
2026-10-28 01:11:25 +00:00
Kerry Lu
7b7604f0ed
test(translation): share rich fixtures across Claude, GPT and Gemini providers (#46585)
* test(integration): add rich anthropic claude haiku 5.5, opus 5.5 and fable 5.1 translation cases

Capture-backed basic cases on /v1/chat/completions, /v1/messages and /v1/responses for claude-haiku-5-5, claude-opus-5-5 and claude-fable-5-1, reusing the shared client templates and ANTHROPIC_MESSAGES_REQUEST/RESPONSE with model-specific id, content and usage overrides. The minimal Opus 5.5 hello cases are replaced by the rich conversation. Sonnet 5.5 and legacy models are unchanged

* test(translation): rich OpenAI captures for gpt-6.1-sol, gpt-6-luna, gpt-6-astra and gpt-5.6-terra

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(translation): rich Azure gpt-6.1-sol, gpt-6-luna, gpt-6-astra and gpt-5.6-terra cases

* test(translation): rich azure_ai Claude cases for haiku-5-5 and fable-5-1

Bind the shared chat_completions, messages and responses client templates and the anthropic provider templates to azure_ai/claude-haiku-5-5 and azure_ai/claude-fable-5-1 from real Azure AI Foundry captures, keeping the Azure dialect exact (/anthropic/v1/messages, x-api-key plus api-key on chat_completions and responses, system as a text block list on /v1/messages). Add prompt_cache_hit variants for both models from the second call inside the cache window. The two deployments need rows in tests/integration/proxy_config.yaml

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(translation): rich Gemini bases for gemini-3.8-flash, gemini-3.1-pro-preview and gemini-4-argon

Add the shared generateContent provider templates in templates/gemini.py from
real captures and bind the three target models on chat completions, messages
and responses with explicit per-model response overrides. gemini-4-argon is
registered in the three basic lists. gemini-3.5-flash is unchanged

* test(translation): enrich vertex_ai claude and gemini cases from live captures

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(integration): rich Bedrock converse, invoke and mantle translation cases from real captures

* test(integration): register the rich translation deployments and follow the rich Gemini 3.8 Flash cases

Add the synthetic deployments the rich provider cases bind (anthropic, openai, azure, azure_ai, gemini, vertex_ai, bedrock converse, invoke and mantle) to tests/integration/proxy_config.yaml, all pointed at the fake provider on 127.0.0.1:8191 with synthetic keys. The Gemini omitted-temperature contract now takes its expected generationConfig and reply text from the rich GEMINI_3_8_FLASH_TEST_CASE literals instead of the retired minimal hello cases

* test(translation): reuse bridged and native response templates

---------

Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: kerry <kerry@berri.ai>
2026-10-14 19:53:09 -07:00
Sukhdeep
87478c7b6d
fix(gemini): do not inject temperature for Gemini 3+ models when omitted (#45107)
* fix(gemini): do not inject temperature for Gemini 3+ models when omitted

map_openai_params added temperature=1.0 to every Gemini 3+ request that did not set one, on both the gemini and vertex_ai routes. Google is deprecating temperature, top_p and top_k for Gemini 3+ and says upcoming models will reject them, so this default turns a request that never set temperature into one that will fail

An omitted temperature now falls back to Google's own default, which their docs give as 1.0 for all Gemini 3 models. Explicit temperature, top_p and top_k are still forwarded

Fixes #38663

* test(gemini): annotate sampling regression tests

* test(gemini): drop the injected temperature from Gemini translation cases

The nine Gemini base cases for chat completions, responses and messages send no temperature, so the provider request no longer carries the removed 1.0 default. Each new body was sent to the real Gemini API for gemini-3.5-flash, gemini-3.8-flash and gemini-3.1-pro-preview and returned 200, and the stored provider responses are unchanged

* test(gemini): drop the default-temperature wording from the thinking level test docstring

* test(gemini): cover omitted and explicit sampling on both Gemini routes

---------

Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
2026-10-12 20:26:49 +00:00