mirror of
https://github.com/BerriAI/litellm.git
synced 2026-10-06 02:48:13 +00:00
10462eddaf
* test(e2e): harness fixes for long_context, complexity router, UI, and unit coverage Point long_context_1m at 1M-capable models, harden complexity-smart-router registration and spend-log assertions, fix key models dropdown selectors, and add gateway/lifecycle/transport and claude_code unit tests * test(e2e): harden remaining stage failures in harness Register complexity-smart-router via create_model + callable probe, fix create-key UI navigation race, retry management writes and budget ALB 502s, mark Vertex count_tokens N/A when unsupported, and tighten tool_search model lists for Azure/Bedrock capability gaps * test(e2e): drop claude_code and harness unit tests from this PR Keep management, router, budget, and shared conftest harness fixes only * test(e2e): restore E2E_RESULT pytest_runtest_makereport hook Accidentally dropped in an earlier harness commit; Grafana status history depends on these structured log lines * test(e2e): drop management control-plane write retries Transient 500 retries do not fix the underlying control plane failures * test(e2e): skip stage-red claude_code cells; fix multi-window budget latency Mark the twelve failing claude_code matrix cells skip until product/config lands. Multi-window budget polls gpt-5.5 with max_tokens=1 instead of Claude so the reset wait stays under ALB target idle timeout rather than masking awselb 502s * test(e2e): require exactly one LLM-tier spend row for complexity router Keep alias membership for compose vs stage model names, but assert len(served) == 1 so a leaked classifier sub-call cannot pass. Also pin LIT-4521 skip and align LIT-4522/23/24 skip reasons * test(e2e): harden router callable probe and multi-window budget exhaustion _router_is_callable treated any non-success chat whose body lacked "Invalid model name" as callable, so an unpropagated probe key (401), a generic 502, or a connection reset let the session proceed and hit real "Invalid model name" failures inside the tests. Require a Success outcome instead; the reload-race 400 and every infra/auth error now correctly read as not-callable. The multi-window budget test capped the tight window at 3e-6, which gpt-5.5 exhausts on the first call but a cheaper CHEAP_OPENAI_MODEL might not within the 20-call loop, turning a reset test into a spurious "window never enforced" failure. Drop the tight cap to 1e-9 so the first billed call exhausts it regardless of model price; the roomy 1m window stays at 1.0 and never blocks. * test(e2e): use a tradeoff-decision prompt for the complexity router classifier "Is P equal to NP?" reads to the LLM classifier as a short yes/no question, so gpt-5.5 classified it SIMPLE and the request routed to the openai backend, which made the test fail even though the classifier was running. The tier definitions key on what the request demands, not how hard the answer is, and a short direct question maps to SIMPLE regardless of subject. Swap in "Should I pay off my mortgage early or invest the extra money instead?". It carries none of the heuristic scorer's reasoning/technical/code keywords and stays short, so heuristic scoring still lands SIMPLE (openai), but the LLM reads it as a decision that has to weigh tradeoffs and lands it above SIMPLE, which the config routes to anthropic. Any non-SIMPLE tier serves anthropic, so the classifier only has to avoid SIMPLE for the test to distinguish a real classifier run from the heuristic fallback.
83 lines
3.2 KiB
Python
83 lines
3.2 KiB
Python
"""tool_search x Azure (Microsoft Foundry).
|
|
|
|
HTTP-probe row. Sends a single `/v1/messages` request whose `tools`
|
|
array includes a `tool_search_tool_regex_20251119` discovery tool, and
|
|
asserts the proxy round-trips it to the upstream without a 400. This
|
|
verifies LiteLLM's tool-search beta-header translation
|
|
(`advanced-tool-use-2025-11-20` for Anthropic-shape providers,
|
|
`tool-search-tool-2025-10-19` for Vertex/Bedrock) survives end-to-end.
|
|
|
|
The (feature, provider) for this cell is inferred from the file path by
|
|
`tests/e2e/claude_code/conftest.py`:
|
|
|
|
tests/e2e/claude_code/tool_search/test_azure.py
|
|
^^^^^^^^^^^ ^^^^^
|
|
feature_id provider
|
|
|
|
Why HTTP probe instead of CLI / MCP fan-out:
|
|
|
|
Real Claude Code activates tool_search by registering >N MCP tools and
|
|
relying on the model's internal heuristic to call the discovery tool
|
|
before any user tool. That setup requires standing up a stub MCP
|
|
server that exposes 50+ tool stubs and depends on Claude Code's
|
|
auto-deferral heuristic continuing to fire at today's tool count --
|
|
both of which break silently when Claude Code's threshold changes
|
|
between releases.
|
|
|
|
The bugs LiteLLM has actually shipped fixes for in this area
|
|
(2.1.117, 2.1.72, 2.1.70 in the Claude Code release notes) are
|
|
beta-header translation and proxy-side type recognition, not MCP
|
|
fan-out behavior. An HTTP probe hits exactly that surface: the
|
|
request goes out with a `tool_search_tool_regex_20251119` tool type,
|
|
the proxy is responsible for attaching the per-provider beta header
|
|
and forwarding, and the upstream either accepts or 400s. A red cell
|
|
here is always a proxy-side regression, not a flaky model-behavior
|
|
artifact.
|
|
|
|
Three Claude tiers are probed in sequence (count is too low to be
|
|
worth the parallelism overhead, and HTTP probes don't compete for
|
|
the proxy's `--num-workers` slots the way CLI subprocess runs do).
|
|
The matrix's "all three must pass" rule still applies via the
|
|
per-cell aggregator.
|
|
"""
|
|
|
|
from __future__ import annotations
|
|
|
|
import pytest
|
|
|
|
from claude_code._env import require_proxy
|
|
from claude_code.http_probe import (
|
|
assert_tool_search_shape,
|
|
probe_tool_search,
|
|
)
|
|
|
|
|
|
AZURE_MODELS = [
|
|
"claude-haiku-4-5-azure",
|
|
"claude-sonnet-4-5-azure",
|
|
"claude-opus-4-7-azure",
|
|
]
|
|
|
|
|
|
@pytest.mark.skip(reason="stage red: Azure Foundry tool_search_server not supported in workspace for probed models")
|
|
@pytest.mark.covers("llm.messages.azure_foundry.tool_search.nonstream.works")
|
|
def test_tool_search_azure(compat_result):
|
|
"""Probe `/v1/messages` with a `tool_search_tool_regex_20251119`
|
|
tool and assert the proxy + upstream accept it for every Azure (Microsoft Foundry)
|
|
tier."""
|
|
base_url, api_key = require_proxy(compat_result)
|
|
|
|
failures = []
|
|
for model in AZURE_MODELS:
|
|
result = probe_tool_search(base_url=base_url, api_key=api_key, model=model)
|
|
shape_error = assert_tool_search_shape(result)
|
|
if shape_error is not None:
|
|
error = f"[{model}] tool_search probe failed: {shape_error}"
|
|
compat_result.add({"status": "fail", "error": error})
|
|
failures.append(error)
|
|
continue
|
|
|
|
compat_result.add({"status": "pass"})
|
|
|
|
if failures:
|
|
pytest.fail("; ".join(failures), pytrace=False)
|