scriptoriumadmin/litellm
1
0
Fork 0
mirror of https://github.com/BerriAI/litellm.git synced 2026-09-21 00:21:49 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions 74

test(integration): keep cost diagnostics and widen shard timeout

Browse source 
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
This commit is contained in:
kerry 2026-09-19 00:19:11 +00:00
parent 69f9106759
commit f836bb481d
5 changed files with 53 additions and 21 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

2
.circleci/config.yml
View file

@ -2987,7 +2987,7 @@ jobs:
- run:
name: Run owned integration contracts
command: bash .circleci/scripts/run_integration.sh << parameters.suite >>
no_output_timeout: 15m
no_output_timeout: 25m
- run:
name: Stop owned database and Redis
when: always

6
.circleci/scripts/run_integration.sh
View file

@ -9,6 +9,10 @@ fi
suite="${1:?integration suite required}"
results="test-results/integration-${suite}"
mkdir -p "$results"
shard_timeout=11m
if [ "$suite" = cost ]; then
shard_timeout=20m
fi
integration_identity="$(.venv/bin/python -c 'import uuid; print(uuid.uuid4().hex)')"
upstream_pid=""
scripted_provider_pid=""
@ -181,7 +185,7 @@ if [ "$suite" = browser ]; then
exit 0
fi
timeout --signal=TERM --kill-after=20s 11m env -i PATH="$PATH" HOME="$HOME" PYTHONPATH="$PYTHONPATH" \
timeout --signal=TERM --kill-after=20s "$shard_timeout" env -i PATH="$PATH" HOME="$HOME" PYTHONPATH="$PYTHONPATH" \
INTEGRATION_RUN_ID="$integration_identity" \
DATABASE_URL="$DATABASE_URL" REDIS_HOST="$REDIS_HOST" REDIS_PORT="$REDIS_PORT" \
INTEGRATION_PROXY_URL="$INTEGRATION_PROXY_URL" INTEGRATION_PEER_URL="$INTEGRATION_PEER_URL" \

4
tests/integration/README.md
View file

@ -4,7 +4,7 @@ These tests exercise a running gateway, PostgreSQL and Redis with an owned local
The `cost` group runs the scripted-provider cost matrix through a dedicated sidecar. The sidecar serves the test-owned cost map over loopback through `LITELLM_MODEL_COST_MAP_URL`; cost goldens are checked into the integration suite and must not be copied into the E2E coverage registry
Use `tests/integration/run.py management`, `accounting`, `database`, `providers`, `extensions` or `sdk` to run a selected group. Set `INTEGRATION_PROXY_URL`, `INTEGRATION_UPSTREAM_URL`, `INTEGRATION_MASTER_KEY` and `DATABASE_URL` to an isolated test deployment. The runner selects the new domain directories explicitly; the legacy OCI and sandbox selections remain separate
Use `tests/integration/run.py management`, `accounting`, `database`, `providers`, `extensions`, `sdk` or `cost` to run a selected group. Set `INTEGRATION_PROXY_URL`, `INTEGRATION_UPSTREAM_URL`, `INTEGRATION_MASTER_KEY` and `DATABASE_URL` to an isolated test deployment. The runner selects the new domain directories explicitly; the legacy OCI and sandbox selections remain separate
Management also requires `INTEGRATION_PEER_URL`, `REDIS_HOST` and `REDIS_PORT`. CircleCI starts two directly addressed proxy processes sharing only that job's stores. The test-only CLI wrapper supplies enterprise route entitlement, following the existing behavior suite's convention. It does not qualify license validation; run it with one worker and no reload
@ -22,7 +22,7 @@ Fixtures must contain synthetic data only. Keep private incident records and sou
Database cases own their temporary schemas, roles, constraints and proxy processes. They prove reader-versus-writer execution with PostgreSQL lock observations, exercise real transaction wait limits and verify rollback after a reached database failure
Accounting cases compare persisted input and output cost components against literal rates, including zero and default prices. Cache state models assert actual upstream calls, response identity and every persisted charge. Generated accounting tests have a 180-second test limit to accommodate the asynchronous spend writer; CircleCI keeps the whole shard capped at 11 minutes
Accounting cases compare persisted input and output cost components against literal rates, including zero and default prices. Cache state models assert actual upstream calls, response identity and every persisted charge. Generated accounting tests have a 180-second test limit to accommodate the asynchronous spend writer; CircleCI keeps other shards capped at 11 minutes and gives the cost shard 20 minutes
Provider contracts exercise actual TCP requests with synthetic credentials and local protocol peers. The S3 verifier uses independently implemented equations, a published known-answer vector, a fixed signing clock and deliberately invalid signed requests. Bedrock cases clear ambient AWS credential sources and check the literal model path, loaded role references, STS requests and bearer-only behavior

12
tests/integration/cost_calculation/conftest.py
View file

@ -54,14 +54,20 @@ def approx_equal(actual: float, expected: float) -> bool:
return abs(actual - expected) <= max(1e-9, abs(expected) * 1e-2)
def assert_total_is_sum_of_components(row: CostRow) -> None:
def assert_total_is_sum_of_components(row: CostRow, context: str) -> None:
breakdown: Final = row.breakdown
total: Final = sum(
cost or 0.0
for cost in (breakdown.input_cost, breakdown.output_cost, breakdown.tool_usage_cost)
)
assert breakdown.total_cost is not None and approx_equal(breakdown.total_cost, total)
assert row.spend is not None and approx_equal(row.spend, breakdown.total_cost)
assert breakdown.total_cost is not None and approx_equal(breakdown.total_cost, total), (
f"{context}: total_cost {breakdown.total_cost} != input_cost {breakdown.input_cost} "
f"+ output_cost {breakdown.output_cost} + tool_usage_cost {breakdown.tool_usage_cost} "
f"(sum {total})"
)
assert row.spend is not None and approx_equal(row.spend, breakdown.total_cost), (
f"{context}: row spend {row.spend} != breakdown total_cost {breakdown.total_cost}"
)
def _row(value: Mapping[str, object]) -> CostRow | None:

50
tests/integration/cost_calculation/test_token_pricing.py
View file

@ -200,24 +200,46 @@ def test_scripted_usage_bills_at_map_rates(
if case.stream:
_assert_stream_has_no_error(response.text)
row: Final = poll_cost_row(key)
context: Final = f"{model.map_key}/{case.name}"
if not case.exact_spend:
assert row.prompt_tokens is not None and row.prompt_tokens > 0
assert row.completion_tokens is not None and row.completion_tokens > 0
if case.image_input:
assert row.prompt_tokens < 4000
assert row.spend is not None and approx_equal(
row.spend, recount_cost(model, case, row.prompt_tokens, row.completion_tokens)
assert row.prompt_tokens is not None and row.prompt_tokens > 0, (
f"{context}: no-usage stream counted no input tokens: prompt_tokens={row.prompt_tokens}"
)
assert_total_is_sum_of_components(row)
assert row.completion_tokens is not None and row.completion_tokens > 0, (
f"{context}: no-usage stream counted no output tokens: completion_tokens={row.completion_tokens}"
)
if case.image_input:
assert row.prompt_tokens < 4000, (
f"{context}: image data URL looks tokenized as text: prompt_tokens={row.prompt_tokens}"
)
recount: Final = recount_cost(model, case, row.prompt_tokens, row.completion_tokens)
assert row.spend is not None and approx_equal(
row.spend, recount
), f"{context}: no-usage stream spend {row.spend} != recount {recount} at map rates"
assert_total_is_sum_of_components(row, context)
return
golden: Final = case.expected_for(model)
if not case.stream:
header: Final = cast(str | None, response.headers.get("x-litellm-response-cost"))
assert header is not None and approx_equal(float(header), golden.spend)
assert row.spend is not None and approx_equal(row.spend, golden.spend)
assert header is not None and approx_equal(float(header), golden.spend), (
f"{context}: x-litellm-response-cost {header} != golden {golden.spend}"
)
assert row.spend is not None and approx_equal(row.spend, golden.spend), (
f"{context}: spend {row.spend} != golden {golden.spend} "
f"(breakdown {row.breakdown.model_dump()})"
)
breakdown: Final = row.breakdown
assert breakdown.input_cost is not None and approx_equal(breakdown.input_cost, golden.input_cost)
assert breakdown.output_cost is not None and approx_equal(breakdown.output_cost, golden.output_cost)
assert row.prompt_tokens == golden.prompt_tokens
assert row.completion_tokens == golden.completion_tokens
assert_total_is_sum_of_components(row)
assert breakdown.input_cost is not None and approx_equal(breakdown.input_cost, golden.input_cost), (
f"{context}: gross input_cost {breakdown.input_cost} != golden {golden.input_cost}; "
"cached/written tokens billed at the input rate"
)
assert breakdown.output_cost is not None and approx_equal(breakdown.output_cost, golden.output_cost), (
f"{context}: output_cost {breakdown.output_cost} != golden {golden.output_cost}"
)
assert row.prompt_tokens == golden.prompt_tokens, (
f"{context}: prompt_tokens {row.prompt_tokens} != golden {golden.prompt_tokens}"
)
assert row.completion_tokens == golden.completion_tokens, (
f"{context}: completion_tokens {row.completion_tokens} != golden {golden.completion_tokens}"
)
assert_total_is_sum_of_components(row, context)