devin-ai-integration[bot] 02feb67c20 test(integration): cut CircleCI proxy boot time with concurrent worker stacks and single-worker test proxies (#46037) ... * test(integration): build worker stacks concurrently and record proxy boot phases The controller built every xdist worker's database, Redis, upstream and gateway proxy one after another before any test ran, about 95 s per extensions shard, and since every multi-worker suite is now isolated the same serial wait hits mcp, security and cost too. The stacks now start on a thread pool bounded by the CPU count. Each stack reads its endpoints and results directory from an explicit environment instead of a patched os.environ, every stack that started is registered for cleanup, and a failure in one stops the others before the error is raised Owned proxy startup.json now records the wall-clock spawn time, the worker count and when the Prisma CLI child ran, when the port started accepting and when the first worker answered readiness, and exit.json records when the stop started and ended, so the boot time can be split and overlapping gateway boots can be seen in the artifacts * test(integration): boot one uvicorn worker where a test does not assert cross-worker behaviour A 2-worker owned proxy takes about twice as long to become ready as a 1-worker one, because the uvicorn parent imports the app and each spawned worker imports it again. The passthrough upstream error visibility cases, the single-request S3 dedupe cases and the single-request spend log payload cases send one or two requests and assert on the response, the proxy log and the spend row, none of which depends on a second worker, so they now boot the default single worker Cases that do rely on two workers keep them, the concurrent S3 burst through a sink outage, previous_response_id replay, response cache hits, the concurrent spend bursts and upstream stop and recovery * test(observability): settle OTel excluded-services traces by a quiet period instead of a full window _trace_spans slept through its whole 15 or 30 s window on every call even when the trace had long stopped growing. It now polls and returns once the trace has spans and has not gained one for five tenant export intervals, and the module pins the tenant batch exporter's schedule delay to 1 s so that bound is set by the test rather than by the SDK default. Absence checks still run only after the trace stopped growing, and against a build that leaks postgres spans to the tenant they still fail These cases also boot one uvicorn worker now. None of them asserts anything about a second worker --------- Co-authored-by: Yuneng Jiang <yuneng@berri.ai>		2026-10-12 09:48:42 -07:00
..
fixtures	refactor(repositories): daily activity repository with centralized bounded usage queries (#43398)	2026-10-01 16:02:01 -07:00
golden	feat(proxy): bounded daily activity routes for all usage entities (#43408)	2026-10-01 16:57:23 -07:00
__init__.py	test: migrate tests/proxy_behavior into integration and unit tiers (#45939)	2026-10-11 08:28:42 -07:00
_daily_activity_fixtures.py	feat(usage): add top users by spend and user activity views (#45310)	2026-10-10 12:21:43 -07:00
test_autorouter_session_rollup.py	test: migrate tests/proxy_behavior into integration and unit tiers (#45939)	2026-10-11 08:28:42 -07:00
test_background_interaction_settlement.py	test(integration): give chaos and timing tests a harness API and gate raw signals (#46031)	2026-10-12 05:27:33 -07:00
test_background_response_poll_retirement.py	fix(proxy): mark background responses stale_expired when provider returns 404 (#41724)	2026-10-08 10:35:52 -07:00
test_baseline_accounting.py	test: migrate tests/proxy_behavior into integration and unit tiers (#45939)	2026-10-11 08:28:42 -07:00
test_batch_completion_accounting.py	feat(spend_logs): configure which metadata fields are stored in LiteLLM_SpendLogs (#44659)	2026-10-08 14:51:44 -07:00
test_batch_enqueued_tokens_redis_lua.py	test(proxy): migrate DB and Redis backed proxy tests into tests/integration (#43996)	2026-10-01 09:23:36 -07:00
test_batch_observability.py	test(integration): regression tests for August cost tracking and budgeting bugs (#42622)	2026-09-23 04:13:03 +00:00
test_batch_poll_starvation.py	test(integration): regression tests for August cost tracking and budgeting bugs (#42622)	2026-09-23 04:13:03 +00:00
test_budget_limit_envelopes.py	test: replace 61 live logging and otel tests with offline unit and integration coverage (#45362)	2026-10-08 15:55:57 -07:00
test_cache_activity.py	test: migrate tests/proxy_behavior into integration and unit tiers (#45939)	2026-10-11 08:28:42 -07:00
test_cache_and_quota.py	test(integration): run the Langfuse DB-callback test on its own scratch database (#43288)	2026-09-26 00:15:56 -07:00
test_chaos_burst_spend_once.py	refactor(types): replace Any with proven types in 7 files (#43704)	2026-09-29 06:12:58 -07:00
test_daily_activity_aggregated_breakdowns.py	refactor(repositories): daily activity repository with centralized bounded usage queries (#43398)	2026-10-01 16:02:01 -07:00
test_daily_activity_key_alias_probes.py	test(integration): give chaos and timing tests a harness API and gate raw signals (#46031)	2026-10-12 05:27:33 -07:00
test_daily_activity_key_metadata_query_timeout.py	test(integration): give chaos and timing tests a harness API and gate raw signals (#46031)	2026-10-12 05:27:33 -07:00
test_daily_activity_key_owner.py	fix(proxy): recover session key owners from daily spend for usage attribution (#43642)	2026-09-29 15:36:09 -07:00
test_daily_activity_key_owner_faults.py	test(integration): give chaos and timing tests a harness API and gate raw signals (#46031)	2026-10-12 05:27:33 -07:00
test_daily_activity_key_owner_traffic.py	fix(proxy): look up hashed key names with two spend log rows per key (#43656)	2026-09-29 20:21:55 -07:00
test_daily_activity_repository.py	feat(usage): add top users by spend and user activity views (#45310)	2026-10-10 12:21:43 -07:00
test_daily_activity_routes.py	fix(daily_activity): keep NULL entity ids when excluding entity ids (#44139)	2026-10-01 21:19:48 -07:00
test_daily_rollup_retry.py	test(integration): add read-replica routing harness to the CircleCI integration suite (#42692)	2026-09-24 00:25:03 -07:00
test_daily_tag_spend_retention.py	test(integration): give chaos and timing tests a harness API and gate raw signals (#46031)	2026-10-12 05:27:33 -07:00
test_disconnected_bedrock_messages_stream_billing.py	test(integration): regression tests for August cost tracking and budgeting bugs (#42622)	2026-09-23 04:13:03 +00:00
test_end_user_spend_without_proxy_user.py	test(integration): regression tests for July cost tracking, budgeting and spend bugs (#42694)	2026-09-23 09:51:56 -07:00
test_failed_dispatch_tokens.py	test(integration): regression tests for August cost tracking and budgeting bugs (#42622)	2026-09-23 04:13:03 +00:00
test_filtered_ledger.py	test: add extension and browser integration contracts	2026-09-16 13:15:53 -07:00
test_global_spend_report.py	test(integration): move legacy proxy, router and Redis tests into tests/integration (#44128)	2026-10-01 23:00:54 -07:00
test_image_generation_key_spend.py	test(integration): move legacy proxy, router and Redis tests into tests/integration (#44128)	2026-10-01 23:00:54 -07:00
test_key_budget_lockout.py	test(integration): move legacy proxy, router and Redis tests into tests/integration (#44128)	2026-10-01 23:00:54 -07:00
test_key_metadata_recovery_dropped_connection.py	fix(spend-tracking): stop caching failed spend-log metadata lookups as confirmed misses (#43560)	2026-10-05 22:36:25 +00:00
test_key_metadata_recovery_probe_bounds.py	fix(proxy-extras): build the SpendLogs indexes in the migration job instead of in migrations (#43948)	2026-10-01 14:12:22 -07:00
test_legacy_spend_logs_row_cap.py	test(integration): regression tests for July cost tracking, budgeting and spend bugs (#42694)	2026-09-23 09:51:56 -07:00
test_messages_stream_usage_cost.py	test(integration): regression tests for July cost tracking, budgeting and spend bugs (#42694)	2026-09-23 09:51:56 -07:00
test_model_router_selected_model.py	test(integration): regression tests for August cost tracking and budgeting bugs (#42622)	2026-09-23 04:13:03 +00:00
test_normalized_error_long_message.py	test(integration): give chaos and timing tests a harness API and gate raw signals (#46031)	2026-10-12 05:27:33 -07:00
test_org_budget_cli_session_token.py	test(integration): regression tests for August cost tracking and budgeting bugs (#42622)	2026-09-23 04:13:03 +00:00
test_passthrough_budget_reservation.py	test(integration): regression tests for August cost tracking and budgeting bugs (#42622)	2026-09-23 04:13:03 +00:00
test_passthrough_request_tags.py	fix(proxy): carry key, team and project tags into pass-through spend logs (#42662)	2026-10-02 19:38:44 +00:00
test_prompt_caching_requests_pagination.py	test(proxy): migrate DB and Redis backed proxy tests into tests/integration (#43996)	2026-10-01 09:23:36 -07:00
test_redis_ttl_preserving_token_increment.py	refactor(proxy): expose public names for private proxy helpers (#45170)	2026-10-07 18:59:27 -07:00
test_reset_budget_leader_election.py	test(integration): regression tests for July cost tracking, budgeting and spend bugs (#42694)	2026-09-23 09:51:56 -07:00
test_responses_bridge_spend_once.py	fix(logging): log spend once for large non-streaming requests on the chat to Responses bridge (#44508)	2026-10-06 21:23:19 +00:00
test_responses_cache_write_itemization.py	test(integration): regression tests for July cost tracking, budgeting and spend bugs (#42694)	2026-09-23 09:51:56 -07:00
test_roi_branch_spend.py	fix(caching): key response cache by router model group in litellm_metadata (#44542)	2026-10-05 17:06:21 +00:00
test_service_tier_stream_billing.py	fix(streaming): keep the served service_tier on streamed chunks and spend rows (#42870)	2026-09-29 12:54:17 -07:00
test_session_total_spend.py	test(integration): regression tests for July cost tracking, budgeting and spend bugs (#42694)	2026-09-23 09:51:56 -07:00
test_shutdown_flush.py	test(integration): give chaos and timing tests a harness API and gate raw signals (#46031)	2026-10-12 05:27:33 -07:00
test_spend_attribution_contracts.py	test: move 76 live top-level tests to offline unit and integration coverage (#45352)	2026-10-08 10:33:35 -07:00
test_spend_calculate.py	fix(pricing): drop the unpublished cached rate from the Gemini Live preview entries (#42651)	2026-09-22 20:35:34 -07:00
test_spend_capture_rate_captured_spend.py	test(proxy): migrate DB and Redis backed proxy tests into tests/integration (#43996)	2026-10-01 09:23:36 -07:00
test_spend_log_read_scope.py	fix(proxy): share ownership permissions for spend logs and traces (#44239)	2026-10-03 01:38:54 +00:00
test_spend_log_tool_payload_content.py	test(integration): cut CircleCI proxy boot time with concurrent worker stacks and single-worker test proxies (#46037)	2026-10-12 09:48:42 -07:00
test_spend_log_write_batching.py	test(integration): regression tests for July cost tracking, budgeting and spend bugs (#42694)	2026-09-23 09:51:56 -07:00
test_spend_logs_metadata_fields.py	feat(spend_logs): configure which metadata fields are stored in LiteLLM_SpendLogs (#44659)	2026-10-08 14:51:44 -07:00
test_spend_rollup_accuracy.py	test(integration): move legacy proxy, router and Redis tests into tests/integration (#44128)	2026-10-01 23:00:54 -07:00
test_stream_alias_billing.py	fix(proxy): stamp the client alias on a copy of each streamed chunk so pricing sees the deployment model (#44341)	2026-10-03 06:44:52 +00:00
test_tag_budget_enforcement.py	feat(proxy): enforce tpm_limit and rpm_limit set on tag objects (#41807)	2026-09-24 18:11:57 -07:00
test_team_budget_enforcement.py	test(integration): move legacy proxy, router and Redis tests into tests/integration (#44128)	2026-10-01 23:00:54 -07:00
test_team_daily_activity_aggregated.py	test(integration): regression tests for August cost tracking and budgeting bugs (#42622)	2026-09-23 04:13:03 +00:00
test_team_member_budget_alerts.py	feat(proxy): email alerts at configured percentages of a team member budget (#42665)	2026-09-26 02:42:45 +00:00
test_team_member_spend.py	test(integration): regression tests for July cost tracking, budgeting and spend bugs (#42694)	2026-09-23 09:51:56 -07:00
test_team_member_spend_flush.py	fix(proxy): pass team member spend rows as jsonb so a $0 flush cannot poison the pool connection (#43029)	2026-09-24 20:44:56 +00:00
test_user_budget_on_team_keys.py	test(integration): regression tests for July cost tracking, budgeting and spend bugs (#42694)	2026-09-23 09:51:56 -07:00
test_vertex_image_regional_uplift.py	test(integration): give chaos and timing tests a harness API and gate raw signals (#46031)	2026-10-12 05:27:33 -07:00
test_vertex_regional_uplift_spend.py	fix(vertex_ai): add regional endpoint uplift to gemini-3.1-flash-image (#44673)	2026-10-05 21:00:22 -07:00