mirror of
https://github.com/BerriAI/litellm.git
synced 2026-09-28 01:32:17 +00:00
5 commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
dependabot[bot]
|
986bdb5889
|
build(deps): bump the github-actions group across 1 directory with 21 updates
Bumps the github-actions group with 21 updates in the / directory: | Package | From | To | | --- | --- | --- | | [actions/checkout](https://github.com/actions/checkout) | `4.2.2` | `7.0.1` | | [actions/setup-python](https://github.com/actions/setup-python) | `5.6.0` | `7.0.0` | | [actions/cache](https://github.com/actions/cache) | `4.3.0` | `6.1.0` | | [actions/upload-artifact](https://github.com/actions/upload-artifact) | `4.6.1` | `7.0.1` | | [actions/download-artifact](https://github.com/actions/download-artifact) | `4.2.1` | `8.0.1` | | [codecov/codecov-action](https://github.com/codecov/codecov-action) | `5.5.4` | `7.1.1` | | [actions/setup-node](https://github.com/actions/setup-node) | `4.4.0` | `7.0.0` | | [github/codeql-action/init](https://github.com/github/codeql-action) | `3.34.1` | `4.38.1` | | [github/codeql-action/analyze](https://github.com/github/codeql-action) | `3.34.1` | `4.38.1` | | [github/codeql-action/upload-sarif](https://github.com/github/codeql-action) | `3.34.1` | `4.38.1` | | [CodSpeedHQ/action](https://github.com/codspeedhq/action) | `4.12.1` | `5.2.1` | | [actions/github-script](https://github.com/actions/github-script) | `7.0.1` | `9.0.0` | | [openai/codex-action](https://github.com/openai/codex-action) | `1.9` | `1.12` | | [azure/setup-helm](https://github.com/azure/setup-helm) | `4.3.1` | `5.0.1` | | [ossf/scorecard-action](https://github.com/ossf/scorecard-action) | `2.4.1` | `2.4.4` | | [actions/stale](https://github.com/actions/stale) | `8.0.0` | `11.0.0` | | [aws-actions/configure-aws-credentials](https://github.com/aws-actions/configure-aws-credentials) | `6.2.0` | `6.3.0` | | [taiki-e/install-action](https://github.com/taiki-e/install-action) | `2.87.8` | `2.87.18` | | [hashicorp/setup-terraform](https://github.com/hashicorp/setup-terraform) | `3.1.2` | `4.0.1` | | [actions/setup-go](https://github.com/actions/setup-go) | `6.2.0` | `7.0.0` | | [zizmorcore/zizmor-action](https://github.com/zizmorcore/zizmor-action) | `0.5.6` | `0.6.4` | Updates `actions/checkout` from 4.2.2 to 7.0.1 - [Release notes](https://github.com/actions/checkout/releases) - [Changelog](https://github.com/actions/checkout/blob/main/CHANGELOG.md) - [Commits](https://github.com/actions/checkout/compare/v4.2.2...3d3c42e5aac5ba805825da76410c181273ba90b1) Updates `actions/setup-python` from 5.6.0 to 7.0.0 - [Release notes](https://github.com/actions/setup-python/releases) - [Commits]( |
||
|
devin-ai-integration[bot]
|
48050d9646
|
chore(ci): drop litellm_internal_staging and litellm_oss_staging references, main is the only trunk (#42745) | ||
|
devin-ai-integration[bot]
|
ae01882535
|
feat(proxy): offload spend tracking to a pod-local collector sidecar (#40545)
* feat(proxy): offload spend tracking to a pod-local spend worker sidecar py-spy on the gateway showed the post-response _PROXY_track_cost_callback, spend-log and DBSpendUpdateWriter work running on the inference workers' event loop, so a DB or Redis stall backed up the request path. When LITELLM_SPEND_WORKER_ENABLED=true, _ProxyDBLogger serializes one compact typed SpendEvent per success and hands it to a SpendEventProducer that ships it over a unix socket (default) or loopback-only TCP to a sidecar started as `python -m gateway.spend_worker`. The sidecar runs the unchanged _ProxyDBLogger pipeline against the pod's PgBouncer (pooled_database_url). When the sidecar is unreachable, the buffer is full, or the gateway shuts down with events still queued or in flight, the producer applies LITELLM_SPEND_WORKER_ON_UNAVAILABLE (fallback in-process, or drop). The sidecar half-closes producers on SIGTERM and drains, the producer treats EOF as unavailable, and the gateway flushes buffered spend counters on shutdown. The sidecar honors LITELLM_LOG so its writes are visible in its own process log. Helm: both charts gain an opt-in spend-worker sidecar container sharing an emptyDir socket dir, and the componentized chart's HPA uses a ContainerResource CPU metric scoped to the gateway container so sidecar CPU does not drive inference scaling. * feat(terraform): opt-in spend-worker sidecar for the AWS and GCP gateway stacks Adds spend_worker_* inputs to both modules. On ECS Fargate the sidecar is a second, non-essential container in the gateway task; on Cloud Run it is a second container in the gateway service. Both listen on loopback TCP, share the gateway's DB/Redis/secret env, and set LITELLM_JOB_ROLE=spend_worker. Disabled by default. Plan-only tests cover both, and the terraform CI workflow now runs the gcp module too Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(proxy): retrieve a completed batch in the in-process spend path test The base now defers cost tracking for batches that are still in flight, so an in_progress batch never reaches update_database Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * refactor(proxy): rename the spend worker sidecar to collector Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(proxy): run the collector from the installed litellm package and finish in-flight fallbacks on shutdown The sidecar command becomes python -m litellm.proxy.collector so the classic image, whose runtime stage copies only the installed package, can run it. The module now assembles DATABASE_URL and the pod-local pgbouncer URL itself, replacing gateway/collector.py The componentized collector sidecar inherits gateway.volumeMounts so custom CA mounts reach it. SpendEventProducer shields an in-progress fallback from the writer task cancellation so close() no longer loses an event already handed to the in-process pipeline Helpers used across modules (address_argument, should_store_prompts_and_responses_in_spend_logs, flush_spend_counters_on_shutdown) become public so the change adds no reportPrivateUsage errors Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * ci(terraform): drop the gcp job duplicated by the aws/gcp matrix Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(collector): keep metrics env off the classic sidecar and reject shared loopback ports The classic chart no longer hands PROMETHEUS_METRICS_PORT and the billing metrics env to the collector container, and gives it the same /.npm scratch mount as the proxy on a read-only root. AWS and GCP now refuse a plan where the spend collector and the metrics sidecar bind the same loopback port. A regression test drives a sidecar crash mid-stream on asyncio and uvloop and checks no event is billed by both the sidecar and the in-process fallback; the producer docstring spells out why a failed drain() cannot double count Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * style(proxy): format pooled_database_url after the pgbouncer rebase Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(proxy): keep the cache-hit preset key and survive dead producers on collector drain Cache hits updated the logging object after the early return, so the offloaded spend event carried preset_cache_key=None and the collector re-hashed reconstructed kwargs. Also guard write_eof() against producer transports uvloop already closed so one dead connection cannot abort the drain Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(terraform): keep the gcp collector port off the metrics sidecar health port Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(proxy): collector connects to Postgres directly under IAM or Entra token auth Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(proxy): mark the collector's DATABASE_URL as pooled when it uses the pod's pgbouncer Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: yassin <yassin@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
devin-ai-integration[bot]
|
b290dd410e
|
feat(terraform/gcp): dependencies-only mode and bring-your-own-network for GKE (#39695)
* feat(terraform/gcp): add dependencies-only mode Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(terraform/gcp): review fixes for dependencies-only mode Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: yassin <yassin@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Yassin Kortam
|
5bc523a09f
|
feat(terraform/aws): make VPC, Aurora, and Redis optional (#36676)
Adds vpc_id/public_subnet_ids/private_subnet_ids to deploy into existing networking, plus create_database/database_url and create_redis/redis_url to use existing data stores or none at all. Defaults keep today's module-owned behavior. A container reads a secret by ARN, which gives Terraform no edge to the _version that writes the value. The managed-Aurora path hid that behind the cluster create; the bring-your-own path has nothing slow in between, so the migration, the bootstrap, and both services now depend on the versions explicitly. Supplied private subnets must cover two AZs whenever Aurora or ElastiCache is module-created, since both subnet groups require it, and a check block warns when a Redis-less stack can run more than one gateway process, because per-key rate limits are counted per process without Redis. |