mirror of
https://github.com/BerriAI/litellm.git
synced 2026-10-16 04:29:31 +00:00
3cc9e7ef8e
/public/providers/fields re-read and re-parsed the bundled 85K provider_create_fields.json on every request, and /public/agents/fields did the same for that file plus agent_create_fields.json. Both are unauthenticated public endpoints, and both did the work inside an `async def`, so the read and the parse ran on the event loop and serialized against every other in-flight request. Measured against the bundled files, that is roughly 0.40 ms of blocking work per request for the provider endpoint and 0.45 ms for the agent endpoint Both now memoize with `lru_cache`, which is what `/public/endpoints` in the same module already does through its `_cached_endpoints` global, so each file is read once per process rather than once per request The agent endpoint mutated the dicts it had just loaded, appending inherited provider credential fields and popping `inherit_credentials_from_provider`. That is only safe while the data is re-read every time, so caching it meant making the merge pure: `_agent_with_inherited_credentials` builds a new mapping per agent, and the cached value is the merged result rather than the raw file contents. Responses are byte identical before and after, checked on both endpoints against a live proxy The bundled records are typed as Mapping[str, object] and narrowed with isinstance at the point of use, rather than typing.Any, which ruff-strict bans Side effects: 3 of the repo's ASYNC230 violations go away, and basedpyright errors in the file drop from 63 to 52 Rebased onto main after /public/providers/fields started validating the bundled JSON through a pydantic adapter on every request. That validation now runs once and the validated records are cached; the agent endpoint keeps merging the raw provider records, so both responses stay byte identical to main. Co-authored-by: Alok Sharma <alok44170@gmail.com> |
||
|---|---|---|
| .. | ||
| docker-entrypoint.d | ||
| litellm-dashboard | ||
| Dockerfile | ||
| nginx.conf | ||