* feat(proxy): hash UI passwords with PBKDF2 in FIPS mode
A FIPS OpenSSL refuses scrypt, so every password write fails there and
nobody can sign in with a password. When OpenSSL enforces FIPS, passwords
now use PBKDF2-HMAC-SHA256 at 600k iterations with a salt from OpenSSL's
DRBG, stored as "pbkdf2:" plus base64 of salt and key. Any other row is
refused with one error log and left as it is, and the boot migration does
nothing, because FIPS mode is for new deployments.
PBKDF2 at 600k takes about five times as long as scrypt, so in FIPS mode the
five request paths run it on one thread instead of the event loop. A single
thread keeps a login burst from using up a CPU quota, which is what stalls
the loop with asyncio.to_thread. A login that queued for that thread asks the
throttle again before it answers, so a burst of guesses can't get past a block
that the failures ahead of it just set.
The breached password check is off in FIPS mode, because it runs the
password through SHA-1.
A standard OpenSSL reads False at every new branch, so standard mode keeps
scrypt, inline, on the same rows.
Resolves LIT-8425
* fix(proxy): take FIPS logins one at a time and recheck the throttle first
A login that waited for the KDF thread asked the throttle again only after
deriving, so a burst for one user kept that thread busy long after the user
was blocked. Logins now wait for their turn on a lock and ask the throttle
when it comes, so a blocked login costs no PBKDF2 and only one login at a time
holds a place in the KDF queue. Password changes and invite claims no longer
wait behind a whole burst of logins.
* fix(proxy): keep PBKDF2 rows on a standard boot and warn when FIPS skips the breach check
A standard boot treated a pbkdf2: row as plaintext and hashed its text, so the
real password stopped working and the old row text signed in. The migration now
leaves a row of that exact shape alone. Any other pbkdf2: text still migrates.
FIPS mode now logs a warning when it skips a breach check that is turned on.
The one-thread test no longer depends on timing, and the new test variables are
annotated Final.
* fix(proxy): keep the breached password check on in FIPS mode
* fix(proxy): choose the password KDF from fips_mode instead of the host OpenSSL
* test(proxy): fake the scrypt probe in the FIPS fixture now that the gate checks scrypt
* fix(proxy): probe scrypt instead of MD5 in the fips_mode OpenSSL check
* test(proxy): fake a working scrypt so the fips_mode check tests pass on FIPS hosts
* feat(proxy): add LITELLM_FIPS_MODE startup gate
With LITELLM_FIPS_MODE=true the proxy refuses to start when TLS verification
is off, when Python's OpenSSL does not enforce FIPS, or when the startup
password migration fails. A value that is not true/false also refuses.
* feat(proxy): refuse FIPS mode when cryptography's OpenSSL is not FIPS
The stock cryptography wheel bundles its own non-FIPS OpenSSL, which serves
JWT, Fernet and AES-GCM even when Python's OpenSSL enforces FIPS.
* fix(proxy): refuse FIPS mode when cryptography's FIPS provider cannot hash
The stock cryptography wheel reads the host's FIPS config and reports FIPS on,
but its bundled OpenSSL cannot load the host's FIPS provider, so every
operation fails. Require a working SHA-256 alongside the FIPS flag.
* fix(proxy): refuse TLS verification off anywhere in FIPS mode
A model's own ssl_verify: false reached the HTTP client after the startup check
passed. get_ssl_verify now raises in FIPS mode whenever verification resolves
to off, which also covers the startup check.
* refactor(proxy): return FIPS boot refusals as values and harden the FIPS tests
* fix(proxy): stop refusing TLS verification off in FIPS mode
Skipping certificate verification is a security choice, not a FIPS one:
the TLS connection still only uses algorithms the FIPS provider allows.
* feat(fips): refuse a database used without FIPS mode, drop the password migration change
FIPS mode marks a new database with a fips_mode row in LiteLLM_Config and refuses one that already holds
LiteLLM rows without it. Without FIPS mode, the proxy refuses a database carrying the marker.
* test(fips): close the database test clients with aclose so the engine stops
* feat(fips): read FIPS mode from general_settings.fips_mode instead of LITELLM_FIPS_MODE
* refactor(fips): keep fips_mode out of the API-writable general settings, it is read from the config file only
* fix(fips): retry the startup marker and state reads once after a dropped connection
A dropped connection during the non-FIPS marker read used to count as an unmarked
database, so a blip at boot let a non-FIPS proxy start on a FIPS database. Both reads
now go through call_with_db_reconnect_retry, the reconnect-and-retry the other startup
reads use. A read that still fails keeps its old outcome.
* fix(fips): refuse a fips_mode key that has no value
fips_mode: os.environ/X with X unset, or a bare fips_mode: / null, resolved to None and
started without FIPS mode and without a message, while an empty string refused. A key
that is present now has to be a boolean; only a config without the key means off.
* refactor(fips): move the database check out of the boot gate
The boot gate keeps the setting and the two OpenSSL checks. The check that a
database matches the FIPS setting moves to its own PR, so a boot without the flag
runs no extra database read.
* test(fips): pin how a fips_mode string is read through enforce_fips_mode
* feat(proxy): LITELLM_FIPS_MODE startup gate with provider assertion, TLS verify guard and loud password migration
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(proxy): match ssl_verify off detection to runtime str_to_bool semantics
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(proxy): drop tautological fips probe test and satisfy CodeQL return checks
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(proxy): inject the fake Prisma client through the module boundary instead of patching _setup_prisma_client
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: yucheng <yucheng@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
