(docs) lm harness

This commit is contained in:
ishaan-jaff 2023-11-08 17:21:12 -08:00
parent 53abc31c27
commit 68e668df13

View file

@ -41,6 +41,31 @@ python3 -m lm_eval \
```
## FLASK - Fine-grained Language Model Evaluation Based on Alignment Skill Sets
Use litellm to evaluate any LLM on FLASK https://github.com/kaistAI/FLASK
**Step 1: Start the local proxy**
```shell
$ litellm --model huggingface/bigcode/starcoder
```
**Step 2: Set OpenAI API Base & Key**
```shell
$ export OPENAI_API_BASE=http://0.0.0.0:8000
```
**Step 3 Run with FLASK**
```shell
cd FLASK
cd gpt_review
```
Run the eval
``shell
python gpt4_eval.py -q '../evaluation_set/flask_evaluation.jsonl'
```
## Debugging
### Making a test request to your proxy