# Lost in the Middle — Original Paper Reproduction Exact reproduction of the experiments from Liu et al. (2023) "Lost in the Middle: How Language Models Use Long Contexts" (arXiv:2307.03172). **Original Repo:** https://github.com/nelson-liu/lost-in-the-middle ## What's Included All files fetched directly from the official GitHub repository via `raw.githubusercontent.com`. ### Core Library - `src/prompting.py` — Prompt construction (exact from `src/lost_in_the_middle/prompting.py`) - `src/metrics.py` — Scoring/normalization (exact from `src/lost_in_the_middle/metrics.py`) ### Prompt Templates - `prompts/qa.prompt` - `prompts/kv_retrieval.prompt` - `prompts/closedbook_qa.prompt` - `prompts/qa_with_query_aware_contextualization.prompt` - `prompts/kv_retrieval_with_query_aware_contextualization.prompt` ### Data Generation - `scripts/make_qa_data_from_retrieval_results.py` — Multi-document QA data creation - `scripts/create_kv_pairs.py` — KV retrieval synthetic data generation ### Experiment Runners - `scripts/get_qa_responses_from_mpt.py` — Multi-document QA runner (MPT-30B original) - `scripts/get_kv_responses_from_mpt.py` — KV retrieval runner (MPT-30B original) ### Mistral-7B Adapters - `scripts/run_qa_mistral.py` — Multi-document QA on Mistral-7B-Instruct - `scripts/run_kv_mistral.py` — KV retrieval on Mistral-7B-Instruct ## Requirements ```bash pip install transformers torch tqdm regex pydantic xopen ``` ## Quick Start: KV Retrieval on Mistral 7B ```bash # 1. Generate synthetic KV data python scripts/create_kv_pairs.py \ --num-keys 100 \ --num-examples 50 \ --output data/kv_100.jsonl.gz # 2. Run evaluation python scripts/run_kv_mistral.py \ --input-path data/kv_100.jsonl.gz \ --model mistralai/Mistral-7B-Instruct-v0.3 \ --gold-index 50 \ --output results/kv_100_gold50.jsonl.gz ``` ## Paper Liu, N. F., Lin, K., Hewitt, J., Paranjape, A., Bevilacqua, M., Petroni, F., & Liang, P. (2023). Lost in the Middle: How Language Models Use Long Contexts. arXiv:2307.03172.
