ACL 2023 Tutorial:
Retrieval-based Language Models and Applications

Schedule

Our tutorial will be held on July 9 (all the times are based on EDT = Toronto local time). Slides may be subject to updates.

Reading List

Bold papers are discussed in detail during our tutorial.

Section 3: Architecture

• REALM: Retrieval-Augmented Language Model Pre-Training (Guu et al., 2020)
• In-Context Retrieval-Augmented Language Models (Ram et al., 2023)
• REPLUG: Retrieval-Augmented Black-Box Language Models (Shi et al., 2023)
• Improving language models by retrieving from trillions of tokens (Borgeaud et al., 2022)
• Generalization through Memorization: Nearest Neighbor Language Models (Khandelwal et al., 2020)
• Active Retrieval Augmented Generation (Jiang et al., 2023)
• Efficient Nearest Neighbor Language Models (He et al., 2021)
• You can't pick your neighbors, or can you? When and how to rely on retrieval in the kNN-LM (Drozdov et al., 2022)
• Neuro-Symbolic Language Modeling with Automaton-augmented Retrieval (Alon et al., 2022)
• Entities as Experts: Sparse Memory Access with Entity Supervision (Févry et al., 2020)
• Mention Memory: incorporating textual knowledge into Transformers through entity mention attention (de Jong et al., 2021)
• Memorizing Transformers (Wu et al., 2022)
• Unlimiformer: Long-Range Transformers with Unlimited Length Input (Bertsch et al. 2023)
• Long-range Language Modeling with Self-retrieval (Rubin & Brent, 2023)

Section 4: Training

• Dense Passage Retrieval for Open-Domain Question Answering (Karpukhin et al., 2020)
• Improving language models by retrieving from trillions of tokens (Borgeaud et al., 2022 ;also in Section 3)
• Atlas: Few-shot Learning with Retrieval Augmented Language Models (Izacard et al., 2022)
• Training Language Models with Memory Augmentation (Zhong et al., 2022)
• Unsupervised Dense Information Retrieval with Contrastive Learning (Izacard et al., 2022)
• In-Context Retrieval-Augmented Language Models (Ram et al., 2023; also in Section 3)
• REPLUG: Retrieval-Augmented Black-Box Language Models (Shi et al., 2023; also in Section 3)
• REALM: Retrieval-Augmented Language Model Pre-Training (Guu et al., 2020; also in Section 3)
• Nonparametric Masked Language Modeling (Min et al., 2023)
• Long-range Language Modeling with Self-retrieval (Rubin et al., 2023)

Section 5: Application

• Atlas: Few-shot Learning with Retrieval Augmented Language Models (Izacard et al., 2022; also in Section 4)
• Teaching language models to support answers with verified quotes (Menick et al., 2022)
• REPLUG: Retrieval-Augmented Black-Box Language Models (Shi et al., 2023; also in Section 3)
• kNN-Prompt: Nearest Neighbor Zero-Shot Inference (Shi et al., 2022)
• Attributed Question Answering: Evaluation and Modeling for Attributed Large Language Models (Bohnet et al., 2023)
• DocPrompting: Generating Code by Retrieving the Docs (Zhou et al., 2022)
• When Not to Trust Language Models: Investigating Effectiveness of Parametric and Non-Parametric Memories (Mallen et al., 2022)
• Nonparametric Masked Language Modeling (Min et al., 2023; also in Section 4)
• FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generation (Min et al., 2023)

Section 6: Extension

• One Question Answering Model for Many Languages with Cross-lingual Dense Passage Retrieval (Asai et al., 2021)
• Retrieval-Augmented Multimodal Language Modeling (Yasunaga et al., 2023)

Section 7: Challenges & Opportunities

• KNN-LM Does Not Improve Open-ended Text Generation (Wang et al., 2023)
• Can Retriever-Augmented Language Models Reason? The Blame Game Between the Retriever and the Language Model (BehnamGhader et al., 2022)
• Demonstrate-Search-Predict: Composing retrieval and language models for knowledge-intensive NLP (Khattab et al., 2022)