WebGPT deep reading
Series · 1 posts
-
WebGPT: Let the Model Browse for Answers, but Do Not Treat It as a Reasoning Agent Loop
Intermediate Agent runtime, safety, and evaluationA source-grounded reading of Nakano et al., arXiv:2112.09332 v3: GPT-3 is given a text browser and trained with human demonstrations and preference / reward modeling to search, quote, and answer. The 175B best-of-64 model is preferred 56% versus demonstrators and 69% versus Reddit; this is browsing QA, not ReAct’s thought–action–observation contract.
Understand it in 90 seconds
- Problem
- Long-form question answering lags humans. Retrieval and synthesis were built as separate pieces. Without citations, people cannot cheaply check paragraph-level facts.
- Core insight
- Outsource search to Bing, leave synthesis to a finetuned GPT-3, and insert a text browser between them. The model may only issue Table 1 commands (search, click, quote, scroll, end), collect quotes while browsing, then write the answer. Training is behavior cloning from human demonstrations, plus a reward model from human preferences, plus rejection sampling at inference.
- Strongest evidence
- 175B best-of-64 is preferred 56% to demonstrators and 69% to the highest-voted ELI5 answers (Section 4.1, Figure 2). Best-of-64 is preferred 68% to plain BC; RL is preferred 58% to BC, but stacking RL on rejection sampling adds little (Section 5.1, Figures 4 and 5).
- Main boundary
- There is no separate thought action. The text browser is a constrained action space, not a general tool loop. Answers can still mis-paraphrase quotes or cherry-pick sources that look convincing to labelers. This is a 2021 OpenAI technical report / arXiv preprint, not a later production browsing product.
For speaking invitations, internal engineering sessions, or architecture exchange, see the topics and public work I can bring into the conversation.
Speaking & contact