Fine-tuning, prompting, retrieval, alignment and agent safety, with a sonnet lab
|
tHE Next Word Problem - Unit 4 (pdf)
|
467.11 K |
A pre-trained model has a training cutoff, no access to private documents and no source for its answers. In fine-tuning, the weights are changed with hundreds to thousands of labelled examples. In prompting, zero to about ten examples are placed in the input. Sampling is controlled by temperature, a rescaling of the logits before the softmax, by top-k, a fixed number of candidates, and by nucleus sampling, the smallest set of words covering a probability p. In retrieval-augmented generation, documents are retrieved at answer time. RLHF has three steps, supervised fine-tuning on demonstrations, a reward model fitted to human preference comparisons and policy optimisation against that reward model. The risk surface grows with the autonomy of an agent, from read-only answering to multi-step execution without review. In a lab, sonnet lines are written by rejection sampling, and 800 attempts give four different lines with ten syllables and a rhyme.