GPT 정리

1. GPT (Generative Pre-Training) • goal: learn a universal representation • generative pre-training (unlabeled text) + discriminative fine-tuning (labeled text) 1.1. Unsupervised pre-training 1.2. Supervised fine-tuning 2. GPT-2 • difference from BERT GPT-2 BERT Direction uni-directional auto-regression mask future tokens bi-directional Tokenizer BPE(Byte-pair Encoding) WordPiece Tokenizer Fine-..

2020.10.07