- Encoder + decoder - Semi-supervised learning - Initially trained unsupervised on data - Fine-tuned using supervised learning - Attention - allows for data to be processed out of sequence - offers context [[Attention Is All You Need]]