• Contents:

LAB: Exploring Transformers in a SageMaker Notebook

PART 1: Tokenization and Positional Encoding

Ref: https://www.udemy.com/course/aws-certified-machine-learning-engineer-associate-mla-c01/learn/lecture/45285951

• Create a cheap notebook instance
• Once instance is ready, import the provided notebook file into the Jupyter notebook
• Install transformers package from Hugging Face, then follow notebook steps in order
• We use BERT model and tokenizer in the demo
• Notice how “I read a good novel” is tokenized into sequences of integers
• Notice the positional encoding function uses an interleaved sinusoidal function

PART 2: Multi-Headed, Masked Self-Attention

Ref: https://www.udemy.com/course/aws-certified-machine-learning-engineer-associate-mla-c01/learn/lecture/45285963

• Observe that “novel” in “I read a good novel” has a big connection to “read”
• For “Attention is a novel idea”, “novel” is tied a lot to “idea”
• Can play a lot more (without a notebook even) at https://huggingface.co/spaces/exbert-project/exbert

PART 3: Import GPT-2 from Hugging Face and generate text

Ref: https://www.udemy.com/course/aws-certified-machine-learning-engineer-associate-mla-c01/learn/lecture/45286023

• In just 3 lines of code, you can import GPT-2 and make it generate text from a prompt