출처 1) Dissecting BERT Part 1: Understanding the Transformer 2) Dissecting BERT Part2: BERT Specifics Part 1 1. Encoder Level 1 : Problem - Transformer의 과제는 번역하기 - Encoder : 입력 문장(출발어) 안의 단어끼리 관계 포착하기 - Decoder : [입력 문장의 정보] + [번역된 단어, 출력문장(도착어)] Level 2 : Flow of Information 1) 각 토큰을 embedding 벡터로 바꾼다. 따라서 전체 input 문장은 (input_length) * (embedding_dim) 크기의 행렬이 된다 2) 이 embedding에 단어의 위치 정보(positio..

http://jalammar.github.io/illustrated-bert/ The Illustrated BERT, ELMo, and co. (How NLP Cracked Transfer Learning) Discussions: Hacker News (98 points, 19 comments), Reddit r/MachineLearning (164 points, 20 comments) Translations: Chinese (Simplified), Persian The year 2018 has been an inflection point for machine learning models handling text (or more accurately, Natu jalammar.github.io BERT (..

1) Seq2Seq Pay Attention to Self Attention: Part I 2) Seq2seq Pay Attention to Self Attention: Part 2 Seq2seq vs. Attention Model - Seq2seq : 하나의 고정길이인 context vector - Attention Model : n개의 context vector ( n = input 문장에 사용된 총 단어의 개수) Attention Model의 context vector 계산하는 방법 - α : attention score (단어의 중요도를 의미함) - h : 은닉 상태 - 모든 은닉상태 * attention score = context vector Attention score, α 계산하는 방법 -..

https://jalammar.github.io/illustrated-transformer/ The Illustrated Transformer Discussions: Hacker News (65 points, 4 comments), Reddit r/MachineLearning (29 points, 3 comments) Translations: Chinese (Simplified), Korean Watch: MIT’s Deep Learning State of the Art lecture referencing this post In the previous post, we looked at Atten jalammar.github.io 1) Encoder - 첫번째(맨 아래) Encoder만 word embed..

https://pytorch.org/tutorials/intermediate/seq2seq_translation_tutorial.html NLP From Scratch: Translation with a Sequence to Sequence Network and Attention — PyTorch Tutorials 1.4.0 documentation Note Click here to download the full example code NLP From Scratch: Translation with a Sequence to Sequence Network and Attention Author: Sean Robertson This is the third and final tutorial on doing “N..

출처 1) Neural Machine Translation By Jointly Learning to Align and Translate 2) Attention: Illustrated Attention 3) Attention and Memory in Deep Learning and NLP 기존 Encoder-Decoder RNN/LSTM 모델의 문제점 - 아무리 긴 input sentence가 주어져도 고정 길이 벡터fixed-length vector로 압축해서 표현해야 함 - Decoder는 Encoder의 마지막 은닉상태만 전달받음 → 엄청 긴 문장이라면 엄청 많이 까먹음 기존 Encoder-Decoder RNN/LSTM 모델의 문제점 해결 - 고정길이벡터 X - input sentence는 여러 벡터..