all(408)
-
Micro- and Macro F1 scores
1. Micro Averaged Metrics 2. Macro Averaged Metrics
2020.10.07 -
Longformer: The Long-Document Transformer 논문 정리
Longformer: The Long-Document Transformer Longformer Transformer complexity O(n^2) scales linearly O(n) scales quadratically attention local windowed attention + global attention (d) self-attention (a) max length 4,096 512 [ Attention Pattern ] 1) Sliding Window • fixed-size window attention for local context • complexity: O(n × w) • n: input sequence length • w: fixed window size (layer마다 달라질 수..
2020.10.07 -
GPT 정리
1. GPT (Generative Pre-Training) • goal: learn a universal representation • generative pre-training (unlabeled text) + discriminative fine-tuning (labeled text) 1.1. Unsupervised pre-training 1.2. Supervised fine-tuning 2. GPT-2 • difference from BERT GPT-2 BERT Direction uni-directional auto-regression mask future tokens bi-directional Tokenizer BPE(Byte-pair Encoding) WordPiece Tokenizer Fine-..
2020.10.07 -
BLEU and BLEURT: evaluation for text generation 정리
BLEU and BLEURT 1. BLEU (Bilingual Evaluation Understudy, 2002) “The closer a machine translation is to a professional human translation, the better it is.” 0-1 사이의 score - BP: Brevity Penalty - p_n: modified n-gram precision - w_n: positive weights (baseline: 1/N) - N: n-gram의 전체 길이 (baseline: N=4) 1.1. modified n-gram precision - unigram precision: 7/7 - (candidate 토큰 중 reference에도 있는 토큰의 개수) ..
2020.10.07 -
StructBERT: Incorporating Language Structures into Pretraining for Deep Language Understandin 논문 정리
StructBERT: incorporated language structures into pre-training (사실상 language structure라기 보단 어순) 1) word-level ordering 2) sentence-level ordering 1) word-level ordering : 기존 BERT처럼 일부 토큰 masking 후, masked되지 않은 토큰 3개(trigram) 골라 순서 섞기 * 4개로 했을 때 성능 차이가 크지 않았고, robustness 고려하여 3개로 선택 → masked된 토큰의 final hidden state → softmax classifier → 본래 토큰 예측 → shuffled된 토큰들의 final hidden state → softmax clas..
2020.10.07 -
[rclone] command line에서 서버 파일을 google drive로 옮기기
rclone.org/downloads/ Rclone downloads © Nick Craig-Wood 2014-2020 Source file downloads.md last updated 2020-08-18 Website hosted on a MEMSET CLOUD VPS, uploaded with rclone and built with Hugo. Logo by @andy23. rclone.org # rclone 다운 받기 curl https://rclone.org/install.sh | sudo bash # config 수정 rclone config # name은 필수로 지정해야 함 # 그 외 모든 건 디폴트 값으로 둬도 무방 # root_folder_id 는 설정해주면 편함 # auto config는..
2020.09.17