ai21.net
語言

Multi-head attention heatmaps

Multi-head attention heatmaps — 1

提示詞

Landscape 16:9 figure of 4 attention heatmaps (2×2 grid), shared 12-token input.

Token labels across X and Y (rotated 45° on X): "The", "quick", "brown", "fox", "jumped", "over", "the", "lazy", "dog", "near", "the", "river".

Four 12×12 cell panels with individual titles:
"Layer 6, Head 3 — subject-verb" (highlighted cells between "fox"/"jumped")
"Layer 9, Head 7 — coreference" (highlighted cells between "the"(×2)/"river")
"Layer 11, Head 2 — prepositional" (highlighted cells between "over"/"dog", "near"/"river")
"Layer 14, Head 1 — sentence-final" (activity concentrated in rightmost column)

Cells: dusty-teal gradient, darker = higher weight. Peak cells outlined in 1px soft-terracotta. Shared vertical color bar on far right with ticks "0.0", "0.25", "0.5", "0.75", "1.0" and label "attention weight".

Title: "Representative multi-head attention patterns in a 16-layer Transformer". Subtitle: "four of 256 heads, hand-picked for illustrative head-role diversity; inspired by Clark et al., 2019."

中文翻譯

16:9 橫版圖,包含 4 張注意力熱力圖(2×2 網格),共享同一個 12 個 token 的輸入。

X 軸與 Y 軸上的 token 標籤(X 軸旋轉 45°):“The”、“quick”、“brown”、“fox”、“jumped”、“over”、“the”、“lazy”、“dog”、“near”、“the”、“river”。

四個 12×12 單元格面板,各有獨立標題:
“第 6 層,第 3 頭 —— 主謂關係”(“fox”/“jumped” 之間的單元格被高亮)
“第 9 層,第 7 頭 —— 指代關係”(“the”(×2)/“river” 之間的單元格被高亮)
“第 11 層,第 2 頭 —— 介詞短語”(“over”/“dog”、“near”/“river” 之間的單元格被高亮)
“第 14 層,第 1 頭 —— 句末”(活躍度集中在最右列)

單元格:灰青色漸變,顏色越深 = 權重越高。峯值單元格以 1px 淺陶土色描邊。最右側有一條共享的垂直色條,刻度爲“0.0”、“0.25”、“0.5”、“0.75”、“1.0”,標籤爲“注意力權重”。

標題:“16 層 Transformer 中具有代表性的多頭注意力模式”。副標題:“256 個頭中的 4 個,爲展示注意力頭角色多樣性而人工挑選;靈感來自 Clark et al., 2019。”

溯源

原作者
wuyoscar
收錄來源