ai21.net
语言

Multi-head attention heatmaps

Multi-head attention heatmaps — 1

提示词

Landscape 16:9 figure of 4 attention heatmaps (2×2 grid), shared 12-token input.

Token labels across X and Y (rotated 45° on X): "The", "quick", "brown", "fox", "jumped", "over", "the", "lazy", "dog", "near", "the", "river".

Four 12×12 cell panels with individual titles:
"Layer 6, Head 3 — subject-verb" (highlighted cells between "fox"/"jumped")
"Layer 9, Head 7 — coreference" (highlighted cells between "the"(×2)/"river")
"Layer 11, Head 2 — prepositional" (highlighted cells between "over"/"dog", "near"/"river")
"Layer 14, Head 1 — sentence-final" (activity concentrated in rightmost column)

Cells: dusty-teal gradient, darker = higher weight. Peak cells outlined in 1px soft-terracotta. Shared vertical color bar on far right with ticks "0.0", "0.25", "0.5", "0.75", "1.0" and label "attention weight".

Title: "Representative multi-head attention patterns in a 16-layer Transformer". Subtitle: "four of 256 heads, hand-picked for illustrative head-role diversity; inspired by Clark et al., 2019."

中文译文

16:9 横版图,包含 4 张注意力热力图(2×2 网格),共享同一个 12 个 token 的输入。

X 轴与 Y 轴上的 token 标签(X 轴旋转 45°):“The”、“quick”、“brown”、“fox”、“jumped”、“over”、“the”、“lazy”、“dog”、“near”、“the”、“river”。

四个 12×12 单元格面板,各有独立标题:
“第 6 层,第 3 头 —— 主谓关系”(“fox”/“jumped” 之间的单元格被高亮)
“第 9 层,第 7 头 —— 指代关系”(“the”(×2)/“river” 之间的单元格被高亮)
“第 11 层,第 2 头 —— 介词短语”(“over”/“dog”、“near”/“river” 之间的单元格被高亮)
“第 14 层,第 1 头 —— 句末”(活跃度集中在最右列)

单元格:灰青色渐变,颜色越深 = 权重越高。峰值单元格以 1px 浅陶土色描边。最右侧有一条共享的垂直色条,刻度为“0.0”、“0.25”、“0.5”、“0.75”、“1.0”,标签为“注意力权重”。

标题:“16 层 Transformer 中具有代表性的多头注意力模式”。副标题:“256 个头中的 4 个,为展示注意力头角色多样性而人工挑选;灵感来自 Clark et al., 2019。”

溯源

原作者
wuyoscar
收录来源