CV-Attention

Attention Mechanism

Attention Score:

• $q_1=W^q{a}_1$
• $k_1=W^k{a}_1$
• $k_2=W^k{a}_2$
• $k_3=W^k{a}_3$
• $k_4=W^k{a}_4$

$a_{1,1}=q_1{k}_1$$a_{1,2}=q_1{k}_2$$a_{1,3}=q_1{k}_3$$a_{1,4}=q_1{k}_4$

$$q_i=W^q{a}_i$$$$k_i=W^k{a}_i$$$$v_i=W^v{a}_i$$

$$A=K^{T}Q$$$$A^{\prime }=softmax(A)$$

$$O=V^T{A}^{\prime }$$

Take away:

$$Q=W^{q}I$$$$K=W^{k}I$$$$V=W^{v}I$$$$A=K^{T}Q$$$$A^{\prime }=softmax(A)$$$$O=V^T{A}^{\prime }$$

Multi-Head Attention

Transformer