<?xml version="1.0" encoding="utf-8" standalone="yes"?><rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom"><channel><title>MSA on 热爱可抵万难</title><link>https://www.jingmengzhiyue.tech/en/tags/msa/</link><description>Recent content from 热爱可抵万难</description><generator>Hugo</generator><language>en</language><copyright>All articles on this blog are licensed under the BY license agreement unless otherwise stated. Please indicate the source when reprinting!</copyright><lastBuildDate>Thu, 28 Mar 2024 23:53:20 +0800</lastBuildDate><atom:link href="https://www.jingmengzhiyue.tech/en/tags/msa/index.xml" rel="self" type="application/rss+xml"/><item><title>The Multi-Head Self-Attention Mechanism in Transformers</title><link>https://www.jingmengzhiyue.tech/en/multi-head-self-attention/</link><pubDate>Thu, 28 Mar 2024 23:47:31 +0800</pubDate><guid>https://www.jingmengzhiyue.tech/en/multi-head-self-attention/</guid><description>
<![CDATA[<h1>The Multi-Head Self-Attention Mechanism in Transformers</h1><p>Author: Jingmengzhiyue()</p>
        
          <p>In natural language processing and other sequence-modeling tasks, the Transformer has become a research focus thanks to its strong performance. Multi-head self-attention is one of the Transformer&rsquo;s core components. What follows looks closely at how it works and how it is implemented.</p>
<h2 id="what-is-self-attention">
<a class="header-anchor" href="#what-is-self-attention"></a>
What Is Self-Attention
</h2><p>The self-attention mechanism lets the model, while encoding a sequence, gather information from the other positions for the representation at each position and update that representation from this information. Unlike recurrent networks such as RNNs and LSTMs, which capture dependence along a sequence step by step, self-attention models dependence between positions directly through attention scores. Modeling long-range dependence directly in this way greatly increases the model&rsquo;s expressive power.</p>
        
        <hr><p>Published on 2024-03-28 at <a href='https://www.jingmengzhiyue.tech/'>热爱可抵万难</a>, last modified on 2024-03-28</p>]]></description><category>Deep Learning</category></item></channel></rss>