TransformerFAM: Feedback attention is working memory
#transformer #rnn
Разобрал одну из новых архитектур трансформеров. Меня привлекла идея реализовать rnn через архитектуру attention и применение feedback токенов.
Работает все довольно просто - будем разбивать текст на блоки токенов и накапливать информацию каждого блока в обучаемых feedback токенах. А дальше каждый feedback токен помимо текстовой информации, будет брать значение предыдущего feedback токена (на этом моменте я вспомнил старые добрые лекции Andrew Ng о rnn 🚶♀️)
Заметил, что тема rnn становится популярна в последнее время (отсылка к новой статье xLSTM, которую я разберу позднее⏰)
TransformerFAM: Feedback attention is working memory
#transformer #rnn
Разобрал одну из новых архитектур трансформеров. Меня привлекла идея реализовать rnn через архитектуру attention и применение feedback токенов.
Работает все довольно просто - будем разбивать текст на блоки токенов и накапливать информацию каждого блока в обучаемых feedback токенах. А дальше каждый feedback токен помимо текстовой информации, будет брать значение предыдущего feedback токена (на этом моменте я вспомнил старые добрые лекции Andrew Ng о rnn 🚶♀️)
Заметил, что тема rnn становится популярна в последнее время (отсылка к новой статье xLSTM, которую я разберу позднее⏰)
At the start of 2018, the company attempted to launch an Initial Coin Offering (ICO) which would enable it to enable payments (and earn the cash that comes from doing so). The initial signals were promising, especially given Telegram’s user base is already fairly crypto-savvy. It raised an initial tranche of cash – worth more than a billion dollars – to help develop the coin before opening sales to the public. Unfortunately, third-party sales of coins bought in those initial fundraising rounds raised the ire of the SEC, which brought the hammer down on the whole operation. In 2020, officials ordered Telegram to pay a fine of $18.5 million and hand back much of the cash that it had raised. And indeed, volatility has been a hallmark of the market environment so far in 2022, with the S&P 500 still down more than 10% for the year-to-date after first sliding into a correction last month. The CBOE Volatility Index, or VIX, has held at a lofty level of more than 30. Just days after Russia invaded Ukraine, Durov wrote that Telegram was "increasingly becoming a source of unverified information," and he worried about the app being used to "incite ethnic hatred." Pavel Durov, Telegram's CEO, is known as "the Russian Mark Zuckerberg," for co-founding VKontakte, which is Russian for "in touch," a Facebook imitator that became the country's most popular social networking site. The SC urges the public to refer to the SC’s I nvestor Alert List before investing. The list contains details of unauthorised websites, investment products, companies and individuals. Members of the public who suspect that they have been approached by unauthorised firms or individuals offering schemes that promise unrealistic returns
from ar
