Spaces:
Sleeping
Sleeping
| <html lang="en"> | |
| <head> | |
| <meta charset="UTF-8" /> | |
| <meta name="viewport" content="width=device-width, initial-scale=1.0"/> | |
| <title>DeepGaze</title> | |
| <link rel="stylesheet" href="static/styles.css" /> | |
| <!-- Load D3 --> | |
| <script src="https://d3js.org/d3.v6.min.js"></script> | |
| <style> | |
| /* Process button styling */ | |
| .text-form button { | |
| background-color: #800000; /* Bordo red */ | |
| color: #fff; | |
| border: none; | |
| padding: 8px 12px; | |
| font-size: 14px; | |
| border-radius: 4px; | |
| cursor: pointer; | |
| } | |
| /* Info container styling for head and layer display */ | |
| .info-container { | |
| display: flex; | |
| align-items: center; | |
| margin: 5px 0; | |
| font-size: 0.9rem; | |
| } | |
| .info-container .label { | |
| margin-right: 5px; | |
| } | |
| .info-container .number-box { | |
| border: 1px solid #800000; | |
| border-radius: 4px; | |
| padding: 2px 6px; | |
| font-weight: bold; | |
| color: #800000; | |
| min-width: 20px; | |
| text-align: center; | |
| } | |
| </style> | |
| </head> | |
| <body> | |
| <!-- Header --> | |
| <header> | |
| <h1>DeepGaze</h1> | |
| </header> | |
| <!-- Main container --> | |
| <div class="container"> | |
| <main> | |
| <!-- About Attention Section --> | |
| <section class="about-section"> | |
| <h2>About Attention</h2> | |
| <p> | |
| Transformer networks (the architecture behind Chat-GPT) are built from multiple layers, and each layer is divided into several attention heads. | |
| Each head computes its own attention matrix by combining "queries" and "keys"—the fundamental elements that help | |
| the network decide how much focus to give to different parts of the input. | |
| </p> | |
| <p> | |
| You can think of each query as a question that a token asks, such as "Are there adjectives in front of me?" | |
| Meanwhile, each key serves as a potential answer, carrying the token's characteristics. When the model compares | |
| queries with keys, it determines the strength of their match and, therefore, how much influence one token should | |
| have on another. | |
| </p> | |
| <p> | |
| For example, consider the phrase "fluffy blue monster." One token might generate a query like, "Is the word in front | |
| of me an adjective?" In this case, the tokens "fluffy" and "blue"—which are adjectives—provide keys that answer this | |
| question strongly, while "monster," being a noun, offers a weaker response. This interplay of questions (queries) | |
| and answers (keys) is what creates the attention matrix for each head. | |
| </p> | |
| <p> | |
| Each attention head focuses on different relationships and patterns within the text, allowing the network to capture | |
| a rich and nuanced understanding of the language. Despite the critical role that these attention mechanisms play, | |
| it's interesting to note that only about one third of all the weights in a large language model are actually in the | |
| attention blocks. So while the famous slogan "attention is all you need" highlights the importance of these connections, | |
| in terms of sheer weight, it's only one third of what you really need! | |
| </p> | |
| <p id="credits">Made with <3 by Ferdi & Samu. Credits for model view below to <a href="https://jessevig.com/" target="_blank">BertViz</a>.</p> | |
| </section> | |
| <!-- Deep Gaze into Attention Heads Section --> | |
| <section class="deep-gaze-section"> | |
| <h2>A Deep Gaze into Attention Heads</h2> | |
| <p>Type in a token sequence (below 50 characters) and hit process. After some loading time, you will be able to see the attention patterns of individual so-called "heads" in the LLM. | |
| Each head focuses on different aspects of the input text, and by visualizing these patterns, you can gain insights into how the model processes and understands language. | |
| </p> | |
| <div id="thumbnailContainer"></div> | |
| <p>Here is an example view of a head, with tokens on each side. If you see a connection between two tokens, it means that the head is paying attention to the relationship between those tokens. This way you can see attention heads which "pay attention" to the previous token, the first token, or other patterns. | |
| Click on an attention head to select the respective head in a layer. Afterwards you can hover over tokens to see the attention weights of the selected head for that token. </p> | |
| <section id="display_info"> | |
| <div class="info-container"> | |
| <span class="label">Head:</span> | |
| <span class="number-box" id="display_head">-</span> | |
| <span class="spacer"></span> | |
| <span class="label">Layer:</span> | |
| <span class="number-box" id="display_layer">-</span> | |
| </div> | |
| </section> | |
| </section> | |
| <!-- Text Input & Process Button --> | |
| <form id="textForm" class="text-form"> | |
| <textarea id="inputText" rows="2" cols="50" maxlength="50" placeholder="Enter your text here..." autofocus></textarea> | |
| <button type="submit">Process</button> | |
| </form> | |
| </section> | |
| <!-- Model View Section --> | |
| <section class="model_view"> | |
| <p>Click on a head that looks interesting to gaze deeper into it in the next section:</p> | |
| <div id="model_view_container"> | |
| <!-- Thumbnails of attention heads will be rendered here --> | |
| </div> | |
| <!-- Display for selected head and layer --> | |
| <div id="display_info"> | |
| <!-- <div id="display_head" class="info-container"> | |
| <span class="label">Head:</span> | |
| <span class="number-box">-</span> | |
| </div> | |
| <div id="display_layer" class="info-container"> | |
| <span class="label">Layer:</span> | |
| <span class="number-box">-</span> | |
| </div> --> | |
| </div> | |
| </section> | |
| <!-- Hover Visualization Section --> | |
| <section class="hover-visualization"> | |
| <h2>Hover Visualization</h2> | |
| <p> | |
| By hovering over each token, you can see which other token is important for that token. The larger the token, the more important it is for the token you are hovering over. The token with the maximal attention is colored in red. | |
| </p> | |
| <div id="tokenContainer"></div> | |
| </section> | |
| </main> | |
| </div> | |
| <!-- External JavaScript (all event listeners and functions are defined here) --> | |
| <script src="static/script.js?v=1111"></script> | |
| </body> | |
| </html> |