Upload folder using huggingface_hub

.gitignore

@@ -1,3 +1,7 @@
+# Additional
+.DS_Store
+.DS_Store/
+
 # Byte-compiled / optimized / DLL files
 __pycache__/
 *.py[cod]

main_rag.ipynb

@@ -369,8 +369,8 @@
   {
    "metadata": {
     "ExecuteTime": {
-     "end_time": "2024-10-02T08:33:42.372123Z",
-     "start_time": "2024-10-02T08:33:42.334621Z"
+     "end_time": "2024-10-02T14:30:10.388569Z",
+     "start_time": "2024-10-02T14:30:10.348317Z"
     }
    },
    "cell_type": "code",
@@ -386,37 +386,49 @@
     "from dotenv import load_dotenv\n",
     "\n",
     "\n",
-    "system_prompt = \"\"\"Task: Generate comprehensive documentation based on the provided documents. The documentation should capture all essential details and information without any information loss, including relevant code fragments from the files. Ensure that the output does not cut corners on tokens; generate as much content as possible within the limits of the provided documents. Do not reference or include information from any sources other than the given documents, do not rely on your knowledge.\n",
+    "system_prompt = \"\"\"You are an expert assistant skilled in generating structured, comprehensive documentation. Your role is to create thorough, accurate documentation based strictly on the provided context, without using external knowledge or general information. The content must be divided into two main sections: one for non-technical users and one for technical users, with each section tailored to their needs. The output should be clear, detailed, and organized, including code snippets, examples, and conflict resolution where applicable. Your final output should be ready in Markdown format.\n",
+    "\"\"\"\n",
     "\n",
-    "Requirements:\n",
     "\n",
-    "1. Length: The documentation should be detailed and extensive, covering all aspects of the content provided in the documents.\n",
+    "template = \"\"\"**Task:**  \n",
+    "Generate comprehensive documentation based solely on the provided documents. **Do not use any external knowledge** or information outside of the provided context. Divide the documentation into two main sections:\n",
     "\n",
-    "2. Structure:\n",
-    "   - Title Page: Include a title that reflects the main theme of the documents.\n",
-    "   - Introduction: Provide a complete and thorough overview of the content, purpose, and scope of the documentation.\n",
-    "   - Main Sections: Divide the documentation into logical sections and subsections based on the themes and topics found in the documents. Each section should include:\n",
-    "     - A clear, descriptive heading that reflects the content within the section.\n",
-    "     - Detailed explanations of key concepts and ideas. Expand on each idea or concept with thorough descriptions, definitions, and context. Ensure the reader fully understands the material.\n",
-    "     - Code fragments: Where applicable, include relevant code snippets from the documents. These should be properly formatted and explained in detail. For each code snippet, provide a clear explanation of its purpose, functionality, and how it fits into the overall system or process described in the document.\n",
-    "     - Examples: Provide relevant examples where applicable, explaining them fully to ensure understanding.\n",
-    "     - Conflict Resolution: If there are contradictory pieces of information across different documents, resolve them using the date provided at the beginning of each document, prioritizing the most recent information. If a document’s date is unknown and conflicts cannot be resolved, explicitly indicate which parts contradict one another, then clearly explain the contradiction. Mention all unresolved conflicts at the end of each section.\n",
+    "- **Non-Technical User:** Focus on explanations that are accessible to non-technical stakeholders (e.g., **business decision-makers or managers**). Highlight the purpose, value, and practical outcomes, avoiding technical jargon. Use examples and high-level explanations to ensure understanding.\n",
     "\n",
-    "3. Content Guidelines:\n",
-    "   - Ensure that all information from the documents is included and accurately represented.\n",
-    "   - The content should not just list bullet points but should be expanded into full sentences and paragraphs where necessary, with elaboration and explanations for all points.\n",
-    "   - Use clear and logical transitions between sections to maintain the flow and coherence of the document.\n",
-    "   - Use bullet points, tables, code fragments, or diagrams where appropriate to enhance understanding and readability, but always accompany these elements with in-depth explanations.\n",
+    "- **Technical User:** Provide in-depth technical content for an audience of **developers, engineers, or system architects**. Include detailed explanations of key technical concepts, code snippets with thorough descriptions, and examples where applicable. Explain the functionality and purpose of each code fragment, along with any potential issues or caveats.\n",
     "\n",
-    "4. Style: The writing style should be professional, formal, and suitable for an audience that may include technical experts, stakeholders, or general readers interested in the topic. Avoid jargon unless it is explained in detail.\n",
+    "**Requirements:**\n",
     "\n",
-    "5. Specifics: Highlight any critical findings, data, or statistics that are present in the documents. For each significant point, explain its relevance and implications. Emphasize any unique or innovative aspects that stand out, and provide complete reasoning or analysis where needed.\n",
+    "1. **Length:** The documentation should be detailed and cover all aspects of the provided content.\n",
     "\n",
-    "Output Format: Ensure that the entire documentation is generated in Markdown format without any additional text or formatting. The output should be ready to be saved directly into a markdown file.\n",
-    "\"\"\"\n",
+    "2. **Structure:**  \n",
+    "   - **Title Page:** Include a title reflecting the main theme of the documents.  \n",
+    "   - **Introduction:** Provide a clear overview of the content, purpose, and scope of the documentation.  \n",
+    "   - **Non-Technical User Section:**  \n",
+    "     - Clear, high-level explanations that avoid technical depth.  \n",
+    "     - Emphasize the **purpose, benefits, and outcomes** for non-technical users.  \n",
+    "     - Include examples and **highlight key points** with bold text where necessary.  \n",
+    "   - **Technical User Section:**  \n",
+    "     - Detailed explanations of key technical concepts.  \n",
+    "     - Include well-explained code snippets, with a focus on their **practical application** and any **potential pitfalls**.  \n",
+    "     - **Conflict Resolution:** If conflicting information is present, resolve it using the document date or highlight unresolved contradictions **inline** within the relevant sections.\n",
+    "\n",
+    "3. **Content Guidelines:**  \n",
+    "   - Represent all information accurately from the documents.  \n",
+    "   - Expand bullet points into full sentences and paragraphs.  \n",
+    "   - Use bullet points, tables, or code fragments where necessary, with in-depth explanations.  \n",
+    "   - Ensure smooth transitions between sections.\n",
+    "\n",
+    "4. **Style:**  \n",
+    "   - The non-technical section should be accessible and easy to understand.  \n",
+    "   - The technical section should be professional and formal, with precise technical language.\n",
+    "\n",
+    "5. **Specifics:**  \n",
+    "   - Highlight any critical findings, data, or statistics from the documents.  \n",
+    "   - Emphasize unique aspects with full reasoning and analysis.  \n",
+    "   - If any sections lack sufficient detail, indicate **gaps or missing information** rather than assuming or fabricating content.\n",
     "\n",
     "\n",
-    "template = \"\"\"\n",
     "User Question: {{question}}\n",
     "Documents to Analyze:\n",
     "{% for doc in documents %}\n",
@@ -476,7 +488,7 @@
    ],
    "id": "d57a03567a4c79a8",
    "outputs": [],
-   "execution_count": 5
+   "execution_count": 9
   },
   {
    "metadata": {},
@@ -487,13 +499,13 @@
   {
    "metadata": {
     "ExecuteTime": {
-     "end_time": "2024-10-01T16:22:14.031811Z",
-     "start_time": "2024-10-01T16:21:52.967813Z"
+     "end_time": "2024-10-02T14:39:37.670209Z",
+     "start_time": "2024-10-02T14:39:06.402337Z"
     }
    },
    "cell_type": "code",
    "source": [
-    "query = \"What if I have a team larger than 500 users?\"\n",
+    "query = \"Federations\"\n",
     "result = rag_pipeline.run({\n",
     "    \"text_embedder\": {\"text\": query},\n",
     "    \"prompt_builder\": {\"question\": query},\n",
@@ -512,7 +524,7 @@
      "name": "stderr",
      "output_type": "stream",
      "text": [
-      "/home/filip/Documents/.venvs/haystack/lib/python3.12/site-packages/transformers/tokenization_utils_base.py:1601: FutureWarning: `clean_up_tokenization_spaces` was not set. It will be set to `True` by default. This behavior will be depracted in transformers v4.45, and will be then set to `False` by default. For more details check this issue: https://github.com/huggingface/transformers/issues/31884\n",
+      "/Users/filipkozlowski/Documents/venvs/ragenv/lib/python3.12/site-packages/transformers/tokenization_utils_base.py:1601: FutureWarning: `clean_up_tokenization_spaces` was not set. It will be set to `True` by default. This behavior will be depracted in transformers v4.45, and will be then set to `False` by default. For more details check this issue: https://github.com/huggingface/transformers/issues/31884\n",
       "  warnings.warn(\n"
      ]
     },
@@ -524,7 +536,7 @@
       "application/vnd.jupyter.widget-view+json": {
        "version_major": 2,
        "version_minor": 0,
-       "model_id": "bd2775892c82492798ce720c3850b222"
+       "model_id": "ec05685f955e41b9aa1f84bb51e7fdaf"
       }
      },
      "metadata": {},
@@ -534,88 +546,121 @@
      "name": "stdout",
      "output_type": "stream",
      "text": [
-      "What if I have a team larger than 500 users?\n",
-      "# Comprehensive Documentation on Team Size and Server Management in Wire\n",
+      "Federations\n",
+      "# Wire Federation Documentation\n",
       "\n",
       "## Introduction\n",
       "\n",
-      "This documentation aims to provide a detailed overview of the infrastructure configuration options available for Wire, particularly for handling teams exceeding 500 users. It will explore the limits imposed on team sizes and conversation memberships, the necessary configurations for managing connections and resources, and best practices for server usage with extensive user bases. The information presented stems entirely from the extracted documents provided, ensuring clarity on how to optimize Wire for larger teams.\n",
+      "This documentation provides a comprehensive overview of the Wire Federation system, detailing its architecture, components, and the API interactions it facilitates between multiple backends. The documentation is structured in two distinct sections: one geared towards non-technical users, explaining the purpose and benefits of Wire Federation, and the other focused on technical users, providing in-depth information, code snippets, and practical guidance for implementation and troubleshooting.\n",
       "\n",
-      "### Objectives of This Documentation:\n",
-      "- Describe the maximum user limits within teams and conversations.\n",
-      "- Explain configurations needed for handling larger teams in Brigade (Brig).\n",
-      "- Provide guidance on file descriptor management for active connections.\n",
-      "- Offer troubleshooting tips and strategies for effective server management.\n",
+      "---\n",
       "\n",
-      "## Table of Contents\n",
-      "1. [Maximum Team Sizes](#maximum-team-sizes)\n",
-      "2. [Managing Large Teams](#managing-large-teams)\n",
-      "   - 2.1 [Configuring Team Size](#configuring-team-size)\n",
-      "   - 2.2 [Team Updates and WebSocket Notifications](#team-updates-and-websocket-notifications)\n",
-      "3. [Conversation Membership Limits](#conversation-membership-limits)\n",
-      "4. [File Descriptor Management](#file-descriptor-management)\n",
-      "5. [Recap and Conclusions](#recap-and-conclusions)\n",
+      "## Non-Technical User Section\n",
       "\n",
-      "## Maximum Team Sizes\n",
+      "### Overview of Wire Federation\n",
       "\n",
-      "By default, the maximum number of users in a team within Wire is capped at **500**. However, this limit can be modified. The configuration in the Brig settings allows you to increase this limit through the following setting:\n",
+      "**Wire Federation** allows separate Wire backend installations to communicate with each other seamlessly, enabling users on different backends to interact just as they would if they were on the same backend. This is particularly valuable for organizations that have multiple client installations or those that require an isolated on-premises deployment but still want to communicate with others.\n",
       "\n",
-      "```yaml\n",
-      "optSettings:\n",
-      "  setMaxTeamSize: 501\n",
-      "```\n",
+      "### Purpose and Benefits\n",
+      "\n",
+      "- **Enhanced Communication:** Wire Federation allows different teams and organizations to collaborate without the need for a single centralized system. Users from different installations can send messages, share files, and make calls just as if they were part of a single system.\n",
+      "\n",
+      "- **Flexibility in Deployment:** Organizations can choose to deploy either in the cloud or on an on-prem backend based on their security and compliance needs, while still being able to interact with users on other installations.\n",
+      "\n",
+      "- **Controlled Interaction:** Administrators have the option to specifically list which backends can communicate with their server, providing an added layer of security and control over who interacts with whom.\n",
+      "\n",
+      "### Practical Outcomes\n",
+      "\n",
+      "- **Interconnected Teams:** Your teams can work together across different installations, increasing productivity and collaboration.\n",
+      "  \n",
+      "- **Decentralized Infrastructure:** Organizations can maintain their independence while still allowing for necessary communication channels, which is critical for companies maintaining sensitive data.\n",
+      "\n",
+      "- **Configuration Options:** Administrators have multiple configurations at their disposal, allowing adaptability as business needs change.\n",
+      "\n",
+      "**Example:** If Team A is deployed on a cloud-based Wire server and Team B is using an on-prem infrastructure, Wire Federation enables both teams to send messages and collaborate without the need for both to be on the same system.\n",
+      "\n",
+      "---\n",
+      "\n",
+      "## Technical User Section\n",
       "\n",
-      "This adjustment permits the creation of teams with user counts higher than the standard limit. \n",
+      "### Technical Architecture of Wire Federation\n",
       "\n",
-      "### Important Notes:\n",
-      "- If your team surpasses **2000 members**, some real-time team update events, such as notifications regarding new members joining, will not be delivered via WebSocket connections. However, this limitation does not significantly obstruct the overall functioning of the application.\n",
-      "- Individual conversations, regardless of team size, maintain a strict upper limit of **2000** members. This cannot be overridden due to backend constraints that only support fan-out messaging to a maximum of 2000 recipients, which is an ongoing area for enhancement in future updates.\n",
+      "**Wire Federation** consists of various components, primarily: **Federation Ingress** and **Federator**. Each backend includes these components to manage communications, ensuring secure and authorized exchanges between different systems.\n",
       "\n",
-      "## Managing Large Teams\n",
+      "#### Key Components\n",
       "\n",
-      "### Configuring Team Size\n",
-      "To modify the maximum size of teams within your Wire infrastructure, follow these steps:\n",
-      "1. Access the Brig configuration file.\n",
-      "2. Locate the `optSettings` section.\n",
-      "3. Insert or edit the `setMaxTeamSize` option as shown previously.\n",
+      "- **Federation Ingress:** Acts as the entry point for incoming requests from other backends and ensures they are directed to the appropriate Federator component.\n",
       "\n",
-      "### Team Updates and WebSocket Notifications\n",
-      "When handling a team that exceeds **2000** members, you should be aware:\n",
-      "- Clients will not receive live websocket notifications for certain updates.\n",
-      "- Despite this, regular functionality should remain intact for most applications.\n",
+      "- **Federator:** Responsible for processing federated requests received from the Federation Ingress, performing necessary authorization checks, and facilitating outbound requests to other federated components.\n",
       "\n",
-      "## Conversation Membership Limits\n",
+      "#### Backend Domains\n",
       "\n",
-      "While teams can have flexible member limits beyond 500, it’s critical to note that conversation memberships are still restricted to **2000** participants. The backend architecture only permits sending a message to a maximum of 2000 recipients at a time, which is why this limit cannot be increased arbitrarily. Adjustments in this area are an ongoing endeavor.\n",
+      "Each Wire installation has two domains:\n",
+      "- **Infrastructure Domain:** The network-accessible domain for the backend (e.g., `wire.infra.example.com`).\n",
+      "- **Backend Domain:** The domain used for identifying users within that backend (e.g., `example.com`).\n",
       "\n",
-      "## File Descriptor Management\n",
+      "This differentiation enables users from different backends to maintain unique identifiers while facilitating federated communication.\n",
       "\n",
-      "For servers hosting connections with many users, managing file descriptors becomes vital. Wire’s Restund server (used for TURN connections) requires careful monitoring of allocations associated with active participant connections. Each allocation between a participant requires **1 or 2 file descriptors.**\n",
+      "#### Communication Process\n",
       "\n",
-      "### Recommendations:\n",
-      "- Ensure your server's file descriptor limits are adequately increased if you operate with a larger user base. \n",
-      "- Currently, a single Restund server supports up to **64,000** allocations. If you anticipate exceeding this number during active calls, deploying additional Restund servers is necessary to maintain performance.\n",
+      "Communication between backends involves mutual authentication and authorization, primarily facilitated through TLS. Key steps include:\n",
+      "1. **Authentication:** Verify the identity of the connecting backend.\n",
+      "2. **Discovery:** Determine the infrastructure domain associated with a backend domain.\n",
+      "3. **Authorization:** Confirm that both backends permit federated communication.\n",
       "\n",
-      "## Recap and Conclusions\n",
+      "### Code Snippet Example\n",
       "\n",
-      "In summary, managing larger teams in Wire encompasses adjusting maximum team size settings, addressing limitations in conversation membership, and effectively managing file descriptors for connections. For infrastructure managers, maintaining these configurations ensures that user experiences remain seamless and efficient, even with rising numbers in team memberships.\n",
+      "Here's an example of a configuration for setting up an SRV record necessary for discovery:\n",
       "\n",
-      "Key takeaways include:\n",
-      "- Default team limit is 500, extendable to 2000 with specific configurations.\n",
-      "- Conversations capped at 2000 members, with backend limitations on message fan-out.\n",
-      "- Monitoring and adjusting file descriptor allocations are essential for handling many active participants.\n",
+      "```bash\n",
+      "# Set up SRV record for backend discovery\n",
+      "_wire-server-federator._tcp.example.com. 600 IN SRV 0 10 443 federator.wire.example.org.\n",
+      "```\n",
+      "\n",
+      "**Explanation:** This SRV record allows other wire-server installations to discover your backend. The parameters include:\n",
+      "- **Service and Protocol:** Defined as `wire-server-federator` and `tcp`.\n",
+      "- **Port:** Typically set to `443` for secure communications.\n",
+      "\n",
+      "### Federation API Conventions\n",
+      "\n",
+      "All API calls related to federation should begin with the `/federation/` prefix. Each path should be structured in kebab-case format, facilitating clear and consistent interaction between federated components.\n",
+      "\n",
+      "Example of an API call to send a message across federated backends:\n",
+      "\n",
+      "```http\n",
+      "POST /federation/send-message\n",
+      "Content-Type: application/json\n",
+      "\n",
+      "{\n",
+      "  \"from\": \"user1@example.com\",\n",
+      "  \"to\": \"user2@otherbackend.com\",\n",
+      "  \"message\": \"Hello from federated backend!\"\n",
+      "}\n",
+      "```\n",
       "\n",
-      "This documentation serves as a foundational guide for those looking to optimize their use of Wire for larger collaborative environments.\n"
+      "### Conflict Resolution & Caveats\n",
+      "- Note that the **Federation development is currently a work in progress**. Features are being continually added and refined.\n",
+      "- **Important:** Changing the backend domain after user interaction can cause undefined behavior, potentially leading to data loss or client crashes. It is critical to lock down your domain configurations after they are set.\n",
+      "\n",
+      "**Example Conflicts:**\n",
+      "- Changing backend identifiers post-federation setup may lead to issues in user identification across systems.\n",
+      "\n",
+      "### Conclusion\n",
+      "Wire Federation represents a robust approach to cross-backend communications, facilitating better collaboration while prioritizing security and control. With active development, users should remain informed about updates to features and best practices of deployment.\n",
+      "\n",
+      "---\n",
+      "\n",
+      "This documentation serves as a reference for both non-technical stakeholders seeking to understand the benefits of Wire Federation and technical users implementing or managing the federation system. For further questions or clarifications, please refer to the appropriate sections or reach out with specific queries.\n"
      ]
     }
    ],
-   "execution_count": 2
+   "execution_count": 10
   },
   {
    "metadata": {
     "ExecuteTime": {
-     "end_time": "2024-10-01T16:22:14.084675Z",
-     "start_time": "2024-10-01T16:22:14.081576Z"
+     "end_time": "2024-10-02T14:39:37.687390Z",
+     "start_time": "2024-10-02T14:39:37.681544Z"
     }
    },
    "cell_type": "code",
@@ -633,113 +678,113 @@
      "output_type": "stream",
      "text": [
       "Finish Reason: stop\n",
-      "Output Tokens: 899\n",
-      "Input Tokens: 15356\n",
-      "\n",
-      "1. 0.578950703 Infrastructure configuration options — Wire 0.0.4 documentation I have a team larger than 500 users\n",
-      "    https://docs.wire.com/how-to/install/infrastructure-configuration.html\n",
-      "2. 0.578950703 Infrastructure configuration options — Wire 0.0.4 documentation I have a team larger than 500 users\n",
-      "    https://docs.wire.com/how-to/install/infrastructure-configuration.html#load-balancer-on-bare-metal-servers\n",
-      "3. 0.578950703 Infrastructure configuration options — Wire 0.0.4 documentation I have a team larger than 500 users\n",
-      "    https://docs.wire.com/how-to/install/infrastructure-configuration.html#persistence-and-high-availability\n",
-      "4. 0.578950703 Infrastructure configuration options — Wire 0.0.4 documentation I have a team larger than 500 users\n",
-      "    https://docs.wire.com/how-to/install/infrastructure-configuration.html#security\n",
-      "5. 0.578950703 Infrastructure configuration options — Wire 0.0.4 documentation I have a team larger than 500 users\n",
-      "    https://docs.wire.com/how-to/install/infrastructure-configuration.html#separate-incoming-websocket-network-traffic-from-the-rest-of-the-https-traffic\n",
-      "6. 0.578950703 Infrastructure configuration options — Wire 0.0.4 documentation I have a team larger than 500 users\n",
-      "    https://docs.wire.com/how-to/install/infrastructure-configuration.html#rd-party-proxying\n",
-      "7. 0.578950703 Infrastructure configuration options — Wire 0.0.4 documentation I have a team larger than 500 users\n",
-      "    https://docs.wire.com/how-to/install/infrastructure-configuration.html#redirect-some-traffic-through-a-http-s-proxy\n",
-      "8. 0.578950703 Infrastructure configuration options — Wire 0.0.4 documentation I have a team larger than 500 users\n",
-      "    https://docs.wire.com/how-to/install/infrastructure-configuration.html#marking-an-installation-as-self-hosted\n",
-      "9. 0.578950703 Infrastructure configuration options — Wire 0.0.4 documentation I have a team larger than 500 users\n",
-      "    https://docs.wire.com/how-to/install/infrastructure-configuration.html#real-aws-services\n",
-      "10. 0.578950703 Infrastructure configuration options — Wire 0.0.4 documentation I have a team larger than 500 users\n",
-      "    https://docs.wire.com/how-to/install/infrastructure-configuration.html#enable-push-notifications-using-the-public-appstore-playstore-mobile-wire-clients\n",
-      "11. 0.578950703 Infrastructure configuration options — Wire 0.0.4 documentation I have a team larger than 500 users\n",
-      "    https://docs.wire.com/how-to/install/infrastructure-configuration.html#metrics-logging\n",
-      "12. 0.578950703 Infrastructure configuration options — Wire 0.0.4 documentation I have a team larger than 500 users\n",
-      "    https://docs.wire.com/how-to/install/infrastructure-configuration.html#load-balancer-on-cloud-provider\n",
-      "13. 0.578950703 Infrastructure configuration options — Wire 0.0.4 documentation I have a team larger than 500 users\n",
-      "    https://docs.wire.com/how-to/install/infrastructure-configuration.html#you-may-want\n",
-      "14. 0.578950703 Infrastructure configuration options — Wire 0.0.4 documentation I have a team larger than 500 users\n",
-      "    https://docs.wire.com/how-to/install/infrastructure-configuration.html#routing-traffic-to-other-namespaces-via-nginz\n",
-      "15. 0.578950703 Infrastructure configuration options — Wire 0.0.4 documentation I have a team larger than 500 users\n",
-      "    https://docs.wire.com/how-to/install/infrastructure-configuration.html#control-nginz-upstreams-routes-into-the-kubernetes-cluster\n",
-      "16. 0.578950703 Infrastructure configuration options — Wire 0.0.4 documentation I have a team larger than 500 users\n",
-      "    https://docs.wire.com/how-to/install/infrastructure-configuration.html#s3-addressing-style\n",
-      "17. 0.578950703 Infrastructure configuration options — Wire 0.0.4 documentation I have a team larger than 500 users\n",
-      "    https://docs.wire.com/how-to/install/infrastructure-configuration.html#controlling-the-speed-of-websocket-draining-during-cannon-pod-replacement\n",
-      "18. 0.578950703 Infrastructure configuration options — Wire 0.0.4 documentation I have a team larger than 500 users\n",
-      "    https://docs.wire.com/how-to/install/infrastructure-configuration.html#smtp-server\n",
-      "19. 0.578950703 Infrastructure configuration options — Wire 0.0.4 documentation I have a team larger than 500 users\n",
-      "    https://docs.wire.com/how-to/install/infrastructure-configuration.html#i-have-a-team-larger-than-500-users\n",
-      "20. 0.578950703 Infrastructure configuration options — Wire 0.0.4 documentation I have a team larger than 500 users\n",
-      "    https://docs.wire.com/how-to/install/infrastructure-configuration.html#configuring-authentication-cookie-throttling\n",
-      "21. 0.392603099 Restund (TURN) servers — Wire 0.0.4 documentation Amount of users and file descriptors\n",
-      "    https://docs.wire.com/understand/restund.html#amount-of-users-and-file-descriptors\n",
-      "22. 0.392603099 Restund (TURN) servers — Wire 0.0.4 documentation Amount of users and file descriptors\n",
-      "    https://docs.wire.com/understand/restund.html\n",
-      "23. 0.392603099 Restund (TURN) servers — Wire 0.0.4 documentation Amount of users and file descriptors\n",
-      "    https://docs.wire.com/understand/restund.html#dns\n",
-      "24. 0.392603099 Restund (TURN) servers — Wire 0.0.4 documentation Amount of users and file descriptors\n",
-      "    https://docs.wire.com/understand/restund.html#what-is-it-used-for\n",
-      "25. 0.392603099 Restund (TURN) servers — Wire 0.0.4 documentation Amount of users and file descriptors\n",
-      "    https://docs.wire.com/understand/restund.html#discovery-and-establishing-a-call\n",
-      "26. 0.369646847 github-action-wire-messenger/node_modules/readable-stream/GOVERNANCE.md \n",
-      "    https://github.com/wireapp/github-action-wire-messenger/blob/main/node_modules/readable-stream/GOVERNANCE.md\n",
-      "27. 0.361026824 openmls/book/src/performance.md \n",
-      "    https://github.com/wireapp/openmls/blob/wire/stable/book/src/performance.md\n",
-      "28. 0.344685107 wire-server/CHANGELOG.md \n",
+      "Output Tokens: 1106\n",
+      "Input Tokens: 14202\n",
+      "\n",
+      "1. 0.614980042 Federation — Wire 0.0.4 documentation Federation\n",
+      "    https://docs.wire.com/understand/configure-federation.html#generate-and-configure-tls-server-and-client-certificates\n",
+      "2. 0.614980042 Federation — Wire 0.0.4 documentation Federation\n",
+      "    https://docs.wire.com/understand/configure-federation.html#configure-helm-charts-federator-and-ingress-and-webapp-subcharts\n",
+      "3. 0.614980042 Federation — Wire 0.0.4 documentation Federation\n",
+      "    https://docs.wire.com/understand/configure-federation.html\n",
+      "4. 0.614980042 Federation — Wire 0.0.4 documentation Federation\n",
+      "    https://docs.wire.com/understand/configure-federation.html#choose-a-backend-domain\n",
+      "5. 0.614980042 Federation — Wire 0.0.4 documentation Federation\n",
+      "    https://docs.wire.com/understand/configure-federation.html#summary-of-necessary-steps-to-configure-federation\n",
+      "6. 0.614980042 Federation — Wire 0.0.4 documentation Federation\n",
+      "    https://docs.wire.com/understand/configure-federation.html#manually-test-that-your-configurations-work-as-expected\n",
+      "7. 0.614980042 Federation — Wire 0.0.4 documentation Federation\n",
+      "    https://docs.wire.com/understand/configure-federation.html#dns-setup-for-federation\n",
+      "8. 0.614980042 Federation — Wire 0.0.4 documentation Federation\n",
+      "    https://docs.wire.com/understand/configure-federation.html#consequences-of-the-choice-of-a-backend-domain\n",
+      "9. 0.614980042 Federation — Wire 0.0.4 documentation Federation\n",
+      "    https://docs.wire.com/understand/configure-federation.html#applying-all-configuration-changes\n",
+      "10. 0.55589819 Federation API — Wire 0.0.4 documentation Example End-to-End Flows\n",
+      "    https://docs.wire.com/understand/federation/api.html\n",
+      "11. 0.554861844 Config Options — Wire 0.0.4 documentation Federation allow list\n",
+      "    https://docs.wire.com/developer/reference/config-options.html\n",
+      "12. 0.554861844 Config Options — Wire 0.0.4 documentation Restrict user search\n",
+      "    https://docs.wire.com/developer/reference/config-options.html\n",
+      "13. 0.538971603 wire-server/docs/src/understand/federation/architecture.md \n",
+      "    https://github.com/wireapp/wire-server/blob/develop/docs/src/understand/federation/architecture.md\n",
+      "14. 0.512375832 Wire Federation — Wire 0.0.4 documentation Wire Federation\n",
+      "    https://docs.wire.com/understand/federation/index.html\n",
+      "15. 0.506851077 wire-server/docs/src/how-to/install/sft.md \n",
+      "    https://github.com/wireapp/wire-server/blob/develop/docs/src/how-to/install/sft.md\n",
+      "16. 0.506233096 wire-ios/WireDomain/Project/WireDomain.docc/federation.md \n",
+      "    https://github.com/wireapp/wire-ios/blob/develop/WireDomain/Project/WireDomain.docc/federation.md\n",
+      "17. 0.496853769 wire-server/CHANGELOG.md \n",
       "    https://github.com/wireapp/wire-server/blob/develop/CHANGELOG.md\n",
-      "29. 0.339702278 wire-server/docs/src/understand/searchability.md \n",
-      "    https://github.com/wireapp/wire-server/blob/develop/docs/src/understand/searchability.md\n",
-      "30. 0.329657584 wire-server/docs/src/how-to/administrate/users.md \n",
-      "    https://github.com/wireapp/wire-server/blob/develop/docs/src/how-to/administrate/users.md\n",
-      "31. 0.32627359 wire-server/CHANGELOG.md \n",
+      "18. 0.491552591 Federation Achitecture — Wire 0.0.4 documentation Backends\n",
+      "    https://docs.wire.com/understand/federation/architecture.html\n",
+      "19. 0.490198672 Federation Achitecture — Wire 0.0.4 documentation Service components\n",
+      "    https://docs.wire.com/understand/federation/architecture.html\n",
+      "20. 0.490190685 wire-server/docs/src/understand/federation/index.md \n",
+      "    https://github.com/wireapp/wire-server/blob/develop/docs/src/understand/federation/index.md\n",
+      "21. 0.490172774 Backend to backend communication — Wire 0.0.4 documentation Backend to backend communication\n",
+      "    https://docs.wire.com/understand/federation/backend-communication.html\n",
+      "22. 0.48189187 Federation Achitecture — Wire 0.0.4 documentation Backend domains\n",
+      "    https://docs.wire.com/understand/federation/architecture.html\n",
+      "23. 0.477519423 Swagger API documentation — Wire 0.0.4 documentation Federation API\n",
+      "    https://docs.wire.com/understand/api-client-perspective/swagger.html\n",
+      "24. 0.464122623 wire-server/docs/src/developer/developer/federation-api-conventions.md \n",
+      "    https://github.com/wireapp/wire-server/blob/develop/docs/src/developer/developer/federation-api-conventions.md\n",
+      "25. 0.463976592 Federation — Wire 0.0.4 documentation SRV record\n",
+      "    https://docs.wire.com/understand/configure-federation.html#summary-of-necessary-steps-to-configure-federation\n",
+      "26. 0.463976592 Federation — Wire 0.0.4 documentation SRV record\n",
+      "    https://docs.wire.com/understand/configure-federation.html#consequences-of-the-choice-of-a-backend-domain\n",
+      "27. 0.463976592 Federation — Wire 0.0.4 documentation SRV record\n",
+      "    https://docs.wire.com/understand/configure-federation.html#dns-setup-for-federation\n",
+      "28. 0.463976592 Federation — Wire 0.0.4 documentation SRV record\n",
+      "    https://docs.wire.com/understand/configure-federation.html#choose-a-backend-domain\n",
+      "29. 0.463976592 Federation — Wire 0.0.4 documentation SRV record\n",
+      "    https://docs.wire.com/understand/configure-federation.html#configure-helm-charts-federator-and-ingress-and-webapp-subcharts\n",
+      "30. 0.463976592 Federation — Wire 0.0.4 documentation SRV record\n",
+      "    https://docs.wire.com/understand/configure-federation.html#manually-test-that-your-configurations-work-as-expected\n",
+      "31. 0.463976592 Federation — Wire 0.0.4 documentation SRV record\n",
+      "    https://docs.wire.com/understand/configure-federation.html#applying-all-configuration-changes\n",
+      "32. 0.462774605 wire-server/docs/src/understand/federation/architecture.md \n",
+      "    https://github.com/wireapp/wire-server/blob/develop/docs/src/understand/federation/architecture.md\n",
+      "33. 0.457686335 wire-server/docs/src/developer/developer/federation-api-conventions.md \n",
+      "    https://github.com/wireapp/wire-server/blob/develop/docs/src/developer/developer/federation-api-conventions.md\n",
+      "34. 0.457345426 wire-server/docs/src/understand/federation/backend-communication.md \n",
+      "    https://github.com/wireapp/wire-server/blob/develop/docs/src/understand/federation/backend-communication.md\n",
+      "35. 0.456024379 wire-server/CHANGELOG.md \n",
       "    https://github.com/wireapp/wire-server/blob/develop/CHANGELOG.md\n",
-      "32. 0.325858176 wire-server/docs/src/understand/single-sign-on/trouble-shooting.md \n",
-      "    https://github.com/wireapp/wire-server/blob/develop/docs/src/understand/single-sign-on/trouble-shooting.md\n",
-      "33. 0.321500659 wire-server/docs/src/understand/block-user-creation.md \n",
-      "    https://github.com/wireapp/wire-server/blob/develop/docs/src/understand/block-user-creation.md\n",
-      "34. 0.320721209 dependency-track/docs/_docs/getting-started/openidconnect-configuration.md \n",
-      "    https://github.com/wireapp/dependency-track/blob/master/docs/_docs/getting-started/openidconnect-configuration.md\n",
-      "35. 0.320463151 wire-server/docs/src/understand/restund.md \n",
-      "    https://github.com/wireapp/wire-server/blob/develop/docs/src/understand/restund.md\n",
-      "36. 0.318782 wire-server/docs/src/understand/searchability.md \n",
-      "    https://github.com/wireapp/wire-server/blob/develop/docs/src/understand/searchability.md\n",
-      "37. 0.306419581 wire-server/docs/src/understand/searchability.md \n",
-      "    https://github.com/wireapp/wire-server/blob/develop/docs/src/understand/searchability.md\n",
-      "38. 0.304954112 wire-server/docs/src/how-to/install/prod-intro.md \n",
-      "    https://github.com/wireapp/wire-server/blob/develop/docs/src/how-to/install/prod-intro.md\n",
-      "39. 0.303582937 wire-server/CHANGELOG.md \n",
+      "36. 0.454764307 wire-server/CHANGELOG.md \n",
       "    https://github.com/wireapp/wire-server/blob/develop/CHANGELOG.md\n",
-      "40. 0.291234136 wire-server/docs/src/developer/reference/config-options.md \n",
-      "    https://github.com/wireapp/wire-server/blob/develop/docs/src/developer/reference/config-options.md\n",
-      "41. 0.288038671 wire-server/docs/src/understand/single-sign-on/trouble-shooting.md \n",
-      "    https://github.com/wireapp/wire-server/blob/develop/docs/src/understand/single-sign-on/trouble-shooting.md\n",
-      "42. 0.286405206 wire-server/CHANGELOG.md \n",
+      "37. 0.44467172 wire-server/services/brig/docs/swagger.md \n",
+      "    https://github.com/wireapp/wire-server/blob/develop/services/brig/docs/swagger.md\n",
+      "38. 0.444358 wire-server/docs/src/understand/configure-federation.md \n",
+      "    https://github.com/wireapp/wire-server/blob/develop/docs/src/understand/configure-federation.md\n",
+      "39. 0.442574441 wire-server/services/brig/docs/swagger.md \n",
+      "    https://github.com/wireapp/wire-server/blob/develop/services/brig/docs/swagger.md\n",
+      "40. 0.434415787 wire-server/docs/src/understand/federation/api.md \n",
+      "    https://github.com/wireapp/wire-server/blob/develop/docs/src/understand/federation/api.md\n",
+      "41. 0.426637262 wire-server/docs/src/understand/configure-federation.md \n",
+      "    https://github.com/wireapp/wire-server/blob/develop/docs/src/understand/configure-federation.md\n",
+      "42. 0.416953623 wire-server/docs/src/understand/federation/api.md \n",
+      "    https://github.com/wireapp/wire-server/blob/develop/docs/src/understand/federation/api.md\n",
+      "43. 0.416618854 wire-server/docs/src/understand/configure-federation.md \n",
+      "    https://github.com/wireapp/wire-server/blob/develop/docs/src/understand/configure-federation.md\n",
+      "44. 0.409066141 wire-server/CHANGELOG.md \n",
       "    https://github.com/wireapp/wire-server/blob/develop/CHANGELOG.md\n",
-      "43. 0.281497568 core-crypto/CHANGELOG.md \n",
-      "    https://github.com/wireapp/core-crypto/blob/main/CHANGELOG.md\n",
-      "44. 0.28122 wire-server/docs/src/how-to/install/infrastructure-configuration.md \n",
-      "    https://github.com/wireapp/wire-server/blob/develop/docs/src/how-to/install/infrastructure-configuration.md\n",
-      "45. 0.279998243 wire-server/docs/src/how-to/administrate/users.md \n",
-      "    https://github.com/wireapp/wire-server/blob/develop/docs/src/how-to/administrate/users.md\n",
-      "46. 0.27901867 uniffi-rs/docs/adr/0001-mvp-webidl.md \n",
-      "    https://github.com/wireapp/uniffi-rs/blob/main/docs/adr/0001-mvp-webidl.md\n",
-      "47. 0.277904153 wire-server/docs/src/developer/reference/config-options.md \n",
+      "45. 0.407607 wire-server/services/brig/docs/swagger.md \n",
+      "    https://github.com/wireapp/wire-server/blob/develop/services/brig/docs/swagger.md\n",
+      "46. 0.4062154 wire-server/docs/src/understand/federation/backend-communication.md \n",
+      "    https://github.com/wireapp/wire-server/blob/develop/docs/src/understand/federation/backend-communication.md\n",
+      "47. 0.395716488 wire-server/docs/src/understand/federation/architecture.md \n",
+      "    https://github.com/wireapp/wire-server/blob/develop/docs/src/understand/federation/architecture.md\n",
+      "48. 0.394247204 wire-server/docs/src/developer/reference/config-options.md \n",
       "    https://github.com/wireapp/wire-server/blob/develop/docs/src/developer/reference/config-options.md\n",
-      "48. 0.275354743 github-action-wire-messenger/node_modules/http-status-codes/README.md \n",
-      "    https://github.com/wireapp/github-action-wire-messenger/blob/main/node_modules/http-status-codes/README.md\n",
-      "49. 0.274806172 wire-server/docs/src/understand/single-sign-on/trouble-shooting.md \n",
-      "    https://github.com/wireapp/wire-server/blob/develop/docs/src/understand/single-sign-on/trouble-shooting.md\n",
-      "50. 0.269872129 wire-server/CHANGELOG.md \n",
-      "    https://github.com/wireapp/wire-server/blob/develop/CHANGELOG.md\n"
+      "49. 0.389835417 wire-server/CHANGELOG.md \n",
+      "    https://github.com/wireapp/wire-server/blob/develop/CHANGELOG.md\n",
+      "50. 0.386147588 wire-server/docs/src/understand/sft.md \n",
+      "    https://github.com/wireapp/wire-server/blob/develop/docs/src/understand/sft.md\n"
      ]
     }
    ],
-   "execution_count": 3
+   "execution_count": 11
   },
   {
    "metadata": {},

pipeline.yml

@@ -16,52 +16,14 @@ components:
       model: gpt-4o-mini
       organization: null
       streaming_callback: null
-      system_prompt: "Task: Generate comprehensive documentation based on the provided\
-        \ documents. The documentation should capture all essential details and information\
-        \ without any information loss, including relevant code fragments from the\
-        \ files. Ensure that the output does not cut corners on tokens; generate as\
-        \ much content as possible within the limits of the provided documents. Do\
-        \ not reference or include information from any sources other than the given\
-        \ documents, do not rely on your knowledge.\n\nRequirements:\n\n1. Length:\
-        \ The documentation should be detailed and extensive, covering all aspects\
-        \ of the content provided in the documents.\n\n2. Structure:\n   - Title Page:\
-        \ Include a title that reflects the main theme of the documents.\n   - Introduction:\
-        \ Provide a complete and thorough overview of the content, purpose, and scope\
-        \ of the documentation.\n   - Main Sections: Divide the documentation into\
-        \ logical sections and subsections based on the themes and topics found in\
-        \ the documents. Each section should include:\n     - A clear, descriptive\
-        \ heading that reflects the content within the section.\n     - Detailed explanations\
-        \ of key concepts and ideas. Expand on each idea or concept with thorough\
-        \ descriptions, definitions, and context. Ensure the reader fully understands\
-        \ the material.\n     - Code fragments: Where applicable, include relevant\
-        \ code snippets from the documents. These should be properly formatted and\
-        \ explained in detail. For each code snippet, provide a clear explanation\
-        \ of its purpose, functionality, and how it fits into the overall system or\
-        \ process described in the document.\n     - Examples: Provide relevant examples\
-        \ where applicable, explaining them fully to ensure understanding.\n     -\
-        \ Conflict Resolution: If there are contradictory pieces of information across\
-        \ different documents, resolve them using the date provided at the beginning\
-        \ of each document, prioritizing the most recent information. If a document\u2019\
-        s date is unknown and conflicts cannot be resolved, explicitly indicate which\
-        \ parts contradict one another, then clearly explain the contradiction. Mention\
-        \ all unresolved conflicts at the end of each section.\n\n3. Content Guidelines:\n\
-        \   - Ensure that all information from the documents is included and accurately\
-        \ represented.\n   - The content should not just list bullet points but should\
-        \ be expanded into full sentences and paragraphs where necessary, with elaboration\
-        \ and explanations for all points.\n   - Use clear and logical transitions\
-        \ between sections to maintain the flow and coherence of the document.\n \
-        \  - Use bullet points, tables, code fragments, or diagrams where appropriate\
-        \ to enhance understanding and readability, but always accompany these elements\
-        \ with in-depth explanations.\n\n4. Style: The writing style should be professional,\
-        \ formal, and suitable for an audience that may include technical experts,\
-        \ stakeholders, or general readers interested in the topic. Avoid jargon unless\
-        \ it is explained in detail.\n\n5. Specifics: Highlight any critical findings,\
-        \ data, or statistics that are present in the documents. For each significant\
-        \ point, explain its relevance and implications. Emphasize any unique or innovative\
-        \ aspects that stand out, and provide complete reasoning or analysis where\
-        \ needed.\n\nOutput Format: Ensure that the entire documentation is generated\
-        \ in Markdown format without any additional text or formatting. The output\
-        \ should be ready to be saved directly into a markdown file.\n"
+      system_prompt: 'You are an expert assistant skilled in generating structured,
+        comprehensive documentation. Your role is to create thorough, accurate documentation
+        based strictly on the provided context, without using external knowledge or
+        general information. The content must be divided into two main sections: one
+        for non-technical users and one for technical users, with each section tailored
+        to their needs. The output should be clear, detailed, and organized, including
+        code snippets, examples, and conflict resolution where applicable. Your final
+        output should be ready in Markdown format.'
     type: haystack.components.generators.openai.OpenAIGenerator
   joiner:
     init_parameters:
@@ -73,9 +35,45 @@ components:
   prompt_builder:
     init_parameters:
       required_variables: null
-      template: "\nUser Question: {{question}}\nDocuments to Analyze:\n{% for doc\
-        \ in documents %}\nDate: {{doc.meta['date']}}\nTitle: {{doc.meta['title']}}\
-        \ - {{doc.meta['headline']}}\nContent: \n{{doc.content}}\n{% endfor %}"
+      template: "**Task:**  \nGenerate comprehensive documentation based solely on\
+        \ the provided documents. **Do not use any external knowledge** or information\
+        \ outside of the provided context. Divide the documentation into two main\
+        \ sections:\n\n- **Non-Technical User:** Focus on explanations that are accessible\
+        \ to non-technical stakeholders (e.g., **business decision-makers or managers**).\
+        \ Highlight the purpose, value, and practical outcomes, avoiding technical\
+        \ jargon. Use examples and high-level explanations to ensure understanding.\n\
+        \n- **Technical User:** Provide in-depth technical content for an audience\
+        \ of **developers, engineers, or system architects**. Include detailed explanations\
+        \ of key technical concepts, code snippets with thorough descriptions, and\
+        \ examples where applicable. Explain the functionality and purpose of each\
+        \ code fragment, along with any potential issues or caveats.\n\n**Requirements:**\n\
+        \n1. **Length:** The documentation should be detailed and cover all aspects\
+        \ of the provided content.\n\n2. **Structure:**  \n   - **Title Page:** Include\
+        \ a title reflecting the main theme of the documents.  \n   - **Introduction:**\
+        \ Provide a clear overview of the content, purpose, and scope of the documentation.\
+        \  \n   - **Non-Technical User Section:**  \n     - Clear, high-level explanations\
+        \ that avoid technical depth.  \n     - Emphasize the **purpose, benefits,\
+        \ and outcomes** for non-technical users.  \n     - Include examples and **highlight\
+        \ key points** with bold text where necessary.  \n   - **Technical User Section:**\
+        \  \n     - Detailed explanations of key technical concepts.  \n     - Include\
+        \ well-explained code snippets, with a focus on their **practical application**\
+        \ and any **potential pitfalls**.  \n     - **Conflict Resolution:** If conflicting\
+        \ information is present, resolve it using the document date or highlight\
+        \ unresolved contradictions **inline** within the relevant sections.\n\n3.\
+        \ **Content Guidelines:**  \n   - Represent all information accurately from\
+        \ the documents.  \n   - Expand bullet points into full sentences and paragraphs.\
+        \  \n   - Use bullet points, tables, or code fragments where necessary, with\
+        \ in-depth explanations.  \n   - Ensure smooth transitions between sections.\n\
+        \n4. **Style:**  \n   - The non-technical section should be accessible and\
+        \ easy to understand.  \n   - The technical section should be professional\
+        \ and formal, with precise technical language.\n\n5. **Specifics:**  \n  \
+        \ - Highlight any critical findings, data, or statistics from the documents.\
+        \  \n   - Emphasize unique aspects with full reasoning and analysis.  \n \
+        \  - If any sections lack sufficient detail, indicate **gaps or missing information**\
+        \ rather than assuming or fabricating content.\n\n\nUser Question: {{question}}\n\
+        Documents to Analyze:\n{% for doc in documents %}\nDate: {{doc.meta['date']}}\n\
+        Title: {{doc.meta['title']}} - {{doc.meta['headline']}}\nContent: \n{{doc.content}}\n\
+        {% endfor %}"
       variables: null
     type: haystack.components.builders.prompt_builder.PromptBuilder
   retriever_docs_wire: