Upload AWS_and_Pinecone_Accounts_for_LiveRAG.md

Operational_Instructions/AWS_and_Pinecone_Accounts_for_LiveRAG.md

@@ -0,0 +1,86 @@
+# AWS and Pinecone Accounts for LiveRAG
+
+## Using AWS Accounts
+
+During the LiveRAG Challenge, participants may use two AWS accounts:
+- **Personal AWS Account** – This is your private AWS account, where you can apply AWS credits provided for the competition.
+- **TII-Provided AWS Account** – This account is managed by TII and grants access to the dense (Pinecone) and sparse (OpenSearch) pre-built indices.
+
+Using these accounts is optional but highly recommended to optimize both cost and effort.
+
+## Using Your Personal AWS Account with AWS Credits
+
+Each participating group receives AWS credits for the duration of the competition.
+The credits are limited, and any usage beyond the allocated amount will be charged to your personal payment method.
+To ensure your credits are properly applied, follow the official AWS instructions [here](https://docs.aws.amazon.com/awsaccountbilling/latest/aboutv2/useconsolidatedbilling-credits.html).
+
+### Personal AWS Account Setup and Credit Application Steps
+
+1. **Create an AWS Account** – Sign up for an AWS account and add a payment method (typically a credit card). If you have an existing AWS account you may reuse it.
+2. **Apply AWS Credits** – Follow the instructions in the provided AWS documentation to redeem and apply your credits.
+3. **Monitor Costs Proactively** – AWS usage, especially GPU-based resources, can be expensive. We highly recommend to closely monitor your costs.
+   - Use AWS Cost Management tools to monitor spending.
+   - Set up billing alarms using AWS CloudWatch to receive notifications when costs exceed predefined thresholds.
+
+### Cost Optimization Recommendations
+
+TII and AWS have estimated the cost of a typical team’s infrastructure, including GPU usage, and AWS has provided credits accordingly.
+However, individual teams may have varying requirements, so cost management is essential:
+- **Shut down unused resources** – Always shut down GPUs and other compute instances when not in use.
+- **Experiment on smaller datasets first** – This approach speeds up iteration cycles and reduces expenses before scaling up to larger datasets.
+
+## Using the TII-Provided AWS Account
+
+The TII-provided AWS account is strictly limited to specific use cases, primarily for accessing pre-built indices. These include:
+
+- A **Sparse OpenSearch Index** hosted by AWS.
+- A **Dense Pinecone Index** hosted by Pinecone, with access credentials stored on AWS.
+
+
+### Access and Setup Instructions
+
+To gain access and configure the TII-Provided AWS account, follow these steps:
+
+1. **Receive Sign-In Details** – You will receive an email with login credentials from the organizers.
+2. **Sign In & Change Password** – Log in to the AWS console and update your password.
+3. **Create AWS Command Line Interface (CLI) Credentials** – Generate an access key and secret for CLI usage.
+4. **Configure AWS CLI Profile** – Run the following command to set up the CLI for this account and follow the instructions:
+
+   ```sh
+   aws configure --profile sigir-participant
+   ```
+
+5. **Verify Access** – Query the indices to confirm successful configuration.
+
+For detailed instructions on account setup, credentials, and index usage, refer to this [example notebook](Operational_Instructions/Indices_Usage_Examples_for_LiveRAG.ipynb)
+
+- **Use AWS CLI Profiles** – We recommend usage of separate AWS profiles. Configure separate AWS CLI profiles for better account management:
+
+  ```sh
+  aws configure --profile personal
+  aws configure --profile sigir-participant
+  ```
+  - `personal`: Your own AWS account.
+  - `sigir-participant`: The TII-provided AWS account (described below).
+
+For beginners, it is recommended to complete introductory AWS tutorials available [here](https://aws.amazon.com/getting-started/).
+
+
+## Using the Pre-Built Indices and Building Your Own Indices 
+We provide you with two basic pre-built dense (Pinecone) and sparse (Opensearch) indices. See usage [here](Indices_Usage_Examples_for_LiveRAG.ipynb). You may use them freely for LiveRAG Challenge tasks only. 
+
+You're welcome to build your own Pinecone/OpenSearch, or other indices. We encourage you to take advantage of the AWS and Pinecone credits we provide for this purpose. OpenSearch can be provisioned and paid by AWS credits.
+
+For instructions pertaining to building your own Pinecone index see [here](Pinecone_for_LiveRAG.md).
+
+## Summary of Key Recommendations
+
+| **Action**            | **Recommendation**                                           |
+|:-----------------------|:-------------------------------------------------------------|
+| **AWS Credits**       | Apply credits early and monitor usage.                      |
+| **Billing Alerts**    | Use AWS CloudWatch to avoid unexpected charges.             |
+| **Resource Management** | Shut down unused resources to reduce costs.               |
+| **Experimentation**   | Start with small datasets before scaling up.                |
+| **Account Management** | Use separate AWS CLI profiles for personal and TII-Provided accounts. |
+
+By following these best practices, you can effectively manage your resources, optimize costs, and maximize your efficiency during the LiveRAG Challenge.