{
"cells": [
{
"cell_type": "markdown",
"metadata": {
"jp-MarkdownHeadingCollapsed": true
},
"source": [
"## SN20 BitAgent\n",
"\n",
"### Setting up ...\n",
"- importing standard libraries + bittensor, no special sauce required\n",
"- fetching subnet 20\n",
"- setting up wallet and validator\n",
"- getting top miner\n",
"- providing protocol (QnATask)"
]
},
{
"cell_type": "code",
"execution_count": 1,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"\u001b[34m2024-04-07 17:47:36.518\u001b[0m | \u001b[1m INFO \u001b[0m | You are connecting to finney network with endpoint wss://entrypoint-finney.opentensor.ai:443.\n",
"\u001b[34m2024-04-07 17:47:36.519\u001b[0m | \u001b[33m\u001b[1m WARNING \u001b[0m | We strongly encourage running a local subtensor node whenever possible. This increases decentralization and resilience of the network.\n",
"\u001b[34m2024-04-07 17:47:36.519\u001b[0m | \u001b[33m\u001b[1m WARNING \u001b[0m | In a future release, local subtensor will become the default endpoint. To get ahead of this change, please run a local subtensor node and point to it.\n",
"\u001b[34m2024-04-07 17:47:36.833\u001b[0m | \u001b[1m INFO \u001b[0m | Connected to finney network and wss://entrypoint-finney.opentensor.ai:443.\n",
"Top Miner UID for Subnet 20: 197\n"
]
}
],
"source": [
"import asyncio\n",
"import requests\n",
"import bittensor as bt \n",
"from rich import print as rprint\n",
"from typing import Optional,List\n",
"\n",
"# working with subnet 20 / upsilon / BitAgent\n",
"subnet = bt.metagraph(netuid=20)\n",
"\n",
"# Wallet and validator setup\n",
"WALLET_NAME = \"TODO\" # TODO\n",
"HOTKEY_NAME = \"TODO\" # TODO\n",
"vali_wallet = bt.wallet(name=WALLET_NAME, hotkey=HOTKEY_NAME)\n",
"vali_dendrite = bt.dendrite(wallet=vali_wallet)\n",
"\n",
"# get the TOP miner on the subnet\n",
"top_miner_uid = int(subnet.I.argmax())\n",
"print(\"Top Miner UID for Subnet 20: \", top_miner_uid)\n",
"\n",
"# the request protocol\n",
"class QnATask(bt.Synapse):\n",
" urls: List[str] = [] # not used right now\n",
" datas: List[dict] = [] # used to pass in relevant context, could be a company knowledge base or a set of wikipedia pages\n",
" prompt: str = \"\" # the query / prompt\n",
" response: Optional[dict] = {}\n",
" timeout: Optional[float] = 3.0\n",
" miner_uids: Optional[List[int]] = [top_miner_uid] # put our TOP miner into the network as the miner to query (if empty list, a random list of miners will be selected)\n",
"\n",
" def toJSON(self):\n",
" return {\"prompt\": self.prompt, \n",
" \"urls\": self.urls, \n",
" \"datas\": self.datas, \n",
" \"response\": self.response,\n",
" \"miner_uids\": self.miner_uids,\n",
" \"dendrite_process_time\": self.dendrite.process_time,\n",
" \"dendrite_status_code\": self.dendrite.status_code,\n",
" \"axon_status_code\": self.axon.status_code,}"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Two ways to query SN20 \n",
"\n",
"### First way is to use your registered validator to query directly to the TOP miner\n",
"- build a task (QnATask) with a \"prompt\" and optional \"datas\"\n",
"- query the network\n",
"- see response answer (1)\n",
"- see top citation (2)\n",
"- see full response object (3)"
]
},
{
"cell_type": "code",
"execution_count": 2,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"1 - Response showing answer from miner: \n",
"\t Meow eats grapes, berries, and occasional bananas.\n",
"2 - Response's topmost relevant citation from miner: \n",
"\t [{'context': 'meow prefers to eat grapes and berries, but also eats the occassional banana', 'source': 'source 3'}]\n",
"3 - Full response: \n",
"\t [QnATask(timeout=3.0, urls=[], datas=[{'source': 'source 1', 'context': 'Some irrelevant context'}, {'source': 'source 2', 'context': 'meow is a monkey in the jungle'}, {'source': 'source 3', 'context': 'meow prefers to eat grapes and berries, but also eats the occassional banana'}, {'source': 'source 4', 'context': 'meow climbs trees for fun'}, {'source': 'source 5', 'context': 'meow is afraid of snakes, but loves bunnies'}], prompt='hey, what does the meow eat?', response={'response': ' Meow eats grapes, berries, and occasional bananas.', 'citations': [{'context': 'meow prefers to eat grapes and berries, but also eats the occassional banana', 'source': 'source 3'}], 'context': 'meow prefers to eat grapes and berries, but also eats the occassional banana'}, miner_uids=[197])]\n"
]
}
],
"source": [
"task = QnATask(prompt=\"hey, what does the meow eat?\", \n",
" datas=[{\"source\": \"source 1\", \"context\": \"Some irrelevant context\"},\n",
" {\"source\": \"source 2\", \"context\": \"meow is a monkey in the jungle\"},\n",
" {\"source\": \"source 3\", \"context\": \"meow prefers to eat grapes and berries, but also eats the occassional banana\"},\n",
" {\"source\": \"source 4\", \"context\": \"meow climbs trees for fun\"},\n",
" {\"source\": \"source 5\", \"context\": \"meow is afraid of snakes, but loves bunnies\"}])\n",
"\n",
"responses = vali_dendrite.query(\n",
" axons=[subnet.axons[top_miner_uid]],\n",
" synapse=task,\n",
" deserialize=False,\n",
" timeout=task.timeout,\n",
")\n",
"\n",
"response = responses[0].response\n",
"print(\"1 - Response showing answer from miner: \\n\\t\", response['response'])\n",
"print(\"2 - Response's topmost relevant citation from miner: \\n\\t\", response['citations'])\n",
"print(\"3 - Full response: \\n\\t\", responses)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Second way is to use your validator (or any wallet) to query one of the subnet validators\n",
"- we'll use the same QnATask from above, but we'll change the prompt\n",
"- query the network via validator axon\n",
"- we are specifying the miner uid for our QnATask to be the TOP miner uid\n",
"- see response answer (1)\n",
"- see top citation (2)\n",
"- see full response object (3)"
]
},
{
"cell_type": "code",
"execution_count": 3,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"1 - Response showing answer: \n",
"\t Meow fears snakes the most.\n",
"2 - Response's topmost relevant citation: \n",
"\t [{'context': 'meow is afraid of snakes, but loves bunnies', 'source': 'source 5'}]\n",
"3 - Full response: \n",
"\t [QnATask(timeout=3.0, urls=[], datas=[{'source': 'source 1', 'context': 'Some irrelevant context'}, {'source': 'source 2', 'context': 'meow is a monkey in the jungle'}, {'source': 'source 3', 'context': 'meow prefers to eat grapes and berries, but also eats the occassional banana'}, {'source': 'source 4', 'context': 'meow climbs trees for fun'}, {'source': 'source 5', 'context': 'meow is afraid of snakes, but loves bunnies'}], prompt='What does meow fear the most?', response={'response': ' Meow fears snakes the most.', 'citations': [{'context': 'meow is afraid of snakes, but loves bunnies', 'source': 'source 5'}], 'context': 'meow is afraid of snakes, but loves bunnies'}, miner_uids=[])]\n"
]
}
],
"source": [
"task.prompt = \"What does meow fear the most?\"\n",
"\n",
"responses = vali_dendrite.query(\n",
" axons=[subnet.axons[0]],\n",
" synapse=task,\n",
" deserialize=False,\n",
" timeout=task.timeout,\n",
")\n",
"response = responses[0].response\n",
"print(\"1 - Response showing answer: \\n\\t\", response['response'])\n",
"print(\"2 - Response's topmost relevant citation: \\n\\t\", response['citations'])\n",
"print(\"3 - Full response: \\n\\t\", responses)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Demonstration of all task types and scoring / incentives ...\n",
"## Generating tasks from the Task API\n",
"We'll get tasks from the Task API that are far more complicated than the one above - \n",
"- Summarization Task\n",
"- QnA with Citations Task\n",
"- QnA Logic Task"
]
},
{
"cell_type": "markdown",
"metadata": {
"jp-MarkdownHeadingCollapsed": true
},
"source": [
"### Setting up ...\n",
"- methods to \n",
" - get the top miner, \n",
" - fetch a task \n",
" - get miner response and \n",
" - evaluate a task\n",
"- setup task IDs for QnA with Citations, Pet Tricks and Summarization"
]
},
{
"cell_type": "code",
"execution_count": 11,
"metadata": {},
"outputs": [],
"source": [
"qna_task = (3,1)\n",
"pet_tricks_task = (6,6)\n",
"summarization_task = (8,1)\n",
"\n",
"def get_top_miner_uid(subnet):\n",
" return subnet.I.argmax()\n",
"\n",
"def get_task(task_id, sub_task_id):\n",
" # keep trying in case it's being restarted \n",
" while True:\n",
" try:\n",
" resp = requests.post(\"https://roguetensor.com/api/task_api/get_new_task\", json={\"task_id\": task_id, \"sub_task_id\": sub_task_id}).json()\n",
" return resp\n",
" except:\n",
" pass\n",
"\n",
"def eval_task(task_id, response):\n",
" # keep trying in case it's being restarted \n",
" while True:\n",
" try:\n",
" resp = requests.post(\"https://roguetensor.com/api/task_api/evaluate_task_response\", json={\"task_id\": task_id, \"response\": response.toJSON()}).json()\n",
" return resp\n",
" except:\n",
" pass\n",
"\n",
"def get_miner_response_to_task(subnet, validator, miner_uid, task):\n",
" print(\"Fetching response from TOP miner: \", miner_uid)\n",
"\n",
" response = None\n",
" while not response:\n",
" response = asyncio.run(validator.call(\n",
" # Send the query to selected miner axons in the network.\n",
" target_axon=subnet.axons[miner_uid],\n",
" # Construct a query. \n",
" synapse=task,\n",
" # All responses have the deserialize function called on them before returning.\n",
" # You are encouraged to define your own deserialization function.\n",
" deserialize=True,\n",
" timeout=25.0\n",
" ))\n",
" return response\n",
"\n",
"def evaluate_miner(subnet, validator, miner_uid, task_id, sub_task_id):\n",
" task_json = get_task(task_id, sub_task_id)\n",
" gen_task_id = task_json[\"task\"][\"task_id\"]\n",
" task = QnATask(prompt=task_json['task']['prompt'], datas=task_json['task']['datas'], urls=task_json['task']['urls'])\n",
" print(\"Got task with prompt: \", task_json['task']['prompt'][:60] + \" ...\")\n",
" miner_response = get_miner_response_to_task(subnet, validator, miner_uid, task)\n",
" print(\"Miner response:\", miner_response.response['response'][:100] + \" ...\")\n",
" eval = eval_task(gen_task_id, miner_response)\n",
" return miner_response, *eval[\"result\"]"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Generated Summary Task Example:"
]
},
{
"cell_type": "code",
"execution_count": 12,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Got task with prompt: Summarize this and make sure to be concise: Did you attend ...\n",
"Fetching response from TOP miner: 197\n",
"Miner response: Pamela and Marie both did not attend the independence march. Pamela stayed at home due to concerns ...\n",
"Scores: 2.25 2.25\n"
]
}
],
"source": [
"task_id, sub_task_id = summarization_task\n",
"miner_response, score, max_score, results, correct_answer_optional = evaluate_miner(subnet, vali_dendrite, top_miner_uid, task_id, sub_task_id)\n",
"print(\"Scores: \", score, max_score)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"#### The results the miner would see:"
]
},
{
"cell_type": "code",
"execution_count": 13,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"Does not error\n",
"✅ You successfully responded to the request.\n",
"You received 0.25 of 0.25 reward.\n",
"Does not take a long time\n",
"✅ You responded to the request in 0.8873927593231201.\n",
"You received 0.5 of 0.5 reward.\n",
"Return summary shorter than original\n",
"✅ You responded with a valid summary length.\n",
"You received 0.5 of 0.5 reward.\n",
"Return valid summary\n",
"✅ You responded with a valid summary.\n",
"You received 1.0 of 1.0 reward.\n",
"\n"
],
"text/plain": [
"\u001b[1;34mDoes not error\u001b[0m\n",
"✅ \u001b[32mYou successfully responded to the request.\u001b[0m\n",
"You received \u001b[1;36m0.25\u001b[0m of \u001b[1;36m0.25\u001b[0m reward.\n",
"\u001b[1;34mDoes not take a long time\u001b[0m\n",
"✅ \u001b[32mYou responded to the request in \u001b[0m\u001b[1;32m0.8873927593231201\u001b[0m\u001b[32m.\u001b[0m\n",
"You received \u001b[1;36m0.5\u001b[0m of \u001b[1;36m0.5\u001b[0m reward.\n",
"\u001b[1;34mReturn summary shorter than original\u001b[0m\n",
"✅ \u001b[32mYou responded with a valid summary length.\u001b[0m\n",
"You received \u001b[1;36m0.5\u001b[0m of \u001b[1;36m0.5\u001b[0m reward.\n",
"\u001b[1;34mReturn valid summary\u001b[0m\n",
"✅ \u001b[32mYou responded with a valid summary.\u001b[0m\n",
"You received \u001b[1;36m1.0\u001b[0m of \u001b[1;36m1.0\u001b[0m reward.\n"
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"rprint((\"\\n\").join(results)) "
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Generated QnA with Citations Task Example:"
]
},
{
"cell_type": "code",
"execution_count": 14,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Got task with prompt: \"What was the purpose of integrating Crashlytics into Fireb ...\n",
"Fetching response from TOP miner: 197\n",
"Miner response: The purpose of integrating Crashlytics into Firebase was to bring the best of both platforms togeth ...\n",
"Scores: 5.25 5.25\n"
]
}
],
"source": [
"task_id, sub_task_id = qna_task\n",
"miner_response, score, max_score, results, correct_answer_optional = evaluate_miner(subnet, vali_dendrite, top_miner_uid, task_id, sub_task_id)\n",
"print(\"Scores: \", score, max_score)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"#### The results the miner would see:"
]
},
{
"cell_type": "code",
"execution_count": 15,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"Does not error\n",
"✅ You successfully responded to the request.\n",
"You received 0.25 of 0.25 reward.\n",
"Does not take a long time\n",
"✅ You responded to the request in 0.8485991954803467.\n",
"You received 0.5 of 0.5 reward.\n",
"Returns expected citation source(s)\n",
"✅ You correctly identified some or all of the correct citation sources (1/1 identified).\n",
"You received 1.5 of 1.5 reward.\n",
"Returns a relevant response\n",
"✅ You responded with a relevant response compared to the context.\n",
"You received 1.0 of 1.0 reward.\n",
"Returns a unique response\n",
"✅ You responded with a novel response compared to the context.\n",
"You received 1.0 of 1.0 reward.\n",
"Returns valid response\n",
"✅ You responded with a valid response.\n",
"You received 1.0 of 1.0 reward.\n",
"\n"
],
"text/plain": [
"\u001b[1;34mDoes not error\u001b[0m\n",
"✅ \u001b[32mYou successfully responded to the request.\u001b[0m\n",
"You received \u001b[1;36m0.25\u001b[0m of \u001b[1;36m0.25\u001b[0m reward.\n",
"\u001b[1;34mDoes not take a long time\u001b[0m\n",
"✅ \u001b[32mYou responded to the request in \u001b[0m\u001b[1;32m0.8485991954803467\u001b[0m\u001b[32m.\u001b[0m\n",
"You received \u001b[1;36m0.5\u001b[0m of \u001b[1;36m0.5\u001b[0m reward.\n",
"\u001b[1;34mReturns expected citation \u001b[0m\u001b[1;34msource\u001b[0m\u001b[1;34m(\u001b[0m\u001b[1;34ms\u001b[0m\u001b[1;34m)\u001b[0m\n",
"✅ \u001b[32mYou correctly identified some or all of the correct citation sources \u001b[0m\u001b[1;32m(\u001b[0m\u001b[1;32m1\u001b[0m\u001b[32m/\u001b[0m\u001b[1;32m1\u001b[0m\u001b[32m identified\u001b[0m\u001b[1;32m)\u001b[0m\u001b[32m.\u001b[0m\n",
"You received \u001b[1;36m1.5\u001b[0m of \u001b[1;36m1.5\u001b[0m reward.\n",
"\u001b[1;34mReturns a relevant response\u001b[0m\n",
"✅ \u001b[32mYou responded with a relevant response compared to the context.\u001b[0m\n",
"You received \u001b[1;36m1.0\u001b[0m of \u001b[1;36m1.0\u001b[0m reward.\n",
"\u001b[1;34mReturns a unique response\u001b[0m\n",
"✅ \u001b[32mYou responded with a novel response compared to the context.\u001b[0m\n",
"You received \u001b[1;36m1.0\u001b[0m of \u001b[1;36m1.0\u001b[0m reward.\n",
"\u001b[1;34mReturns valid response\u001b[0m\n",
"✅ \u001b[32mYou responded with a valid response.\u001b[0m\n",
"You received \u001b[1;36m1.0\u001b[0m of \u001b[1;36m1.0\u001b[0m reward.\n"
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"rprint((\"\\n\").join(results)) "
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Generated Pet Tricks QnA Logic Task Example:"
]
},
{
"cell_type": "code",
"execution_count": 16,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Got task with prompt: Given the following Trick Descriptions with numerical IDs:\n",
" ...\n",
"Fetching response from TOP miner: 197\n",
"Miner response: 4 ...\n",
"Scores: 1.75 1.75\n"
]
}
],
"source": [
"task_id, sub_task_id = pet_tricks_task\n",
"miner_response, score, max_score, results, correct_answer_optional = evaluate_miner(subnet, vali_dendrite, top_miner_uid, task_id, sub_task_id)\n",
"print(\"Scores: \", score, max_score)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"#### The results the miner would see:"
]
},
{
"cell_type": "code",
"execution_count": 17,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"Does not error\n",
"✅ You successfully responded to the request.\n",
"You received 0.25 of 0.25 reward.\n",
"Does not take a long time\n",
"✅ You responded to the request in 0.3985426425933838.\n",
"You received 0.5 of 0.5 reward.\n",
"Returns expected value\n",
"✅ You responded with a valid answer.\n",
"You received 1.0 of 1.0 reward.\n",
"\n"
],
"text/plain": [
"\u001b[1;34mDoes not error\u001b[0m\n",
"✅ \u001b[32mYou successfully responded to the request.\u001b[0m\n",
"You received \u001b[1;36m0.25\u001b[0m of \u001b[1;36m0.25\u001b[0m reward.\n",
"\u001b[1;34mDoes not take a long time\u001b[0m\n",
"✅ \u001b[32mYou responded to the request in \u001b[0m\u001b[1;32m0.3985426425933838\u001b[0m\u001b[32m.\u001b[0m\n",
"You received \u001b[1;36m0.5\u001b[0m of \u001b[1;36m0.5\u001b[0m reward.\n",
"\u001b[1;34mReturns expected value\u001b[0m\n",
"✅ \u001b[32mYou responded with a valid answer.\u001b[0m\n",
"You received \u001b[1;36m1.0\u001b[0m of \u001b[1;36m1.0\u001b[0m reward.\n"
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"rprint((\"\\n\").join(results)) "
]
},
{
"cell_type": "markdown",
"metadata": {
"jp-MarkdownHeadingCollapsed": true
},
"source": [
"## That's It!\n",
"\n",
"You saw how to:\n",
" 1) Query the top miner uid\n",
" 2) Demonstrate each reward/penalty mechanism/Scoring of the top miner response\n",
" 3) Query the subnet 2 different ways (as a validator to a miner and through a registered validator axon)"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3 (ipykernel)",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.10.12"
}
},
"nbformat": 4,
"nbformat_minor": 4
}