codebert-embed-crewai-base / README.md

itsanan

Add new SentenceTransformer model

fbef0ac verified 17 days ago

preview code

raw

history blame contribute delete

44.7 kB

metadata

language:
  - en
license: apache-2.0
tags:
  - sentence-transformers
  - sentence-similarity
  - feature-extraction
  - dense
  - generated_from_trainer
  - dataset_size:900
  - loss:MatryoshkaLoss
  - loss:MultipleNegativesRankingLoss
base_model: microsoft/codebert-base
widget:
  - source_sentence: Explain the test_code_docs_search_tool logic
    sentences:
      - |-
        def test_anthropic_call_with_interceptor_tracks_requests(self) -> None:
                """Test that interceptor tracks Anthropic API requests."""
                interceptor = AnthropicTestInterceptor()
                llm = LLM(model="anthropic/claude-3-5-haiku-20241022", interceptor=interceptor)

                # Make a simple completion call
                result = llm.call(
                    messages=[{"role": "user", "content": "Say 'Hello World' and nothing else"}]
                )

                # Verify custom headers were added
                for request in interceptor.outbound_calls:
                    assert "X-Anthropic-Interceptor" in request.headers
                    assert request.headers["X-Anthropic-Interceptor"] == "anthropic-test-value"
                    assert "X-Request-ID" in request.headers
                    assert request.headers["X-Request-ID"] == "test-request-456"

                # Verify response was tracked
                for response in interceptor.inbound_calls:
                    assert "X-Response-Tracked" in response.headers
                    assert response.headers["X-Response-Tracked"] == "true"

                # Verify result is valid
                assert result is not None
                assert isinstance(result, str)
                assert len(result) > 0
      - |-
        def on_inbound(self, message: httpx.Response) -> httpx.Response:
                """Pass through inbound response.

                Args:
                    message: The inbound response.

                Returns:
                    The response unchanged.
                """
                return message
      - |-
        def test_code_docs_search_tool(mock_adapter):
            mock_adapter.query.return_value = "test documentation"

            docs_url = "https://crewai.com/any-docs-url"
            search_query = "test documentation"
            tool = CodeDocsSearchTool(docs_url=docs_url, adapter=mock_adapter)
            result = tool._run(search_query=search_query)
            assert "test documentation" in result
            mock_adapter.add.assert_called_once_with(docs_url, data_type=DataType.DOCS_SITE)
            mock_adapter.query.assert_called_once_with(
                search_query, similarity_threshold=0.6, limit=5
            )

            mock_adapter.query.reset_mock()
            mock_adapter.add.reset_mock()

            tool = CodeDocsSearchTool(adapter=mock_adapter)
            result = tool._run(docs_url=docs_url, search_query=search_query)
            assert "test documentation" in result
            mock_adapter.add.assert_called_once_with(docs_url, data_type=DataType.DOCS_SITE)
            mock_adapter.query.assert_called_once_with(
                search_query, similarity_threshold=0.6, limit=5
            )
  - source_sentence: Explain the test_openai_get_client_params_with_base_url_priority logic
    sentences:
      - |-
        def test_openai_get_client_params_with_base_url_priority():
            """
            Test that base_url takes priority over api_base in _get_client_params
            """
            llm = OpenAICompletion(
                model="gpt-4o",
                base_url="https://priority.openai.com/v1",
                api_base="https://fallback.openai.com/v1",
            )
            client_params = llm._get_client_params()
            assert client_params["base_url"] == "https://priority.openai.com/v1"
      - |-
        def get_context_window_size(self) -> int:
                """Get the context window size for the LLM.

                Returns:
                    The number of tokens/characters the model can handle.
                """
                # Default implementation - subclasses should override with model-specific values
                return DEFAULT_CONTEXT_WINDOW_SIZE
      - |-
        def _inject_date_to_task(self, task: Task) -> None:
                """Inject the current date into the task description if inject_date is enabled."""
                if self.inject_date:
                    from datetime import datetime

                    try:
                        valid_format_codes = [
                            "%Y",
                            "%m",
                            "%d",
                            "%H",
                            "%M",
                            "%S",
                            "%B",
                            "%b",
                            "%A",
                            "%a",
                        ]
                        is_valid = any(code in self.date_format for code in valid_format_codes)

                        if not is_valid:
                            raise ValueError(f"Invalid date format: {self.date_format}")

                        current_date = datetime.now().strftime(self.date_format)
                        task.description += f"\n\nCurrent Date: {current_date}"
                    except Exception as e:
                        self._logger.log("warning", f"Failed to inject date: {e!s}")
  - source_sentence: How to implement async _get_connection?
    sentences:
      - |-
        def mock_env():
            with patch.dict(os.environ, {"CREWAI_PERSONAL_ACCESS_TOKEN": "test_token"}):
                os.environ.pop("CREWAI_PLUS_URL", None)
                yield
      - |-
        async def _get_connection(self) -> SnowflakeConnection:
                """Get a connection from the pool or create a new one."""
                if self._pool_lock is None:
                    raise RuntimeError("Pool lock not initialized")
                if self._connection_pool is None:
                    raise RuntimeError("Connection pool not initialized")
                async with self._pool_lock:
                    if not self._connection_pool:
                        conn = await asyncio.get_event_loop().run_in_executor(
                            self._thread_pool, self._create_connection
                        )
                        self._connection_pool.append(conn)
                    return self._connection_pool.pop()
      - >-
        async def _arun(self, selector: str, thread_id: str = "default",
        **kwargs) -> str:
                """Use the async tool."""
                try:
                    # Get the current page
                    page = await self.get_async_page(thread_id)

                    # Click on the element
                    selector_effective = self._selector_effective(selector=selector)
                    from playwright.async_api import TimeoutError as PlaywrightTimeoutError

                    try:
                        await page.click(
                            selector_effective,
                            strict=self.playwright_strict,
                            timeout=self.playwright_timeout,
                        )
                    except PlaywrightTimeoutError:
                        return f"Unable to click on element '{selector}'"
                    except Exception as click_error:
                        return f"Unable to click on element '{selector}': {click_error!s}"

                    return f"Clicked element '{selector}'"
                except Exception as e:
                    return f"Error clicking on element: {e!s}"
  - source_sentence: Example usage of test_personal_access_token_from_environment
    sentences:
      - |-
        async def close(self):
                                return None
      - |-
        def test_structured_state_persistence(tmp_path):
            """Test persistence with Pydantic model state."""
            db_path = os.path.join(tmp_path, "test_flows.db")
            persistence = SQLiteFlowPersistence(db_path)

            class StructuredFlow(Flow[TestState]):
                initial_state = TestState

                @start()
                @persist(persistence)
                def count_up(self):
                    self.state.counter += 1
                    self.state.message = f"Count is {self.state.counter}"

            # Run flow and verify state changes are saved
            flow = StructuredFlow(persistence=persistence)
            flow.kickoff()

            # Load and verify state
            saved_state = persistence.load_state(flow.state.id)
            assert saved_state is not None
            assert saved_state["counter"] == 1
            assert saved_state["message"] == "Count is 1"
      - |-
        def test_personal_access_token_from_environment(tool):
            assert tool.personal_access_token == "test_token"
  - source_sentence: Best practices for handle_a2a_polling_started
    sentences:
      - |-
        def external_supported_storages() -> dict[str, Any]:
                return {
                    "mem0": ExternalMemory._configure_mem0,
                }
      - |-
        def handle_a2a_polling_started(
                self,
                task_id: str,
                polling_interval: float,
                endpoint: str,
            ) -> None:
                """Handle A2A polling started event with panel display."""
                content = Text()
                content.append("A2A Polling Started\n", style="cyan bold")
                content.append("Task ID: ", style="white")
                content.append(f"{task_id[:8]}...\n", style="cyan")
                content.append("Interval: ", style="white")
                content.append(f"{polling_interval}s\n", style="cyan")

                self.print_panel(content, "⏳ A2A Polling", "cyan")
      - |-
        def test_agent_with_knowledge_sources_generate_search_query():
            content = "Brandon's favorite color is red and he likes Mexican food."
            string_source = StringKnowledgeSource(content=content)

            with (
                patch("crewai.knowledge") as mock_knowledge,
                patch(
                    "crewai.knowledge.storage.knowledge_storage.KnowledgeStorage"
                ) as mock_knowledge_storage,
                patch(
                    "crewai.knowledge.source.base_knowledge_source.KnowledgeStorage"
                ) as mock_base_knowledge_storage,
                patch("crewai.rag.chromadb.client.ChromaDBClient") as mock_chromadb,
            ):
                mock_knowledge_instance = mock_knowledge.return_value
                mock_knowledge_instance.sources = [string_source]
                mock_knowledge_instance.query.return_value = [{"content": content}]

                mock_storage_instance = mock_knowledge_storage.return_value
                mock_storage_instance.sources = [string_source]
                mock_storage_instance.query.return_value = [{"content": content}]
                mock_storage_instance.save.return_value = None

                mock_chromadb_instance = mock_chromadb.return_value
                mock_chromadb_instance.add_documents.return_value = None

                mock_base_knowledge_storage.return_value = mock_storage_instance

                agent = Agent(
                    role="Information Agent with extensive role description that is longer than 80 characters",
                    goal="Provide information based on knowledge sources",
                    backstory="You have access to specific knowledge sources.",
                    llm=LLM(model="gpt-4o-mini"),
                    knowledge_sources=[string_source],
                )

                task = Task(
                    description="What is Brandon's favorite color?",
                    expected_output="The answer to the question, in a format like this: `{{name: str, favorite_color: str}}`",
                    agent=agent,
                )

                crew = Crew(agents=[agent], tasks=[task])
                result = crew.kickoff()

                # Updated assertion to check the JSON content
                assert "Brandon" in str(agent.knowledge_search_query)
                assert "favorite color" in str(agent.knowledge_search_query)

                assert "red" in result.raw.lower()
pipeline_tag: sentence-similarity
library_name: sentence-transformers
metrics:
  - cosine_accuracy@1
  - cosine_accuracy@3
  - cosine_accuracy@5
  - cosine_accuracy@10
  - cosine_precision@1
  - cosine_precision@3
  - cosine_precision@5
  - cosine_precision@10
  - cosine_recall@1
  - cosine_recall@3
  - cosine_recall@5
  - cosine_recall@10
  - cosine_ndcg@10
  - cosine_mrr@10
  - cosine_map@100
model-index:
  - name: CodeBERT Fine-tuned on CrewAI
    results:
      - task:
          type: information-retrieval
          name: Information Retrieval
        dataset:
          name: dim 768
          type: dim_768
        metrics:
          - type: cosine_accuracy@1
            value: 0.57
            name: Cosine Accuracy@1
          - type: cosine_accuracy@3
            value: 0.57
            name: Cosine Accuracy@3
          - type: cosine_accuracy@5
            value: 0.57
            name: Cosine Accuracy@5
          - type: cosine_accuracy@10
            value: 0.65
            name: Cosine Accuracy@10
          - type: cosine_precision@1
            value: 0.57
            name: Cosine Precision@1
          - type: cosine_precision@3
            value: 0.57
            name: Cosine Precision@3
          - type: cosine_precision@5
            value: 0.57
            name: Cosine Precision@5
          - type: cosine_precision@10
            value: 0.325
            name: Cosine Precision@10
          - type: cosine_recall@1
            value: 0.11399999999999996
            name: Cosine Recall@1
          - type: cosine_recall@3
            value: 0.3420000000000001
            name: Cosine Recall@3
          - type: cosine_recall@5
            value: 0.57
            name: Cosine Recall@5
          - type: cosine_recall@10
            value: 0.65
            name: Cosine Recall@10
          - type: cosine_ndcg@10
            value: 0.6132795614223119
            name: Cosine Ndcg@10
          - type: cosine_mrr@10
            value: 0.5833333333333334
            name: Cosine Mrr@10
          - type: cosine_map@100
            value: 0.6323349876959563
            name: Cosine Map@100
      - task:
          type: information-retrieval
          name: Information Retrieval
        dataset:
          name: dim 512
          type: dim_512
        metrics:
          - type: cosine_accuracy@1
            value: 0.56
            name: Cosine Accuracy@1
          - type: cosine_accuracy@3
            value: 0.56
            name: Cosine Accuracy@3
          - type: cosine_accuracy@5
            value: 0.56
            name: Cosine Accuracy@5
          - type: cosine_accuracy@10
            value: 0.68
            name: Cosine Accuracy@10
          - type: cosine_precision@1
            value: 0.56
            name: Cosine Precision@1
          - type: cosine_precision@3
            value: 0.56
            name: Cosine Precision@3
          - type: cosine_precision@5
            value: 0.56
            name: Cosine Precision@5
          - type: cosine_precision@10
            value: 0.34
            name: Cosine Precision@10
          - type: cosine_recall@1
            value: 0.11199999999999999
            name: Cosine Recall@1
          - type: cosine_recall@3
            value: 0.336
            name: Cosine Recall@3
          - type: cosine_recall@5
            value: 0.56
            name: Cosine Recall@5
          - type: cosine_recall@10
            value: 0.68
            name: Cosine Recall@10
          - type: cosine_ndcg@10
            value: 0.6249193421334678
            name: Cosine Ndcg@10
          - type: cosine_mrr@10
            value: 0.5799999999999998
            name: Cosine Mrr@10
          - type: cosine_map@100
            value: 0.6328444860345127
            name: Cosine Map@100
      - task:
          type: information-retrieval
          name: Information Retrieval
        dataset:
          name: dim 256
          type: dim_256
        metrics:
          - type: cosine_accuracy@1
            value: 0.54
            name: Cosine Accuracy@1
          - type: cosine_accuracy@3
            value: 0.54
            name: Cosine Accuracy@3
          - type: cosine_accuracy@5
            value: 0.54
            name: Cosine Accuracy@5
          - type: cosine_accuracy@10
            value: 0.67
            name: Cosine Accuracy@10
          - type: cosine_precision@1
            value: 0.54
            name: Cosine Precision@1
          - type: cosine_precision@3
            value: 0.54
            name: Cosine Precision@3
          - type: cosine_precision@5
            value: 0.54
            name: Cosine Precision@5
          - type: cosine_precision@10
            value: 0.335
            name: Cosine Precision@10
          - type: cosine_recall@1
            value: 0.10799999999999997
            name: Cosine Recall@1
          - type: cosine_recall@3
            value: 0.324
            name: Cosine Recall@3
          - type: cosine_recall@5
            value: 0.54
            name: Cosine Recall@5
          - type: cosine_recall@10
            value: 0.67
            name: Cosine Recall@10
          - type: cosine_ndcg@10
            value: 0.6103292873112568
            name: Cosine Ndcg@10
          - type: cosine_mrr@10
            value: 0.5616666666666664
            name: Cosine Mrr@10
          - type: cosine_map@100
            value: 0.622676615058847
            name: Cosine Map@100
      - task:
          type: information-retrieval
          name: Information Retrieval
        dataset:
          name: dim 128
          type: dim_128
        metrics:
          - type: cosine_accuracy@1
            value: 0.47
            name: Cosine Accuracy@1
          - type: cosine_accuracy@3
            value: 0.47
            name: Cosine Accuracy@3
          - type: cosine_accuracy@5
            value: 0.47
            name: Cosine Accuracy@5
          - type: cosine_accuracy@10
            value: 0.58
            name: Cosine Accuracy@10
          - type: cosine_precision@1
            value: 0.47
            name: Cosine Precision@1
          - type: cosine_precision@3
            value: 0.47
            name: Cosine Precision@3
          - type: cosine_precision@5
            value: 0.47
            name: Cosine Precision@5
          - type: cosine_precision@10
            value: 0.29
            name: Cosine Precision@10
          - type: cosine_recall@1
            value: 0.09399999999999999
            name: Cosine Recall@1
          - type: cosine_recall@3
            value: 0.28200000000000003
            name: Cosine Recall@3
          - type: cosine_recall@5
            value: 0.47
            name: Cosine Recall@5
          - type: cosine_recall@10
            value: 0.58
            name: Cosine Recall@10
          - type: cosine_ndcg@10
            value: 0.5295093969556788
            name: Cosine Ndcg@10
          - type: cosine_mrr@10
            value: 0.48833333333333323
            name: Cosine Mrr@10
          - type: cosine_map@100
            value: 0.5581789904714569
            name: Cosine Map@100
      - task:
          type: information-retrieval
          name: Information Retrieval
        dataset:
          name: dim 64
          type: dim_64
        metrics:
          - type: cosine_accuracy@1
            value: 0.5
            name: Cosine Accuracy@1
          - type: cosine_accuracy@3
            value: 0.5
            name: Cosine Accuracy@3
          - type: cosine_accuracy@5
            value: 0.5
            name: Cosine Accuracy@5
          - type: cosine_accuracy@10
            value: 0.6
            name: Cosine Accuracy@10
          - type: cosine_precision@1
            value: 0.5
            name: Cosine Precision@1
          - type: cosine_precision@3
            value: 0.5
            name: Cosine Precision@3
          - type: cosine_precision@5
            value: 0.5
            name: Cosine Precision@5
          - type: cosine_precision@10
            value: 0.3
            name: Cosine Precision@10
          - type: cosine_recall@1
            value: 0.1
            name: Cosine Recall@1
          - type: cosine_recall@3
            value: 0.3
            name: Cosine Recall@3
          - type: cosine_recall@5
            value: 0.5
            name: Cosine Recall@5
          - type: cosine_recall@10
            value: 0.6
            name: Cosine Recall@10
          - type: cosine_ndcg@10
            value: 0.5540994517778899
            name: Cosine Ndcg@10
          - type: cosine_mrr@10
            value: 0.5166666666666665
            name: Cosine Mrr@10
          - type: cosine_map@100
            value: 0.5748485156077728
            name: Cosine Map@100

CodeBERT Fine-tuned on CrewAI

This is a sentence-transformers model finetuned from microsoft/codebert-base. It maps sentences & paragraphs to a 768-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more.

Model Details

Model Description

Model Type: Sentence Transformer
Base model: microsoft/codebert-base
Maximum Sequence Length: 512 tokens
Output Dimensionality: 768 dimensions
Similarity Function: Cosine Similarity
Language: en
License: apache-2.0

Model Sources

Documentation: Sentence Transformers Documentation
Repository: Sentence Transformers on GitHub
Hugging Face: Sentence Transformers on Hugging Face

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'max_seq_length': 512, 'do_lower_case': False, 'architecture': 'RobertaModel'})
  (1): Pooling({'word_embedding_dimension': 768, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': True, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("itsanan/codebert-embed-crewai-base")
# Run inference
sentences = [
    'Best practices for handle_a2a_polling_started',
    'def handle_a2a_polling_started(\n        self,\n        task_id: str,\n        polling_interval: float,\n        endpoint: str,\n    ) -> None:\n        """Handle A2A polling started event with panel display."""\n        content = Text()\n        content.append("A2A Polling Started\\n", style="cyan bold")\n        content.append("Task ID: ", style="white")\n        content.append(f"{task_id[:8]}...\\n", style="cyan")\n        content.append("Interval: ", style="white")\n        content.append(f"{polling_interval}s\\n", style="cyan")\n\n        self.print_panel(content, "⏳ A2A Polling", "cyan")',
    'def test_agent_with_knowledge_sources_generate_search_query():\n    content = "Brandon\'s favorite color is red and he likes Mexican food."\n    string_source = StringKnowledgeSource(content=content)\n\n    with (\n        patch("crewai.knowledge") as mock_knowledge,\n        patch(\n            "crewai.knowledge.storage.knowledge_storage.KnowledgeStorage"\n        ) as mock_knowledge_storage,\n        patch(\n            "crewai.knowledge.source.base_knowledge_source.KnowledgeStorage"\n        ) as mock_base_knowledge_storage,\n        patch("crewai.rag.chromadb.client.ChromaDBClient") as mock_chromadb,\n    ):\n        mock_knowledge_instance = mock_knowledge.return_value\n        mock_knowledge_instance.sources = [string_source]\n        mock_knowledge_instance.query.return_value = [{"content": content}]\n\n        mock_storage_instance = mock_knowledge_storage.return_value\n        mock_storage_instance.sources = [string_source]\n        mock_storage_instance.query.return_value = [{"content": content}]\n        mock_storage_instance.save.return_value = None\n\n        mock_chromadb_instance = mock_chromadb.return_value\n        mock_chromadb_instance.add_documents.return_value = None\n\n        mock_base_knowledge_storage.return_value = mock_storage_instance\n\n        agent = Agent(\n            role="Information Agent with extensive role description that is longer than 80 characters",\n            goal="Provide information based on knowledge sources",\n            backstory="You have access to specific knowledge sources.",\n            llm=LLM(model="gpt-4o-mini"),\n            knowledge_sources=[string_source],\n        )\n\n        task = Task(\n            description="What is Brandon\'s favorite color?",\n            expected_output="The answer to the question, in a format like this: `{{name: str, favorite_color: str}}`",\n            agent=agent,\n        )\n\n        crew = Crew(agents=[agent], tasks=[task])\n        result = crew.kickoff()\n\n        # Updated assertion to check the JSON content\n        assert "Brandon" in str(agent.knowledge_search_query)\n        assert "favorite color" in str(agent.knowledge_search_query)\n\n        assert "red" in result.raw.lower()',
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 768]

# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings, embeddings)
print(similarities)
# tensor([[1.0000, 0.7350, 0.6480],
#         [0.7350, 1.0000, 0.8133],
#         [0.6480, 0.8133, 1.0000]])

Evaluation

Metrics

Information Retrieval

Dataset: dim_768
Evaluated with InformationRetrievalEvaluator with these parameters:
```
{
    "truncate_dim": 768
}
```

Metric	Value
cosine_accuracy@1	0.57
cosine_accuracy@3	0.57
cosine_accuracy@5	0.57
cosine_accuracy@10	0.65
cosine_precision@1	0.57
cosine_precision@3	0.57
cosine_precision@5	0.57
cosine_precision@10	0.325
cosine_recall@1	0.114
cosine_recall@3	0.342
cosine_recall@5	0.57
cosine_recall@10	0.65
cosine_ndcg@10	0.6133
cosine_mrr@10	0.5833
cosine_map@100	0.6323

Information Retrieval

Dataset: dim_512
Evaluated with InformationRetrievalEvaluator with these parameters:
```
{
    "truncate_dim": 512
}
```

Metric	Value
cosine_accuracy@1	0.56
cosine_accuracy@3	0.56
cosine_accuracy@5	0.56
cosine_accuracy@10	0.68
cosine_precision@1	0.56
cosine_precision@3	0.56
cosine_precision@5	0.56
cosine_precision@10	0.34
cosine_recall@1	0.112
cosine_recall@3	0.336
cosine_recall@5	0.56
cosine_recall@10	0.68
cosine_ndcg@10	0.6249
cosine_mrr@10	0.58
cosine_map@100	0.6328

Information Retrieval

Dataset: dim_256
Evaluated with InformationRetrievalEvaluator with these parameters:
```
{
    "truncate_dim": 256
}
```

Metric	Value
cosine_accuracy@1	0.54
cosine_accuracy@3	0.54
cosine_accuracy@5	0.54
cosine_accuracy@10	0.67
cosine_precision@1	0.54
cosine_precision@3	0.54
cosine_precision@5	0.54
cosine_precision@10	0.335
cosine_recall@1	0.108
cosine_recall@3	0.324
cosine_recall@5	0.54
cosine_recall@10	0.67
cosine_ndcg@10	0.6103
cosine_mrr@10	0.5617
cosine_map@100	0.6227

Information Retrieval

Dataset: dim_128
Evaluated with InformationRetrievalEvaluator with these parameters:
```
{
    "truncate_dim": 128
}
```

Metric	Value
cosine_accuracy@1	0.47
cosine_accuracy@3	0.47
cosine_accuracy@5	0.47
cosine_accuracy@10	0.58
cosine_precision@1	0.47
cosine_precision@3	0.47
cosine_precision@5	0.47
cosine_precision@10	0.29
cosine_recall@1	0.094
cosine_recall@3	0.282
cosine_recall@5	0.47
cosine_recall@10	0.58
cosine_ndcg@10	0.5295
cosine_mrr@10	0.4883
cosine_map@100	0.5582

Information Retrieval

Dataset: dim_64
Evaluated with InformationRetrievalEvaluator with these parameters:
```
{
    "truncate_dim": 64
}
```

Metric	Value
cosine_accuracy@1	0.5
cosine_accuracy@3	0.5
cosine_accuracy@5	0.5
cosine_accuracy@10	0.6
cosine_precision@1	0.5
cosine_precision@3	0.5
cosine_precision@5	0.5
cosine_precision@10	0.3
cosine_recall@1	0.1
cosine_recall@3	0.3
cosine_recall@5	0.5
cosine_recall@10	0.6
cosine_ndcg@10	0.5541
cosine_mrr@10	0.5167
cosine_map@100	0.5748

Training Details

Training Dataset

Unnamed Dataset

Size: 900 training samples
Columns: anchor and positive
Approximate statistics based on the first 900 samples:
anchor positive
type string string
details
min: 6 tokens
mean: 13.96 tokens
max: 141 tokens

min: 20 tokens
mean: 254.94 tokens
max: 512 tokens

	anchor	positive
type	string	string
details	min: 6 tokens mean: 13.96 tokens max: 141 tokens	min: 20 tokens mean: 254.94 tokens max: 512 tokens

Samples:

anchor	positive
`Example usage of DeeplyNestedFlow`	`class DeeplyNestedFlow(Flow): @start() def a(self): execution_order.append("a") @start() def b(self): execution_order.append("b") @start() def c(self): execution_order.append("c") @start() def d(self): execution_order.append("d") # Nested: (a AND b) OR (c AND d) @listen(or_(and_(a, b), and_(c, d))) def result(self): execution_order.append("result")`
`Explain the test_agent_with_knowledge_sources_generate_search_query logic`	def test_agent_with_knowledge_sources_generate_search_query(): content = "Brandon's favorite color is red and he likes Mexican food." string_source = StringKnowledgeSource(content=content) with ( patch("crewai.knowledge") as mock_knowledge, patch( "crewai.knowledge.storage.knowledge_storage.KnowledgeStorage" ) as mock_knowledge_storage, patch( "crewai.knowledge.source.base_knowledge_source.KnowledgeStorage" ) as mock_base_knowledge_storage, patch("crewai.rag.chromadb.client.ChromaDBClient") as mock_chromadb, ): mock_knowledge_instance = mock_knowledge.return_value mock_knowledge_instance.sources = [string_source] mock_knowledge_instance.query.return_value = [{"content": content}] mock_storage_instance = mock_knowledge_storage.return_value mock_storage_instance.sources = [string_source] mock_storage_instance.query.return_value = [{"content": content}]...
`Example usage of agent`	`def agent(self) -> Agent \| None: """Get the current agent associated with this memory.""" return self._agent`

Loss: MatryoshkaLoss with these parameters:

{
    "loss": "MultipleNegativesRankingLoss",
    "matryoshka_dims": [
        768,
        512,
        256,
        128,
        64
    ],
    "matryoshka_weights": [
        1,
        1,
        1,
        1,
        1
    ],
    "n_dims_per_step": -1
}

Training Hyperparameters

Non-Default Hyperparameters

eval_strategy: epoch
per_device_train_batch_size: 4
per_device_eval_batch_size: 4
gradient_accumulation_steps: 16
learning_rate: 2e-05
num_train_epochs: 4
lr_scheduler_type: cosine
warmup_ratio: 0.1
fp16: True
load_best_model_at_end: True
optim: adamw_torch
batch_sampler: no_duplicates

All Hyperparameters

Click to expand

overwrite_output_dir: False
do_predict: False
eval_strategy: epoch
prediction_loss_only: True
per_device_train_batch_size: 4
per_device_eval_batch_size: 4
per_gpu_train_batch_size: None
per_gpu_eval_batch_size: None
gradient_accumulation_steps: 16
eval_accumulation_steps: None
torch_empty_cache_steps: None
learning_rate: 2e-05
weight_decay: 0.0
adam_beta1: 0.9
adam_beta2: 0.999
adam_epsilon: 1e-08
max_grad_norm: 1.0
num_train_epochs: 4
max_steps: -1
lr_scheduler_type: cosine
lr_scheduler_kwargs: None
warmup_ratio: 0.1
warmup_steps: 0
log_level: passive
log_level_replica: warning
log_on_each_node: True
logging_nan_inf_filter: True
save_safetensors: True
save_on_each_node: False
save_only_model: False
restore_callback_states_from_checkpoint: False
no_cuda: False
use_cpu: False
use_mps_device: False
seed: 42
data_seed: None
jit_mode_eval: False
bf16: False
fp16: True
fp16_opt_level: O1
half_precision_backend: auto
bf16_full_eval: False
fp16_full_eval: False
tf32: None
local_rank: 0
ddp_backend: None
tpu_num_cores: None
tpu_metrics_debug: False
debug: []
dataloader_drop_last: False
dataloader_num_workers: 0
dataloader_prefetch_factor: None
past_index: -1
disable_tqdm: False
remove_unused_columns: True
label_names: None
load_best_model_at_end: True
ignore_data_skip: False
fsdp: []
fsdp_min_num_params: 0
fsdp_config: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}
fsdp_transformer_layer_cls_to_wrap: None
accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
parallelism_config: None
deepspeed: None
label_smoothing_factor: 0.0
optim: adamw_torch
optim_args: None
adafactor: False
group_by_length: False
length_column_name: length
project: huggingface
trackio_space_id: trackio
ddp_find_unused_parameters: None
ddp_bucket_cap_mb: None
ddp_broadcast_buffers: False
dataloader_pin_memory: True
dataloader_persistent_workers: False
skip_memory_metrics: True
use_legacy_prediction_loop: False
push_to_hub: False
resume_from_checkpoint: None
hub_model_id: None
hub_strategy: every_save
hub_private_repo: None
hub_always_push: False
hub_revision: None
gradient_checkpointing: False
gradient_checkpointing_kwargs: None
include_inputs_for_metrics: False
include_for_metrics: []
eval_do_concat_batches: True
fp16_backend: auto
push_to_hub_model_id: None
push_to_hub_organization: None
mp_parameters:
auto_find_batch_size: False
full_determinism: False
torchdynamo: None
ray_scope: last
ddp_timeout: 1800
torch_compile: False
torch_compile_backend: None
torch_compile_mode: None
include_tokens_per_second: False
include_num_input_tokens_seen: no
neftune_noise_alpha: None
optim_target_modules: None
batch_eval_metrics: False
eval_on_start: False
use_liger_kernel: False
liger_kernel_config: None
eval_use_gather_object: False
average_tokens_across_devices: True
prompts: None
batch_sampler: no_duplicates
multi_dataset_batch_sampler: proportional
router_mapping: {}
learning_rate_mapping: {}

Training Logs

Epoch	Step	Training Loss	dim_768_cosine_ndcg@10	dim_512_cosine_ndcg@10	dim_256_cosine_ndcg@10	dim_128_cosine_ndcg@10	dim_64_cosine_ndcg@10
0.7111	10	7.1051	-	-	-	-	-
1.0	15	-	0.1170	0.06	0.0608	0.0825	0.0762
1.3556	20	6.4716	-	-	-	-	-
2.0	30	5.4463	0.1879	0.1770	0.1625	0.1816	0.1987
2.7111	40	3.7856	-	-	-	-	-
3.0	45	-	0.4987	0.5133	0.4587	0.4249	0.4425
3.3556	50	2.4942	-	-	-	-	-
4.0	60	1.71	0.6133	0.6249	0.6103	0.5295	0.5541

The bold row denotes the saved checkpoint.

Framework Versions

Python: 3.12.12
Sentence Transformers: 5.2.2
Transformers: 4.57.6
PyTorch: 2.9.0+cu126
Accelerate: 1.12.0
Datasets: 4.0.0
Tokenizers: 0.22.2

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

MatryoshkaLoss

@misc{kusupati2024matryoshka,
    title={Matryoshka Representation Learning},
    author={Aditya Kusupati and Gantavya Bhatt and Aniket Rege and Matthew Wallingford and Aditya Sinha and Vivek Ramanujan and William Howard-Snyder and Kaifeng Chen and Sham Kakade and Prateek Jain and Ali Farhadi},
    year={2024},
    eprint={2205.13147},
    archivePrefix={arXiv},
    primaryClass={cs.LG}
}

MultipleNegativesRankingLoss

@misc{henderson2017efficient,
    title={Efficient Natural Language Response Suggestion for Smart Reply},
    author={Matthew Henderson and Rami Al-Rfou and Brian Strope and Yun-hsuan Sung and Laszlo Lukacs and Ruiqi Guo and Sanjiv Kumar and Balint Miklos and Ray Kurzweil},
    year={2017},
    eprint={1705.00652},
    archivePrefix={arXiv},
    primaryClass={cs.CL}
}