Spaces:
Paused
Paused
updated documentation and pyprojects
Browse files- README.md +25 -8
- pyproject.toml +1 -0
README.md
CHANGED
|
@@ -28,6 +28,7 @@ MedRAX is built on a robust technical foundation:
|
|
| 28 |
- **Disease Classification**: Leverages DenseNet-121 from TorchXRayVision for detecting 18 pathology classes
|
| 29 |
- **X-ray Generation**: Utilizes RoentGen for synthetic CXR generation
|
| 30 |
- **Web Browser**: Provides web search capabilities and URL content retrieval using Google Custom Search API
|
|
|
|
| 31 |
- **Python Sandbox**: Executes Python code in a secure, stateful sandbox environment using `langchain-sandbox` and Pyodide. Supports custom data analysis, calculations, and dynamic package installations. Pre-configured with medical analysis packages including pandas, numpy, pydicom, SimpleITK, scikit-image, Pillow, scikit-learn, matplotlib, seaborn, and openpyxl. **Requires Deno runtime.**
|
| 32 |
- **Utilities**: Includes DICOM processing, visualization tools, and custom plotting capabilities
|
| 33 |
<br><br>
|
|
@@ -159,6 +160,7 @@ selected_tools = [
|
|
| 159 |
"ChestXRaySegmentationTool",
|
| 160 |
"PythonSandboxTool", # Python code execution
|
| 161 |
"WebBrowserTool", # Web search and URL access
|
|
|
|
| 162 |
# Add or remove tools as needed
|
| 163 |
]
|
| 164 |
|
|
@@ -174,17 +176,10 @@ agent, tools_dict = initialize_agent(
|
|
| 174 |
|
| 175 |
The following tools will automatically download their model weights when initialized:
|
| 176 |
|
| 177 |
-
### Classification
|
| 178 |
```python
|
| 179 |
# TorchXRayVision-based classifier (original)
|
| 180 |
TorchXRayVisionClassifierTool(device=device)
|
| 181 |
-
|
| 182 |
-
# ArcPlus SwinTransformer-based classifier (new)
|
| 183 |
-
ArcPlusClassifierTool(
|
| 184 |
-
model_path="/path/to/Ark6_swinLarge768_ep50.pth.tar", # Optional
|
| 185 |
-
num_classes=18, # Default
|
| 186 |
-
device=device
|
| 187 |
-
)
|
| 188 |
```
|
| 189 |
|
| 190 |
### Segmentation Tool
|
|
@@ -263,6 +258,7 @@ No additional model weights required:
|
|
| 263 |
ImageVisualizerTool()
|
| 264 |
DicomProcessorTool(temp_dir=temp_dir)
|
| 265 |
WebBrowserTool() # Requires Google Search API credentials
|
|
|
|
| 266 |
```
|
| 267 |
<br>
|
| 268 |
|
|
@@ -281,6 +277,25 @@ ChestXRayGeneratorTool(
|
|
| 281 |
2. Place weights in `{model_dir}/roentgen`
|
| 282 |
3. Optional tool, can be excluded if not needed
|
| 283 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 284 |
### Knowledge Base Setup (MedicalRAGTool)
|
| 285 |
|
| 286 |
The `MedicalRAGTool` uses a Pinecone vector database to store and retrieve medical knowledge. To use this tool, you need to set up a Pinecone account and a Cohere account.
|
|
@@ -383,6 +398,8 @@ If you are running a local LLM using frameworks like [Ollama](https://ollama.com
|
|
| 383 |
|
| 384 |
**WebBrowserTool**: Requires Google Custom Search API credentials, which can be set in the `.env` file.
|
| 385 |
|
|
|
|
|
|
|
| 386 |
**PythonSandboxTool**: Requires Deno runtime installation:
|
| 387 |
```bash
|
| 388 |
# Verify Deno is installed
|
|
|
|
| 28 |
- **Disease Classification**: Leverages DenseNet-121 from TorchXRayVision for detecting 18 pathology classes
|
| 29 |
- **X-ray Generation**: Utilizes RoentGen for synthetic CXR generation
|
| 30 |
- **Web Browser**: Provides web search capabilities and URL content retrieval using Google Custom Search API
|
| 31 |
+
- **DuckDuckGo Search**: Offers privacy-focused web search capabilities using DuckDuckGo search engine for medical research, fact-checking, and accessing current medical information without API keys
|
| 32 |
- **Python Sandbox**: Executes Python code in a secure, stateful sandbox environment using `langchain-sandbox` and Pyodide. Supports custom data analysis, calculations, and dynamic package installations. Pre-configured with medical analysis packages including pandas, numpy, pydicom, SimpleITK, scikit-image, Pillow, scikit-learn, matplotlib, seaborn, and openpyxl. **Requires Deno runtime.**
|
| 33 |
- **Utilities**: Includes DICOM processing, visualization tools, and custom plotting capabilities
|
| 34 |
<br><br>
|
|
|
|
| 160 |
"ChestXRaySegmentationTool",
|
| 161 |
"PythonSandboxTool", # Python code execution
|
| 162 |
"WebBrowserTool", # Web search and URL access
|
| 163 |
+
"DuckDuckGoSearchTool", # Privacy-focused web search
|
| 164 |
# Add or remove tools as needed
|
| 165 |
]
|
| 166 |
|
|
|
|
| 176 |
|
| 177 |
The following tools will automatically download their model weights when initialized:
|
| 178 |
|
| 179 |
+
### Classification Tool
|
| 180 |
```python
|
| 181 |
# TorchXRayVision-based classifier (original)
|
| 182 |
TorchXRayVisionClassifierTool(device=device)
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 183 |
```
|
| 184 |
|
| 185 |
### Segmentation Tool
|
|
|
|
| 258 |
ImageVisualizerTool()
|
| 259 |
DicomProcessorTool(temp_dir=temp_dir)
|
| 260 |
WebBrowserTool() # Requires Google Search API credentials
|
| 261 |
+
DuckDuckGoSearchTool() # No API key required, privacy-focused search
|
| 262 |
```
|
| 263 |
<br>
|
| 264 |
|
|
|
|
| 277 |
2. Place weights in `{model_dir}/roentgen`
|
| 278 |
3. Optional tool, can be excluded if not needed
|
| 279 |
|
| 280 |
+
### ArcPlus SwinTransformer-based Classifier
|
| 281 |
+
```python
|
| 282 |
+
ArcPlusClassifierTool(
|
| 283 |
+
model_path="/path/to/Ark6_swinLarge768_ep50.pth.tar", # Optional
|
| 284 |
+
num_classes=18, # Default
|
| 285 |
+
device=device
|
| 286 |
+
)
|
| 287 |
+
```
|
| 288 |
+
|
| 289 |
+
The ArcPlus classifier requires manual setup as the pre-trained model is not publicly available for automatic download:
|
| 290 |
+
|
| 291 |
+
1. **Request Access**: Visit [https://github.com/jlianglab/Ark](https://github.com/jlianglab/Ark) and request the pretrained model through their Google Forms
|
| 292 |
+
2. **Download Model**: Once approved, download the `Ark6_swinLarge768_ep50.pth.tar` file
|
| 293 |
+
3. **Place in Directory**: Drag the downloaded file into your `model-weights` directory
|
| 294 |
+
4. **Initialize Tool**: The tool will automatically look for the model file in the specified `cache_dir`
|
| 295 |
+
|
| 296 |
+
The ArcPlus model provides advanced chest X-ray classification across 6 medical datasets (MIMIC, CheXpert, NIH, RSNA, VinDr, Shenzhen) with 52+ pathology categories.
|
| 297 |
+
```
|
| 298 |
+
|
| 299 |
### Knowledge Base Setup (MedicalRAGTool)
|
| 300 |
|
| 301 |
The `MedicalRAGTool` uses a Pinecone vector database to store and retrieve medical knowledge. To use this tool, you need to set up a Pinecone account and a Cohere account.
|
|
|
|
| 398 |
|
| 399 |
**WebBrowserTool**: Requires Google Custom Search API credentials, which can be set in the `.env` file.
|
| 400 |
|
| 401 |
+
**DuckDuckGoSearchTool**: No API key required. Uses DuckDuckGo's privacy-focused search engine for medical research and fact-checking.
|
| 402 |
+
|
| 403 |
**PythonSandboxTool**: Requires Deno runtime installation:
|
| 404 |
```bash
|
| 405 |
# Verify Deno is installed
|
pyproject.toml
CHANGED
|
@@ -75,6 +75,7 @@ dependencies = [
|
|
| 75 |
"seaborn>=0.12.0",
|
| 76 |
"huggingface_hub>=0.17.0",
|
| 77 |
"iopath>=0.1.10",
|
|
|
|
| 78 |
]
|
| 79 |
|
| 80 |
[project.optional-dependencies]
|
|
|
|
| 75 |
"seaborn>=0.12.0",
|
| 76 |
"huggingface_hub>=0.17.0",
|
| 77 |
"iopath>=0.1.10",
|
| 78 |
+
"duckduckgo-search>=4.0.0",
|
| 79 |
]
|
| 80 |
|
| 81 |
[project.optional-dependencies]
|