Post
3072
VQASynth adds a code-as-action annotation agent
Data synthesis frameworks may distill a pipeline of expert models through templates or LLM re-phrasing, but templates limit the variety of the training data.
Wanting more flexibility for VQASynth, we've added an experimental SpatialAnnotator agent with scene understanding tools-as-methods, built on NVIDIA-labs OO Agents (NOOA): https://github.com/NVIDIA-NeMo/labs-OO-Agents
In NOOA an agent is a Python class. Fields hold state, docstrings are prompts, type annotations are contracts, and a method body of
GitHub: https://github.com/remyxai/VQASynth#agent-with-vqasynth-tools
Data synthesis frameworks may distill a pipeline of expert models through templates or LLM re-phrasing, but templates limit the variety of the training data.
Wanting more flexibility for VQASynth, we've added an experimental SpatialAnnotator agent with scene understanding tools-as-methods, built on NVIDIA-labs OO Agents (NOOA): https://github.com/NVIDIA-NeMo/labs-OO-Agents
In NOOA an agent is a Python class. Fields hold state, docstrings are prompts, type annotations are contracts, and a method body of
... becomes an LLM-driven loop. Scene understanding tools stay ordinary deterministic methods.from nooa import Agent
class SpatialAnnotator(Agent, llm=llm):
"""Answer spatial questions about an image using scene understanding tools."""
scene: SceneContext # masks, depth, intrinsics, canonical transform
# Deterministic Python. Bodies omitted here.
def measure(self, a, b=None, kind="object_to_object") -> Measurement
def count(self, target: str) -> Count
# Agentic. The runtime hands this to the model.
async def annotate(self, image: Image, question: str) -> SpatialAnswer:
"""Answer the spatial question. This is offline labeling, so there is
no human available for clarification. Prefer a canonical
interpretation, and decline if the question cannot be grounded."""
...GitHub: https://github.com/remyxai/VQASynth#agent-with-vqasynth-tools