Spaces:

---

pollen-robotics

/

Reachy_Mini

like 124

Running

Hi Pollen Robotics community!

I'm working on a vision-based object tracking project with Reachy Mini Wireless and need help with head movement.

WHAT WORKS PERFECTLY:

• Object detection via OpenCV
• Camera access (mini.media.get_frame())
• Precise pixel position tracking
• Blue object detected reliably at position (cx, cy)

THE PROBLEM:
When I call mini.look_at_image(cx, cy), the head moves in the wrong direction:

• Object detected at right side of image → head rotates LEFT
• Expected: head turns toward object
• Actual: head rotates around left axis, often away from target

APPROACHES TESTED (all failed):

1. Direct approach:
   mini.look_at_image(cx, cy, duration=0.3)
   → Wrong direction

2. Two-step with perform_movement=False:
   target_pose = mini.look_at_image(cx, cy, perform_movement=False)
   mini.set_target_head_pose(target_pose)
   → Still wrong direction

3. Small incremental movements:
   Actions: ±5 pixels, duration=1.0s
   → Same issue

4. Various durations tested:
   0.3s, 1.0s, 2.0s → No improvement

OBSERVATIONS:

• Head consistently rotates around wrong axis
• Movement appears to be inverting direction
• Sometimes moves away from target instead of toward it
• Happens with both direct look_at_image() and via pose matrix

MY QUESTIONS:

1. Is look_at_image() the correct method for tracking moving objects?
2. Could there be a calibration issue with pixel→head coordinate transform?
3. Is there a known issue with coordinate systems in current SDK version?
4. Do I need to apply any transformation to pixel coordinates first?
5. Any example code for smooth object tracking with head movement?

TECHNICAL DETAILS:

• Hardware: Reachy Mini Wireless ($449 version)
• RPi: Raspberry Pi 5
• SDK: reachy-mini from pip (latest)
• Camera resolution: 1280x720
• Image center: (640, 360)
• Typical object position: (700-900, 200-300) when detected right of center

CODE EXAMPLE:

from reachy_mini import ReachyMini
import cv2
import numpy as np

mini = ReachyMini()
frame = mini.media.get_frame()

# Detect blue object (works perfectly)
hsv = cv2.cvtColor(frame, cv2.COLOR_BGR2HSV)
mask = cv2.inRange(hsv, np.array([105, 150, 100]), 
                         np.array([120, 255, 255]))
contours, _ = cv2.findContours(mask, cv2.RETR_EXTERNAL, 
                                cv2.CHAIN_APPROX_SIMPLE)

if contours:
    M = cv2.moments(max(contours, key=cv2.contourArea))
    cx = int(M["m10"] / M["m00"])
    cy = int(M["m01"] / M["m00"])
    
    # Object detected at (cx, cy)
    # Want head to look at this position
    mini.look_at_image(cx, cy, duration=0.5)  # ← Moves wrong direction

GOAL:
Build a smooth head tracking system where Reachy follows a moving object by keeping it centered in the camera view.

Has anyone successfully implemented object tracking with head movement? Any guidance would be greatly appreciated!

Thanks in advance for your help!