Franka Bin Picking

Zero-shot vision that understands every feature of an object.

An efficient, robust perception pipeline for object pose and grasp recognition, taken all the way to pick execution on a Franka arm.

6d-pose
instance-seg
grasp-gen
closed-loop

Franka arm in Isaac Sim reaching into a pink bin of couplers

README.md

Perception to action, end to end.

01 Foundation Stereo→02 CNOS segmentation→03 FoundationPose 6-DoF→04 GraspNet + filtering→05 RMPflow pick on Franka

## problem statement

Industrial bin picking is difficult because objects are often randomly oriented, partially occluded, and densely packed, which makes traditional rule-based perception and fixed grasp strategies unreliable.

The goal was to build an autonomous manipulation pipeline that understands cluttered bin scenes, estimates object geometry and pose, generates feasible grasps, and executes picks with minimal task-specific training.

NVIDIA Isaac SimROS 2Foundation StereoCNOSFoundationPoseGraspNet

## implementation notes

Used Foundation Stereo to reconstruct dense depth and scene geometry from stereo RGB images.
Applied CNOS for object-level instance segmentation in cluttered scenes.
Used FoundationPose to estimate the 6-DoF pose of detected objects.
Integrated GraspNet to generate candidate 6-DoF grasp poses from the reconstructed scene.
Transformed perception and grasp outputs into the robot coordinate frame for execution.
Filtered grasp candidates on reachability, approach direction, and collision feasibility.
Integrated the complete perception-to-action pipeline with a Franka manipulator.
Validated the system in simulation using NVIDIA Isaac Sim.

01 / Stereo depth estimation

Foundation Stereo, fine-tuned for better sim-to-model alignment.

Foundation Stereo architecture with the attentive hybrid cost filtering module

Side-tuning adapter design choices with disparity results with and without the adapter — **reference** · Foundation Stereo architecture and side-tuning adapter (STA) design

## fine-tuning

Fine-tuned SOTA models for better sim-model alignment.

Fine-tuned side-tuning adapter weights
Optimized feature-fusion layers
Refined disparity prediction layers
Adapted normalization statistics and parameters

model: Foundation Stereo

Left stereo image of couplers in a pink bin — **left** image

Right stereo image of couplers in a pink bin — **right** image

Reconstructed point cloud of the bin from stereo depth — **output** · stereo depth estimation

Point cloud of the simulated bin of couplers — **sim** · point cloud of the Isaac Sim bin

02 / Zero-shot segmentation

CNOS matches what the camera sees against renders of the CAD model, with no task-specific training.

Onboarding (preparation). Start with 3D CAD models of the objects (here, a coupler). Render each model from many viewpoints with pyrender or BlenderProc. Every render is turned into a descriptor, a numerical fingerprint, with a pretrained feature extractor like DINOv2. These fingerprints form the reference database: a library of how the object looks from every side.
Proposal (looking at the real scene). Take an RGB photo of the scene. A proposal extractor such as SAM or FastSAM finds regions that might be objects. Each region is cropped and passed through DINOv2 again to produce proposal descriptors, fingerprints of the candidate objects.
Matching (comparison and decision). Compare each proposal descriptor with each reference descriptor using cosine similarity, which gives a score per template. Scores are aggregated per object across all views of the same CAD model, and the system picks the best-matching object ID with a confidence score.

CNOS pipeline: onboarding with CAD templates, proposals from the RGB frame, and descriptor matching — **reference** · CNOS onboarding, proposal and matching stages

## why synthetic data?

Training with synthetic images: a teacher trained on synthetic data pseudo-labels real images for a student model

Depth predictions from models trained on real versus synthetic images — **reference** · Depth Anything V2

Expands training diversity with varied shapes, lighting, and object configurations.
Reduces dependence on expensive real-world data collection and manual annotation.
Enables controlled generation of rare, difficult, and edge-case scenarios.
Improves zero-shot generalization to unseen objects and real-world environments.

pyrender · less realism

blenderproc · more realism

BlenderProc render of the coupler, angle 1

BlenderProc render of the coupler, angle 2

BlenderProc render of the coupler, angle 3

BlenderProc render of the coupler, angle 4

03 / 6D pose estimation

FoundationPose estimates where the selected object is and how it is oriented in 3D.

FoundationPose overview: model-based and model-free setups, pose estimation and tracking, with a benchmark comparison chart — The model takes either reference images or a textured CAD model and builds the object representation used for pose estimation. It estimates the initial pose, then tracks it frame to frame instead of solving from scratch every time.

Segmentation first. From the RGB frame, generate an instance segmentation mask for each visible adapter.
Top-object selection. Analyze the masks to find which adapter is physically on top, the most accessible for pickup.
Mask-guided pose estimation. Pass the selected mask with the RGB/depth frame to the 6D pose estimation model.
6-DoF output. The model returns the adapter pose as 3D position plus 3D orientation, ready for grasping.

Pose estimate drawn as a 3D bounding box on the top coupler in the bin — **pose** · 6-DoF estimate on the top-most coupler

selection · the next coupler to pick, one at a time

perception · segment → pose → grasp

04 / Grasp estimation & control

GraspNet proposes grasps; filtering and RMPflow turn them into safe motion.

Grasps come from GraspNet-1Billion, a large-scale benchmark for general object grasping that also covers related areas like 6D pose estimation and unseen-object segmentation.

On top of it, I implemented a multi-stage custom filtering process to pick the best achievable grasp position and orientation for the arm.

Point cloud of the bin with candidate grasp poses drawn in blue — **graspnet** · candidate grasps on the reconstructed scene

Franka arm in Isaac Sim lifting a coupler from the bin — **execution** · Franka pick in Isaac Sim

RMPflow generates IK and motion planning for the target position and orientation given by GraspNet.

A safe pre-approach and post-grasp pose is computed from the target pose for safe retrieval of objects.