Lerrel Pinto

I am an Assistant Professor of Computer Science at NYU Courant and part of the CILVR group. Before that, I was at UC Berkeley for a postdoc, at CMU Robotics Institute for a PhD, and at IIT Guwahati for undergrad.

Research: I run the General-purpose Robotics and AI Lab (GRAIL) with the goal of getting robots to generalize and adapt in the messy world we live in. Our research focuses broadly on robot learning and decision making, with an emphasis on large-scale learning (both data and models), representation learning for sensory data, developing algorithms to model actions and behavior, reinforcement learning for adapting to new scenarios, and building open-sourced affordable robots. A talk on our recent robotics efforts is here. If you are interested in joining our lab, please read this.

Recent News

Recent Talks

Here are some public talks that covers my recent research:

Courses Taught at NYU

Selected Research and Publications


OK-Robot: What Really Matters in Integrating Open-Knowledge Models for Robotics
OK-Robot: What Really Matters in Integrating Open-Knowledge Models for Robotics

VQ-BeT: Behavior Generation with Latent Actions
VQ-BeT: Behavior Generation with Latent Actions

Hierarchical State Space Models for Continuous Sequence-to-Sequence Modeling
Hierarchical State Space Models for Continuous Sequence-to-Sequence Modeling

diff History for Neural Language Agents
diff History for Neural Language Agents

On Bringing Robots Home
On Bringing Robots Home

See to Touch: Learning Tactile Dexterity through Visual Incentives
See to Touch: Learning Tactile Dexterity through Visual Incentives

NetHack is Hard to Hack
NetHack is Hard to Hack



Teach a Robot to FISH: Versatile Imitation from One Minute of Demonstrations
Teach a Robot to FISH: Versatile Imitation from One Minute of Demonstrations

CLIP-Fields: Weakly Supervised Semantic Fields for Robotic Memory
CLIP-Fields: Weakly Supervised Semantic Fields for Robotic Memory

From play to policy: Conditional behavior generation from uncurated robot data
From play to policy: Conditional behavior generation from uncurated robot data

Holo-Dex: Teaching Dexterity with Immersive Mixed Reality
Holo-Dex: Teaching Dexterity with Immersive Mixed Reality

Behavior Transformers: Cloning k modes with one stone
Behavior Transformers: Cloning k modes with one stone

Watch and Match: Supercharging Imitation with Regularized Optimal Transport
Watch and Match: Supercharging Imitation with Regularized Optimal Transport

Dexterous Imitation Made Easy: A Learning-Based Framework for Efficient Dexterous Manipulation
Dexterous Imitation Made Easy: A Learning-Based Framework for Efficient Dexterous Manipulation

Don't Change the Algorithm, Change the Data: Exploratory Data for Offline Reinforcement Learning
Don't Change the Algorithm, Change the Data: Exploratory Data for Offline Reinforcement Learning

The Surprising Effectiveness of Representation Learning for Visual Imitation
The Surprising Effectiveness of Representation Learning for Visual Imitation


Mastering Visual Continuous Control: Improved Data-Augmented Reinforcement Learning
Mastering Visual Continuous Control: Improved Data-Augmented Reinforcement Learning

Reinforcement Learning with Prototypical Representations
Reinforcement Learning with Prototypical Representations

Learning Cross-Domain Correspondence for Control with Dynamics Cycle-Consistency
Learning Cross-Domain Correspondence for Control with Dynamics Cycle-Consistency

Self-Supervised Policy Adaptation during Deployment
Self-Supervised Policy Adaptation during Deployment

Task-Agnostic Morphology Evolution
Task-Agnostic Morphology Evolution

Visual Imitation Made Easy
Visual Imitation Made Easy

Learning Predictive Representations for Deformable Objects Using Contrastive Estimation
Learning Predictive Representations for Deformable Objects Using Contrastive Estimation

Robust Policies via Mid-Level Visual Representations
Robust Policies via Mid-Level Visual Representations

Automatic Curriculum Learning through Value Disagreement
Automatic Curriculum Learning through Value Disagreement

Generalized Hindsight for Reinforcement Learning
Generalized Hindsight for Reinforcement Learning

Reinforcement Learning with Augmented Data
Reinforcement Learning with Augmented Data

Learning to Manipulate Deformable Objects without Demonstrations
Learning to Manipulate Deformable Objects without Demonstrations

Swoosh! Rattle! Thump! - Actions that Sound
Swoosh! Rattle! Thump! - Actions that Sound

Hierarchically Decoupled Imitation for Morphological Transfer
Hierarchically Decoupled Imitation for Morphological Transfer

PyRobot: An Open-source Robotics Framework for Research and Benchmarking
PyRobot: An Open-source Robotics Framework for Research and Benchmarking

Multiple Interactions Made Easy (MIME): Large Scale Demonstrations Data for Imitation
Multiple Interactions Made Easy (MIME): Large Scale Demonstrations Data for Imitation

Robot Learning in Homes: Improving Generalization and Reducing Dataset Bias
Robot Learning in Homes: Improving Generalization and Reducing Dataset Bias

Asymmetric Actor Critic for Image-Based Robot Learning
Asymmetric Actor Critic for Image-Based Robot Learning

CASSL: Curriculum Accelerated Self-Supervised Learning
CASSL: Curriculum Accelerated Self-Supervised Learning

Learning to Fly by Crashing
Learning to Fly by Crashing

Robust Adversarial Reinforcement Learning
Robust Adversarial Reinforcement Learning


Supervision via Competition: Robot Adversaries for Learning Tasks
Supervision via Competition: Robot Adversaries for Learning Tasks

The Curious Robot: Learning Visual Representations via Physical Interactions
The Curious Robot: Learning Visual Representations via Physical Interactions

Supersizing Self-supervision: Learning to Grasp from 50K Tries and 700 Robot Hours
Supersizing Self-supervision: Learning to Grasp from 50K Tries and 700 Robot Hours