Zhangyu Wang
Google Scholar / GitHub
Email: zhangyuw@usc.edu
I am a student in the B.S./M.S. (Progressive Degree Program) in Computer Science at the University of Southern California. I am fortunate to be working as a research intern at the Stanford Vision and Learning Lab, advised by Prof. Jiajun Wu, and at USC LiraLab, advised by Prof. Erdem Bıyık. Before that, I had the pleasure of working with Prof. Gaurav S. Sukhatme at the USC Robotic Embedded Systems Laboratory.
My research interest focuses on robot learning, specifically on leveraging existing knowledge in pretrained Foundation Models and VLMs for generalizable manipulation and scalable learning or data collection methods.
Outside of research, I enjoy playing badminton.
Research
My work centers on robot learning: helping robots acquire precise, generalizable manipulation skills by combining scalable data collection with knowledge from pretrained foundation models. To this end, my research includes:
- Scalable data collection for dexterous manipulation, through teleoperation and human-in-the-loop pipelines;
- Foundation models for manipulation, using VLMs for precise, self-correcting agentic control;
- Real-to-sim reconstruction and physical property estimation for model predictive control.
News
- [Nov 2026] Our papers Cut-GS and Dosesteer have been submitted to CVPR 2027.
- [Sep 2026] Our work DITTO-X is submitted to ICRA 2027.
- [July 2026] I will be working as a research intern at Stanford Vision and Learning Lab, advised by Prof. Jiajun Wu.
- [May 2026] Our paper ATLAS is accepted to SCAI @ SIGIR 2026.
Show more...Show less...
- [Sep 2025] I will be working as a research intern at USC Robotic Embedded Systems Laboratory, advised by Prof. Gaurav Sukhatme.
- [May 2023] Our paper Offset-Guided Attention Network for Room-Level Aware Floor Plan Segmentation is accepted to IEEE Access.
Publications (* indicates equal contribution; † indicates equal advising)
Dosesteer: Steering diffusion-based dose prediction models towards clinical targets via Dual-Response Guidance
DITTO-X: Forward and Reverse Teleoperation for Dexterous Manipulation and Human Intervention
An actuated exoskeleton supports forward and reverse teleoperation for dexterous manipulation, matching the operator's hand to the robot during policy execution and enabling human intervention.
ATLAS: Agentic Text Landscape Exploration for Embedding-Free Long Context Memory
An agentic retrieval framework that explores long documents through evidence-driven text search, without vector embeddings or explicit graph construction.
Offset-Guided Attention Network for Room-Level Aware Floor Plan Segmentation
Offset-guided attention and feature fusion improve semantic consistency within rooms and align room, wall, and door predictions in floor-plan segmentation.
Projects (* indicates equal contribution)
Self-Supervised Precise Agentic Manipulation Ongoing
An agentic, VLM-based pipeline for precise manipulation that learns from real interaction observations and actively corrects its actions when keypoint or target estimates are inaccurate. It reaches ~80% success on the ManiSkill PlugCharger task, compared with ~60% for the baseline.
Probe-Guided Optimization for Object Hallucination Mitigation in Large Vision-Language Models
Linear probes trained on synthetic CLEVR-MATE data detect cross-modal entity binding errors. Probe-guided hidden-state interventions improve VLM answers without retraining the base models.
Zero-Shot 3D Model Interior Generation
Feedforward generation of 3D food models with interiors, so that cut-open views render realistic cross-sections for foods with varied exterior shapes. Distilling score distillation sampling (SDS) knowledge into a feedforward model generates an interior in under one second, instead of slow per-object optimization.
Real2Sim-Guided Model Predictive Control for Few-Shot Tabletop Manipulation
An MPC pipeline with a custom reward model for few-shot tabletop manipulation. Its Real2Sim stage combines VLMs, Grounded SAM, TRELLIS, and FoundationPose to reconstruct common tabletop tasks (Push-T, cube picking, peg insertion) and update object poses in real time; zeroth-order optimization recovers mass and friction from video with 70%+ accuracy.
Post-Training Optimization for High-Precision Isodose Line Prediction
Dose predictions from state-of-the-art models are not yet accurate enough at isodose lines for clinical use. Post-training methods for head-and-neck dose prediction improve HD95 on the V50%, V12Gy, and V80% isodose lines over the baseline.
LLM-Guided Machine-Learning-Ready Metabolic Knowledge Graph Construction
Combines LLMs with expert knowledge to build a machine-learning-friendly metabolism dataset linking more than 20,000 papers to their KEGG pathways and GEO results. Expert verification of data quality is underway; paper in progress.
Training CLIP with Small Batch Size: A Method to Leverage Memory-Heavy Structural Data
Brings memory-heavy 3D protein structures into CLIP-style contrastive learning, where they force small batch sizes. Residual Loss Incorporation and Random Row-wise CLIP (with sigmoid and combined objectives), built on a trainable GearNet structure encoder, lower the minimum validation loss from 4.8 to 3.8–4.3 compared with a pre-embedded ProteinCLIP baseline; downstream protein-protein interaction performance remains an open problem.
On Path to Multimodal Generalist: General-Level and General-Bench
Non-author contributor, credited in the paper's contributor without authorship table. Involved in the evaluation of VLMs on various visual and language tasks.
EEG/EMG Signal-Based Control
Uses the correlation between EEG and EMG signals during actions to extract subject-agnostic features for EEG/EMG-guided manipulation, applying a learnable wavelet transform to EMG and exploring filters for noisy EEG signals.
Education & Experience
Education
- Master of Science in Computer Science, University of Southern California (2026 – 2027 [expected])
- Bachelor of Science in Computer Science, University of Southern California (2023 – 2027 [expected])
Experience
- Research Intern, Stanford Vision and Learning Lab (2026 – current)
- Research Intern, USC LiraLab (2026 – current)
- Research Intern, USC Robotic Embedded Systems Laboratory (2025 – 2026)
- Research Intern, Zheng Research Group, Osaka University (2025 – 2026)
- AI Group Intern, Shenzhen Zhangjiang Robot, Shenzhen, China (2025)
Honors & Awards
- USC Dean's Merit Scholarship (2023 – current)
- USC Undergraduate Research Associates Grant (Spring 2026)
Skills
- Languages: Python, Java, C++
- Libraries & tools: PyTorch, NumPy, pandas, React, RapidAPI
- Robotics & vision: Isaac Sim, ManiSkill, SAM / Grounded SAM, FoundationPose, diffusion policy