UM EECS 598-010

U-M EECS 598-010

Interactive Robot Manipulators

Fall 2015

Interactive Robot Manipulators (EECS 598-010) is a graduate-level seminar course covering core topics and recent advances in robotic mobile manipulation with a view into human-robot interaction. This course is intended to prepare students for conducting research along the computational and empirical dimensions of robotics. The primary emphasis of this course is the computational aspects of robot reasoning in relation to perception and decision making. That is, assuming we have an existing robot manipulation platform, how do we program it to see things in its environment, appropriately manipulate the things its environment, act to achive a goal given by a person, and (perhaps) learn new abilities from a person? We examine these questions through coverage of textbook topics in robot modeling and control, analysis of recent and seminal research publications, and development of a novel project.


Chad Jenkins
Office: Beyster 3644

Office Hours

Monday 3-4:30pm
Tuesday 2-3:30pm (when not conflicting with travel or other necessary meetings)

Meeting time/place

Monday/Wednesday 1:30-3pm in Dow 2150

Mailing list


Any one of the following: graduate standing, or permission from the Instructor.

Highly Recommended Background

Linear alegebra and introductory robotics (e.g., EECS 467)

Recommended Background

Differential equations, Computer vision, Artificial Intelligence, and Computer graphics

Course Structure and Policies


The progression for the course will: 1) start with an introductory review of core topics in robot modeling and control leading up to an preparatory project, 2) follow with analysis of recent published papers for in-class presentation, and 3) end with final project for implementing a new idea from course topics.

Introductory Review and Project

The introductory review will cover forward kinematics, quaternions, PID control, and inverse kinematics, leading into sample-based motion planning. The preparatory project will build on this review for implementing a kinematic robot model with inverse kinematics control. This preparatory project is due on October 21 and can be implemented in any programming language such that: 1) coordinate frames of the robot can be spatially displayed and 2) the code can be compiled and run by the course staff from build instructions.

Analysis and Presentation of Published Research

Course coverage of research papers will survey recent and/or seminal publications in scene estimation, 3D point cloud processing, grasp estimation, semantic mapping, state estimation, sequential planning, affordance modeling, and learning from demonstration, as well as selected perspectives from human-robot interaction. Each student is expected to review 2 papers, following the proper paper review structure. One of these reviewed papers is expected to be presented in class, following the proper paper presentation structure. Selection of papers will occur during the course meetingperiods up through September 21, the deadline for paper selection. First come, first served.

To ensure thoroughness in discussion, each paper in class is expected to have a primary reviewer (i.e., the presenter) and a secondary reviewer. To prepare in-class presentation, both the primary presenter and secondary reviewers must meet with the course instructor one week before the scheduled presentation. At this meeting, the primary and secondary reviewers must have completed reviews of the paper for discussion, and the primary reviewer must present the current draft of their slides.

Final Project and Presentation

The coverage of research papers will lead up to a final project for reimplementing a research paper from the course with an additional approved feature. In their ideal form, a final project would extend a concept from the course in a new and interesting direction that would be worthy of future publication. Course projects must be implemented independently. Proposals for course projects will be evaluated on an ongoing basis up to November 2, 2015, the deadline for final project approval. Course projects must include an intermediate milestone (for review in class on November 30, 2015) and final deliverable (for in-class presentation and review on December 14, 2015). While not required, final projects are highly encouraged to include a descriptive web page and/or publication-formatted report (such as for the R:SS 2016 Conference perhaps).

Textbooks (optional)

There is no required textbook for this course. However, enrolled students are highly recommended to review the following books for:

Forward kinematics and inverse kinematics (Note: we will not use D-H parameters):

Robot Modeling and Control
M. Spong, S. Hutchinson, and M. Vidyasagar
Wiley, 2005

Motion planning and state estimation:

Principles of Robot Motion
H. Choset, K. Lynch, S. Hutchinson, G. Kantor, W. Burgard, L. Kavraki and S. Thrun
MIT Press, 2005

Supervisory control and human-robot interaction:

Telerobotics, automation, and human supervisory control
T. Sheridan
MIT Press, 1992.

JavaScript, as a good baseline for those with little but some programming experience:

JavaScript: The Good Parts
Douglas Crockford
O'Reilly Media / Yahoo Press, 2008


There will be 7 graded assignments in EECS 598-010 to ensure proper preparation for success in robotics research:

Preparatory project (25%)
2 Paper reviews (5% each)
In-class paper presentation (15%)
Final project proposal and approval (5%)
Final project: intermediate milestone (5%)
Final project presentation (40%)

Except for the final project presentation, EECS 598-010 projects are graded as “checked” (completed) or “not checked” (incomplete). The grade for the final project presentation will be assessed by the instructor based on the quality of the final project implementation and in-class presentation. The timing and due dates for these projects will be announced on an ongoing basis.

Late Policy

Do not submit assignments late. All assignments must be completed on time for full consideration in grading. The course staff reserves the right to not grade late submissions.

Grading will be finalized on December 14, 2015, after which no reconsideration of grading will be considered.


To receive an A grade in EECS 598-010, the assigned grades for all student coursework must sum to at least 95% of the possible (100%) scoring.

For a B grade, coursework grades must sum to at least 85% of the possible scoring.

For a C grade, coursework grades must sum to at least 75% of the possible scoring.

For a D grade, coursework grades must sum to at least 65% of the possible scoring.

A failing grade will be assessed for coursework that does not meet the standard for a D grade.

The instructor reserves the option to assign appropriate course grades with plus or minus modifiers.


Git-based repositories will be used for project implementation, version control, submission, and grading. Project implementations are submitted as branches in your assigned repository. These branches must be submitted prior to the due date for each assignment. Depending on the nature of the assignment, your implementation will be checked out and executed by the course staff during the grading assessment. You will be notified by the course staff whether your submission is sufficient to be considered checked/completed. If your assignment is insufficient for receiving a check, feedback will be provided in your repository and should be corrected immediately.

You are expected to provide a private git repository for your work in this course with the course instructor added as a read/write collaborator. If needed, the course staff can assist in the setup of an online git repository through providers such as github or bitbucket). Your repository is expected to have the following subdirectories, at a minimum, as well as the file "grading.txt" with your updated course grading status:

p_project: containing implementation and build and execution instructions for the preparatory project
reviews: containing reviews of assigned papers in plain text format
presentation: containing paper presentation slides in PDF format
proposal: containing final project proposal in plain text format
f_project: containing final project implementation and slides (in PDF format)

Please refer to the Pro Git book for an in-depth introduction to git and version control. As different people often learn through different styles, the Git Magic tutorial has also proved quite useful when a different perpective is needed. git: the simple guide has often been a great and accessible quick start resource.

We expect students to use these repositories for collaborative development as well as project submission. It is the responsibility of each student group to ensure their repository adheres to the Collaboration Policy and submission standards for each assignment. Submission standards and examples will be described for each assignment as needed.

Collaboration Policy

This policy covers all course material and assignments unless otherwise stated. Course material, concepts, and documentation may be discussed with anyone. Assignments may be discussed with the other students at the conceptual level. Discussions may make use of a whiteboard or paper. Discussions with others (or people outside of your assigned group) cannot include writing or debugging code on a computer or collaborative analysis of source code that is not your own. You may take notes away from these discussions, provided these notes do not include any source code.

The code for your implementation may not be shown to anyone outside of your group, including granting access to repositories or careless lack of protection. You do not need to hide the screen from anyone, but you should not attempt to show anyone your code. When you are done using any robot device such that another group may use it, you must remove all code you have put onto the device. You may not share your code with others outside of your group. At any time, you may show others the implemented program running on a device or simulator, but you may not discuss specific debugging details about your code while doing so.

This policy applies not only applies to collaboration during the current semester, but also any past or future instantiations of this course. Although course concepts are intended for general use, your implementation for this course must remain private after the completion of the course. It is expressly prohibited to share any code previously written and graded for this course with students currently enrolled in this course. Similarly, it is expressly prohibited for any students currently enrolled in this course to refer to any code previously written and graded for this course.

Should you fail to abide by this policy, you will receive no credit for this course. The University of Michigan reserves the right to pursue any means necessary to ensure compliance. This includes, but is not limited to prosecution through The College of Engineering’s Honor Council, which can result in your suspension or expulsion from the University of Michigan. Please refer to the Engineering Honor Council for additional information.

Course Schedule (tentative and subject to change)

Papers below are categorized into one of three "Type" labels related to how it will covered in class:

P: In-class student-led presentation of primary course topic
D: In-class group discussion with group contributions
S: Supplementary paper that will not be covered in class

Sep 9 Initialization
Introduction, Paper assignments, etc.
Instructor -- Slides (intro)
Assigned: Preparatory Project
Assigned: Paper Presentation Selections
Sep 14-16 Forward Kinematics
Instructor -- Slides (FK,quat)
Sep 21 Quaternions
Instructor --
Sep 23 Inverse Kinematics
Instructor -- Slides (IK)
Due: Paper Presentation Selections
Assigned: Project Proposal and Approval
Sep 28 PID Control
Instructor -- Slides (PID)
Sep 30 No class: IROS 2015
Oct 5 Motion Planning
P J. Kuffner, S. LaValle, RRT-Connect: An Efficient Approach to Single-Query Path Planning, ICRA, 2000. Elizabeth Open
D E. You, K. Hauser, Assisted Teleoperation Strategies for Aggressively Controlling a Robot Arm with 2D Input, R:SS, 2011. Open
Oct 7 Hands-on tutorial with Fetch robot Fetch robot docs
Oct 8 Technical Talk by TracLabs, Inc (Location: TBD)
Oct 12 Semantic Mapping and Symbol Grounding
P R. Rusu, Z. Marton, M. Beetz et al., Towards 3D point cloud based object maps for household environments, Robotics and Autonomous Systems, 56(11), 2008.
Irina Open
Oct 14 Principal Components Analysis / Moving Least Squares
D J. Shlens, A Tutorial on Principal Component Analysis, arXiv, 1404:1100, 2014.
D MathWorld entry Least Squares Fitting -- Polynomial
P M. Alexa, J. Behr, D. Cohen-Or et al., Computing and Rendering Point Set Surfaces, IEEE TVGC, 9(1), 2003.
Jiatao Open
Oct 19 No class: Fall Study Break
Oct 21 ICP Registration
D K. Arun, T. Huang, S. Blostein, Least-Squares Fitting of Two 3-D Point Sets, IEEE PAMI, 9(5), 1987.
P P. Besl, N. McKay, A Method for Registration of 3-D Shapes, IEEE PAMI, 14(2), 1992.
Jordan Open
Oct 26 Particle Filtering
P F. Dellaert, D. Fox et al., Monte Carlo Localization for Mobile Robots, ICRA, 1999.
Irina Open
D J. Deutscher, A. Blake, I. Reid, Articulated Body Motion Capture by Annealed Particle Filtering, CVPR, 2000.
D C. Choi, H. Christensen, RGB-D Object Tracking: A Particle Filter Approach on GPU, IROS, 2013.
Oct 28 Manipulation in Cluttered Scenes
P Z. Sui, O. Jenkins, K. Desingh, Axiomatic Particle Filtering for Goal-directed Robotic Manipulation, IROS, 2015.
Elizabeth Open
D R. Fikes, N. Nilsson, STRIPS: A New Approach to the Application of Theorem Proving and Problem Solving, IJCAI, 1971.
D M. Ghallab, A. Howe, C. Knoblock, et al., PDDL - The Planning Domain Definition Language, IROS, 2013.
Nov 2 Grasp Planning
P A. ten Pas, R. Platt, Localizing Handle-Like Grasp Affordances in 3D Point Clouds, ISRR, 2014. Jiatao Open
D A. Leeper, K. Hsiao, L. Takayama et al., Strategies for Human-in-the-Loop Robotic Grasping, HRI, 2012. Open
Nov 4 Physics-based Particle Filtering
P L. Zhang, S. Lyu, J. Trinkle, A Dynamic Bayesian Approach to Real-Time Estimation and Filtering in Grasp Acquisition, ICRA, 2013.
Jordan Open
P M. Vondrak. L. Sigal, O. Jenkins, Dynamical Simulation Priors for Human Motion Tracking, IEEE PAMI, 35(1), 2013.
S T. Jakobsen, Advanced Character Physics, Gamasutra Magazine, January 21, 2003.
Nov 9 Relational Perception
Discriminative versus Generative Inference
D B. Rosman, S Ramamoorthy, Learning Spatial Relations Between Objects, IJRR, 30(11), 2011.
D C. Chao, M. Cakmak, A. Thomaz, Towards Grounding Concepts for Transfer in Goal Learning from Demonstration, ICDL, 2011.
Nov 11 Pictorial Structures
D P. Felzenszwalb, D. Huttenlocher, Pictorial Structures for Object Recognition, IJCV, 61(1), 2013.
D J. Yedidia, W. Freeman, Y. Weiss, Understanding Belief Propagation and its Generalizations, Exploring Artificial Intelligence in the New Millennium, 8, 2003.
Nov 16-18 Project Sections and Hacking
Nov 23 Geometry Annotation and Registration
D T. Chen, Z. Zhu, D. Cohen-Or et al., 3-Sweep: Extracting Editable Objects from a Single Photo, ACM ToG, 32(6), 2013.
D C. Papazov, S. Haddadin, D. Burschka et al., Rigid 3D Geometry Matching for Grasping of Known Objects in Cluttered Scenes, IJRR, 31(4), 2012.
Nov 25 No class: Thanksgiving Break
Nov 30 Final Project Intermediate Milestones
Due: Intermediate Project Presentation
Dec 2-9 TBD
Dec 14 Final Project Presentations
Due: Final Project Presentation

Preparatory Project: Robot Kinematics and Control

Due 12pm, Wednesday, October 21, 2015

The purpose of the prepatory project is to ensure you have working knowledge of robot kinematics and control. For this project, you will implement methods for representing robot forward kinematics, 3D rotation by quaternions, and endeffector control by inverse kinematics. This project can be implemented in the programming environment of your choice and submitted through your repository. If you are relatively inexperienced in computer programming, I would recommend using Matlab (if you have prior experience) or JavaScript (my current prototyping language of choice). The preparatory project is a subset of Assignments 4-6 from my Introduction to Autonomous Robotics course, which has support code stencils for HTML5/JavaScript. Note: regardless of your choice of language, your implementation will require routines from linear algebra, including matrix multiplication, cross product, pseudoinverse, and matrix inversion.

The project is defined as follows: your system must read as input:

a URDF-like kinematic description of an arbitrary robot (an example in JSON is provided is below with additional JavaScript examples available)

display as output:

1) a spatial display of the robot's joint and link frame axes and
2) a spatial display of the robot's endeffector (specified separately as a point in given link frame)

and provide interactive controls to:

1) rotate each joint individually about its motor axis
2) invoke an inverse kinematics controller to reach the robot's endeffector to a target location in world coordinates

// Example URDF-like robot description in JSON format
robot = {

Paper Presentation and Review

Proper Paper Review Structure

In preparation.

Proper Paper Presentation Structure

In preparation.

Final Project Information

Final Project Proposal Structure

Proposal Approval Due 4:30pm, Monday, November 2, 2015

The final project proposal is a 1-2 page plain text file (2 pages max, not including appendices). Project proposals must be submitted through your repository and will be evaluated during office hours or as needed. This proposal should clearly address the following:

Problem: identify the problem to solve, why is it important to solve, state your metric for success
Approach: what parts of the problem will you solve, how will realize your solution, what parts of your solution are new
Evaluation: how will you measure the success of your solution to the problem
Intermediate Milestone: what will you show as your intermediate milestone to ensure progress is being made

Final Project Intermediate Milestone

Due 1:30pm, Monday, November 30, 2015

The Intermediate Milestone will be a 5 minute in-class presentation on the progress of your final project with time for interactive questions and answers.

Final Project Intermediate Milestone

Due 1:30pm, Monday, December 14, 2015

The Final Project Presentation will be a 15 minute in-class presentation on the results of your final project with time for interactive questions and answers.