Analysis
Addendum: We followed up the initial experimentation with an additional run of ode trials using a tuned set of parameters. The analysis has been updated to include the new data.
Plots of control performance for the Industrial Arm Grasping and Multiple Block Grasping performance are depicted in Figures 1 and 2.
Figure 1 - Metric performance for the industrial arm grasping experimental control over three simulators (ODE, BULLET, and DART). BULLET becomes unstable immediately. ODE with default parameters is able to maintain a stable grasp for approximately two seconds of virtual time and ODE with tuned parameters is able to maintain a stable grasp for approximately 14 seconds. DART is able to maintain a stable grasp for nearly 35 seconds.
Figure 2 - Metric performance for the multi-block grasping experimental control (using a single block) over three simulators (ODE, BULLET, and DART). BULLET becomes unstable immediately. DART exhibits a concerning discontinuity in performance, but generally is able to maintain a grasp for the length of the experiment. ODE maintains a stable grasp for both the default parameters and the tuned parameters.
Figure 3 - Metric performance for the multi-block grasping experiment (using a single block) and very high friction value over three simulators (ODE, BULLET, and DART). Performance appears identical to the high friction control.
We analyzed the resulting data toward answering the following questions:
Do very high friction coefficients yield less stable simulations? These experiments (and Figure 3) indicate that high friction coefficients (108) do not affect the stability of the simulations. This result is likely due to simulations with direct solvers using particular formulations of the contact model on cases of infinite friction; ODE's user manual seems to indicate as such.
How important is varying inertia to performing the task? Using unit mass and inertia yields significantly better performance than the control (realistic inertias). Any integer scaling factor reduces the performance significantly worse than the control. These results indicate that simulation users must take care in setting inertial values.
Does using tessellated geometries impact performance? Both DART and BULLET "segfault" with tessellated geometry. ODE drops the grasped object nearly immediately. Representing geometries using primitive types is currently the best solution, from metric performance, software support, and software robustness standpoints.
Did a particular simulation library outperform others across the board? DART tends to outperform ODE with default values according to our metric, which may be due to the relatively small number of splitting method iterations (50) that is the default for GAZEBO. When the number of splitting method iterations is raised (500), ODE exhibits better performance in the industrial arm grasp experiments but is still outperformed by DART; however, in the multiple block grasp experiments, ODE maintains a stable grasp through a total of 11 grasping objects while DART fails to maintain the grasp through a full simulation run once the number grasping objects reaches 9. ODE is able to simulate all scenarios, even if does not always yield the highest performance; the library crashes less frequently than the other libraries.
======================