Gazebo missing crash logs

[josiahls]

Note: After some fiddling with the run commands, I got everything to work. It was due to no display available (obviously) However there is no error / log output that indicates this.

Error 1: Gazebo / gear crashes

... logging to /home/ariac/.ros/log/e5501c2e-3fc2-11ea-b2fb-0242ac120016/roslaunch-06fab0b7b083-322.log
Checking log directory for disk usage. This may take awhile.
Press Ctrl-C to interrupt
Done checking log file disk usage. Usage is <1GB.

started roslaunch server http://172.18.0.22:44625/

SUMMARY
========

PARAMETERS
 * /rosdistro: melodic
 * /rosversion: 1.14.3

NODES
  /
    ariac_sim (osrf_gear/gear.py)

auto-starting new master
process[master]: started with pid [332]
ROS_MASTER_URI=http://172.18.0.22:11311

setting /run_id to e5501c2e-3fc2-11ea-b2fb-0242ac120016
process[rosout-1]: started with pid [343]
started core service [/rosout]
writing file /tmp/ariac/arm1.urdf.xacro
writing file /tmp/ariac/arm2.urdf.xacro[350]
writing file /tmp/ariac/gear.world
writing file /tmp/ariac/gear.launch
writing file /tmp/ariac/gear.urdf.xacro
Running command: roslaunch /tmp/ariac/gear.launch world_path:=/tmp/ariac/gear.world gear_urdf_xacro:=/tmp/ariac/gear.urdf.xacro arm_urdf_dir:=/tmp/ariac/
... logging to /home/ariac/.ros/log/e5501c2e-3fc2-11ea-b2fb-0242ac120016/roslaunch-06fab0b7b083-353.log
Checking log directory for disk usage. This may take awhile.
Press Ctrl-C to interrupt
Done checking log file disk usage. Usage is <1GB.

xacro: in-order processing became default in ROS Melodic. You can drop the option.
xacro: in-order processing became default in ROS Melodic. You can drop the option.
xacro: in-order processing became default in ROS Melodic. You can drop the option.
started roslaunch server http://172.18.0.22:43853/

SUMMARY
========

PARAMETERS
 * /ariac/arm1/arm/action_monitor_rate: 10
[Model data loads successfully] 
 * /sensors_tf_publisher/tf_prefix: 
 * /use_sim_time: True

NODES
  /ariac/arm2/
    arm_controller_spawner (controller_manager/controller_manager)
    fake_joint_calibration (rostopic/rostopic)
    joint_state_controller_spawner (controller_manager/controller_manager)
    robot_state_publisher (robot_state_publisher/robot_state_publisher)
    spawn_gazebo_model (gazebo_ros/spawn_model)
    tf_relay (osrf_gear/tf2_relay)
  /ariac/arm1/
    arm_controller_spawner (controller_manager/controller_manager)
    fake_joint_calibration (rostopic/rostopic)
    joint_state_controller_spawner (controller_manager/controller_manager)
    robot_state_publisher (robot_state_publisher/robot_state_publisher)
    spawn_gazebo_model (gazebo_ros/spawn_model)
    tf_relay (osrf_gear/tf2_relay)
  /
    gazebo (gazebo_ros/gzserver)
    gazebo_gui (gazebo_ros/gzclient)
    gazebo_unpauser (osrf_gear/gazebo_unpauser.py)
    sensors_tf_publisher (robot_state_publisher/robot_state_publisher)
    spawn_gazebo_sensors (gazebo_ros/spawn_model)
    startup_ariac (osrf_gear/startup.sh)

ROS_MASTER_URI=http://172.18.0.22:11311

process[startup_ariac-1]: started with pid [377]
Ensuring scoring log file exists
File already exists: /home/ariac/.ariac/log/performance.log
[startup_ariac-1] process has finished cleanly
log file: /home/ariac/.ros/log/e5501c2e-3fc2-11ea-b2fb-0242ac120016/startup_ariac-1*.log
process[gazebo-2]: started with pid [379]
[ INFO] [1579991693.966447553]: Finished loading Gazebo ROS API Plugin.
[ INFO] [1579991693.968303032]: waitForService: Service [/gazebo/set_physics_properties] has not been advertised, waiting...
process[gazebo_gui-3]: started with pid [384]
[ INFO] [1579991694.429378132]: Finished loading Gazebo ROS API Plugin.
process[sensors_tf_publisher-4]: started with pid [450]
process[spawn_gazebo_sensors-5]: started with pid [514]
terminate called after throwing an instance of 'boost::exception_detail::clone_impl<boost::exception_detail::error_info_injector<boost::lock_error> >'
  what():  boost: mutex lock failed in pthread_mutex_lock: Invalid argument
Aborted (core dumped)
================================================================================REQUIRED process [gazebo_gui-3] has died!
process has died [pid 384, exit code 134, cmd /opt/ros/melodic/lib/gazebo_ros/gzclient __name:=gazebo_gui __log:=/home/ariac/.ros/log/e5501c2e-3fc2-11ea-b2fb-0242ac120016/gazebo_gui-3.log].
log file: /home/ariac/.ros/log/e5501c2e-3fc2-11ea-b2fb-0242ac120016/gazebo_gui-3*.log
Initiating shutdown!
================================================================================
[ariac/arm2/tf_relay-6] killing on exit
[spawn_gazebo_sensors-5] killing on exit
[sensors_tf_publisher-4] killing on exit
Traceback (most recent call last):
  File "/opt/ros/melodic/lib/gazebo_ros/spawn_model", line 31, in <module>
    from tf.transformations import quaternion_from_euler
  File "/opt/ros/melodic/lib/python2.7/dist-packages/tf/__init__.py", line 30, in <module>
    from tf2_ros import TransformException as Exception, ConnectivityException, LookupException, ExtrapolationException
  File "/opt/ros/melodic/lib/python2.7/dist-packages/tf2_ros/__init__.py", line 39, in <module>
    from .buffer_interface import *
  File "/opt/ros/melodic/lib/python2.7/dist-packages/tf2_ros/buffer_interface.py", line 32, in <module>
    import roslib; roslib.load_manifest('tf2_ros')
  File "/opt/ros/melodic/lib/python2.7/dist-packages/roslib/launcher.py", line 62, in load_manifest
    sys.path = _generate_python_path(package_name, _rospack) + sys.path
  File "/opt/ros/melodic/lib/python2.7/dist-packages/roslib/launcher.py", line 93, in _generate_python_path
    m = rospack.get_manifest(pkg)
  File "/usr/lib/python2.7/dist-packages/rospkg/rospack.py", line 167, in get_manifest
    return self._load_manifest(name)
  File "/usr/lib/python2.7/dist-packages/rospkg/rospack.py", line 211, in _load_manifest
    retval = self._manifests[name] = parse_manifest_file(self.get_path(name), self._manifest_name, rospack=self)
  File "/usr/lib/python2.7/dist-packages/rospkg/manifest.py", line 410, in parse_manifest_file
    from rosdep2.rospack import init_rospack_interface, is_ros_package, is_system_dependency, is_view_empty
  File "/usr/lib/python2.7/dist-packages/rosdep2/__init__.py", line 45, in <module>
    from .lookup import RosdepDefinition, RosdepView, RosdepLookup, \
  File "/usr/lib/python2.7/dist-packages/rosdep2/lookup.py", line 44, in <module>
    from .sources_list import SourcesListLoader
  File "/usr/lib/python2.7/dist-packages/rosdep2/sources_list.py", line 48, in <module>
    from .gbpdistro_support import get_gbprepo_as_rosdep_data, download_gbpdistro_as_rosdep_data
  File "/usr/lib/python2.7/dist-packages/rosdep2/gbpdistro_support.py", line 18, in <module>
    from .platforms.debian import APT_INSTALLER
  File "/usr/lib/python2.7/dist-packages/rosdep2/platforms/debian.py", line 36, in <module>
    from .pip import PIP_INSTALLER
  File "/usr/lib/python2.7/dist-packages/rosdep2/platforms/pip.py", line 33, in <module>
    import pkg_resources
  File "/usr/lib/python2.7/dist-packages/pkg_resources/__init__.py", line 3088, in <module>
    @_call_aside
  File "/usr/lib/python2.7/dist-packages/pkg_resources/__init__.py", line 3072, in _call_aside
    f(*args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/pkg_resources/__init__.py", line 3124, in _initialize_master_working_set
    list(map(working_set.add_entry, sys.path))
  File "/usr/lib/python2.7/dist-packages/pkg_resources/__init__.py", line 614, in add_entry
[gazebo_gui-3] killing on exit
    for dist in find_distributions(entry, True):
  File "/usr/lib/python2.7/dist-packages/pkg_resources/__init__.py", line 1964, in find_on_path
    for dist in factory(fullpath):
  File "/usr/lib/python2.7/dist-packages/pkg_resources/__init__.py", line 2034, in distributions_from_metadata
[gazebo-2] killing on exit
    root, entry, metadata, precedence=DEVELOP_DIST,
  File "/usr/lib/python2.7/dist-packages/pkg_resources/__init__.py", line 2473, in from_location
    py_version=py_version, platform=platform, **kw
  File "/usr/lib/python2.7/dist-packages/pkg_resources/__init__.py", line 2835, in _reload_version
    md_version = _version_from_file(self._get_metadata(self.PKG_INFO))
  File "/usr/lib/python2.7/dist-packages/pkg_resources/__init__.py", line 2437, in _version_from_file
    line = next(iter(version_lines), '')
  File "/usr/lib/python2.7/dist-packages/pkg_resources/__init__.py", line 2626, in _get_metadata
    if self.has_metadata(name):
  File "/usr/lib/python2.7/dist-packages/pkg_resources/__init__.py", line 2666, in __getattr__
    def __getattr__(self, attr):
KeyboardInterrupt
process[ariac/arm2/tf_relay-6]: started with pid [550]
RLException: cannot add process [ariac/arm2/spawn_gazebo_model-7] after process monitor has been shut down
The traceback for the exception was written to the log file
================================================================================REQUIRED process [ariac_sim-2] has died!
process has died [pid 350, exit code 1, cmd /opt/ros/melodic/lib/osrf_gear/gear.py --development-mode --visualize-sensor-views -f /opt/ros/melodic/share/osrf_gear/config/sample.yaml /opt/ros/melodic/share/osrf_gear/config/sample_user_config.yaml __name:=ariac_sim __log:=/home/ariac/.ros/log/e5501c2e-3fc2-11ea-b2fb-0242ac120016/ariac_sim-2.log].
log file: /home/ariac/.ros/log/e5501c2e-3fc2-11ea-b2fb-0242ac120016/ariac_sim-2*.log
Initiating shutdown!
================================================================================
[ariac_sim-2] killing on exit
[rosout-1] killing on exit
[master] killing on exit
shutting down processing monitor...
... shutting down processing monitor complete
done

Error 2: Says there are logs but there are none
There are a few log locations:
/home/ariac/.ros/log/e5501c2e-3fc2-11ea-b2fb-0242ac120016/gazebo_gui-3*.log
/home/ariac/.ros/log/e5501c2e-3fc2-11ea-b2fb-0242ac120016/ariac_sim-2*.log
However if I cd to each:

ariac@06fab0b7b083:~/.ros/log/e5501c2e-3fc2-11ea-b2fb-0242ac120016$ ls
ariac-arm2-tf_relay-6-stdout.log  master.log  roslaunch-06fab0b7b083-322.log  roslaunch-06fab0b7b083-353.log  rosout-1-stdout.log  rosout.log  sensors_tf_publisher-4-stdout.log

Reproduce

/bin/bash $HOME/ariac-docker/ariac-competitor/ariac_network.bash
/bin/bash $HOME/ariac-docker/ariac-server/build-images.sh

docker run --rm -it --name test -e XAUTHORITY=/tmp/.docker.xauth \
 -e ROS_IP=172.18.0.22 -e ROS_MASTER_URI=http://172.18.0.22:11311 \
--ip 172.18.0.22 --net ariac-network -v /etc/localtime:/etc/localtime:ro \
-v /tmp/.docker.xauth:/tmp/.docker.xauth -v /dev/log:/dev/log \
-v logs:/home/cloudsim/gazebo-logs ariac-server-melodic /bin/bash

Then, attempt

roslaunch osrf_gear sample_environment.launch

per: wiki/2019/tutorials/gear_interface

Personal Side
I just want to be able to start the ARIAC warehouse environment and have the gazebo GUI show up. Possibly interactively send commands to see how everything behaves. I understand that there is a script for capturing video, but I want something realtime. My primary concern is that the logs are missing, so I have no idea what gazebo is expecting from me.

TD:DR
I start the docker file after setting up the ariac_network.bash, build-images.sh, and finally the final command in run_container.bash with interactivity turned on so that I have access to the bash terminal. Executing a basic osrf_gear launch file crashes without any log files. The crash is related to the missing display of which I have solved