Gentoo removing python2 support in some packages (breaking ROS build)

[awesomebytes]

Hello,

I've recently noticed my CI builds breaking cause some packages are being removed from the support for python2 targets.

For now I found:

• dev-util/rosdep (which we have an overlay on dev-python/rosdep instead, weird, I opened a bug on Gentoo to try to earn some time to not break my builds https://bugs.gentoo.org/708552 and try to move to py3)
• dev-python/rospkg (which we have an overlay but it's pretty old, 1.1.4 vs 1.2.3 upstream in pip)

So with this issue I would like to ask for 2 things:

1. Add overlays of what's needed to support Python 2 until the EOF of Ubuntu 18 (April 2023) which would be the EOF of Melodic I think.
2. Some help on how to emerge both py2 and py3 versions of ROS packages. I just tried and I got stuck in the... not finding gencpp, genlisp... etc cause they are generated for Py2 only, not for Py3.

I'm obviously up to report and help. But I'd like to know there will be some support.

[flabrosse]

I've spent the last few days trying to update my system, due to the mess of python2 and python3, which I made worse by trying to emerge ros. I think I now have a configuration that will emerge, fingers crossed.

However, tf is still needed by many packages in melodic, and still only works with python2. Are there plans to update this package? I don't know python much so can't really assess if going from 2 to 3 is a big job. Or is it time to go to a newer version of ROS (knowing that some of my collaborators still use kinetic!).

[allenh1]

However, tf is still needed by many packages in melodic, and still only works with python2. Are there plans to update this package? I don't know python much so can't really assess if going from 2 to 3 is a big job. Or is it time to go to a newer version of ROS (knowing that some of my collaborators still use kinetic!).

I highly encourage you to move to tf2 instead of tf if at all possible. The latest release, noetic supports python3 by default, so I'd encourage you to go there. Or, if you're feeling really brave, I'd like to get ROS 2 Foxy working on Gentoo. Though I already generate ebuilds for Foxy and the like, I have not successfully emerged on my own systems and or made the packages work.

@flabrosse I'll spend some time in the near future getting my own Gentoo system up and running with noetic, I have not tested it yet.

I'll also start working on Foxy, if I get around to it. I did get a good amount of it done, but will need to figure out the eclass some more.

[flabrosse]

Hi @allenh1, ROS 2 is not an option, as I said, some of my collaborators still use kinetic... Also, I am not the one using tf, some packages depend on it:

equery depends tf
 * These packages depend on tf:
ros-melodic/diagnostic_common_diagnostics-1.9.3 (ros-melodic/tf)
ros-melodic/geometry-1.12.1-r1 (ros-melodic/tf)
ros-melodic/hector_mapping-0.4.1-r1 (ros-melodic/tf)
ros-melodic/interactive_marker_tutorials-0.10.5-r1 (ros-melodic/tf)
ros-melodic/interactive_markers-1.11.5-r1 (ros-melodic/tf)
ros-melodic/laser_geometry-1.6.5-r1 (ros-melodic/tf)
ros-melodic/robot_state_publisher-1.14.1-r1 (ros-melodic/tf)
ros-melodic/rqt_nav_view-0.5.7 (ros-melodic/tf)
ros-melodic/rqt_pose_view-0.5.8 (ros-melodic/tf)
ros-melodic/rviz-1.13.13-r1 (ros-melodic/tf)
ros-melodic/tf_conversions-1.12.1-r1 (ros-melodic/tf)
ros-melodic/turtle_tf-0.2.2 (ros-melodic/tf)
ros-melodic/warehouse_ros-0.9.4-r1 (ros-melodic/tf)

[flabrosse]

Help! ;-)

Can't catkin_make any more. This insists on using python2, but most packages are not compiled with python2. So the build fails with not finding catkin_pkg. I guess I could force the use of the overlay version (which still supports 2_7) but I'm not convinced this will not break at some other package.

Any idea on how to force catkin_make to use python3?

[flabrosse]

Actually, I just tried to recompile the overlay catkin_pkg with python2_7 and emerge wants me to remove this version of python...

[flabrosse]

The running of python2 seems to be coming from empy, still built with python2_7 support, because tf only supports python2_7. Because of that, I can't even reemerge empy. And tf is needed by many packages (see above).

This to me indicates that melodic is dead on gentoo.

[flabrosse]

It seems that python2 is specified in

/opt/ros/melodic/etc/catkin/profile.d/1.ros_python_version.sh

created by package ros_environment. For some reason the value in there is 2, despite the package being compiled without support for python2_7.

If I change the value to 3 in this file and remove devel and build (maybe only one is needed) from my workspace, then python3 is used (even after reverting to 2 in the file above).

Seems there's a bug in ros_environment.

[flabrosse]

What I feared happened. I was in the process of upgrading a robot that had not been upgraded since March. To simplify the task I removed all the ROS packages. I now can't install ROS anymore because I can't install empy because of the reason explained a few posts above. And at the moment I don't seem to be able to install noetic (due to missing package, see #997).

[flabrosse]

More specific info on the issue:

emerge -av ros-melodic/catkin

These are the packages that would be merged, in order:

Calculating dependencies... done!

emerge: there are no ebuilds built with USE flags to satisfy "dev-python/empy[python_targets_python2_7(-)?,python_targets_python3_6(-)?,-python_single_target_python2_7(-),-python_single_target_python3_6(-)]".
!!! One of the following packages is required to complete your request:
- ros-melodic/catkin-0.7.26-r1::ros-overlay (Change USE: -python_targets_python2_7, this change violates use flag constraints defined by ros-melodic/catkin-0.7.26-r1: 'any-of ( python_targets_python2_7 python_targets_python3_6 )')
(dependency required by "ros-melodic/catkin-0.7.26-r1::ros-overlay" [ebuild])
(dependency required by "ros-melodic/catkin" [argument])

[flabrosse]

And the same happens with noetic.

[allenh1]

@flabrosse wow -- lots of issues there. Let me see what I can do... I'll start by updating things in the overlay, then I'll bust out my docker environment and try and get things going.

[flabrosse]

Thanks @allenh1 for looking into this. Just synced the protage tree as well as the overlay and tried to install one of the noetic packages:

emerge -av ros-noetic/costmap_2d

These are the packages that would be merged, in order:

Calculating dependencies... done!

emerge: there are no ebuilds built with USE flags to satisfy "dev-python/empy[python_targets_python2_7(-)?,python_targets_python3_6(-)?,-python_single_target_python2_7(-),-python_single_target_python3_6(-)]".
!!! One of the following packages is required to complete your request:
- ros-noetic/costmap_2d-1.17.1-r1::ros-overlay (Change USE: -python_targets_python2_7, this change violates use flag constraints defined by ros-noetic/costmap_2d-1.17.1-r1: 'any-of ( python_targets_python2_7 python_targets_python3_6 )')
(dependency required by "ros-noetic/costmap_2d-1.17.1-r1::ros-overlay" [ebuild])
(dependency required by "ros-noetic/costmap_2d" [argument])

noetic still tries to install support for python2_7 (PYTHON_COMPAT=( python{2_7,3_5,3_6} ) from the ebuild), which fails.
Can't 2_7 just be removed?

[allenh1]

@flabrosse yep! I'll patch superflore accordingly (and add 3.7-3.9 while I'm at it).

[flabrosse]

Would that imply that all versions of python will need to be installed? The fact that it is trying to install for 2.7 and fails because not installed makes me think it would fail on 3.9 (still masked in the tree).