How to mix with python packages ?

[asmodehn]

So looking at the readme, this is all very C++ centric.
I have been working on bridging ROS world with usual software world for a few months now, with python (since ROS supports python).

Perhaps I overlooked something but so far I couldn't find any simple way to make a ros package depending on a ( list of ) pip packages (versioned), or vice-versa make a pip package that depends on ros package.

It could simply be due to the fact that debs and pip have quite different usecases, and the closest I came to, was to use https://github.com/spotify/dh-virtualenv to embed all my pips in a venv. Still work in progress though...

So I was curious if there was any other way, or what would be your take on that problem.

[gerkey]

When you say that you want to make a ROS package depend on a list of pip packages, what do you want to do with that dependency information? Do you want to automate installation or configuration of an environment?

[asmodehn]

2 usecases, when working from source :

• a python package that already has pip dependencies. I want to use that python package from my usual catkin workspace, and catkin will get the pip dependencies in it s workspace properly (and at the proper version, without having to modify anything, like "porting" the python package to ROS).
• a ROS package that depends on python code. I want to be able to specify pip dependencies (not catkin packages) and use them directly from the ROS python code.

Then when building a deb package (ROS buildfarm), these dependencies are also packaged somehow (quite tricky) and when I install the package, I get the dependencies at the right version installed on my system, at least for that package...

[NikolausDemmel]

How are the two use cases different? Both sound to me like you want catkin packages to depend on specific verions of pip packages and then for those pip packages to automatically get installed and packaged. This sounds like "internal" use (i.e. with catkin), not external use (i.e. without catkin), which is what this repository is about.

Supporting this in the gerneal does indeed seem tricky. For example, what would you expect to happen when you have a catkin workspace with two ROS package, one depending on pip package foo=1.0, and another depending on foo=2.0?

[asmodehn]

Interesting point about "internal" vs "external"...
From my perspective, I would like my packages to be external (can be used without ROS code and robot stuff, but with usual python workflow), but still be usable with catkin(-> cmake), which is just a build system afterall, and for me is distinct from ROS in the sense that it is able to build non-ROS code as well... (although now that I think about it it might not be true for python...)

Context : I am working on web-related backend for robots. I need my package to:

• depend on a bunch of already existing web related pip packages.
• be usable like a usual python package (virtual env, pip dependencies, etc.) so that web python developers can jump into it, and so that all the specific web code made for python still work (virtual envs, import strategies, etc. -> no non-standard hacks)
• be usable from catkin, so that it can be included in workspaces and deployed on robots, without having to have a special extra workflow. Also be able to use the ros buildfarm and release it as a package for ROS people to use.

Following your semantic :

• If I make my package an "internal" package I cannot use all these pip dependencies without forking each of them and maintaining a catkin port of them all.
• If I make my package an "external" package, then robot engineer will have to learn an extra workflow to be able to use the package, but they probably wont even know about it anyway since it wont be in ROS packages at all to start with... And it will also make it more troublesome to develop with since I wont be able to include it in a workspace and I will need to jump between two very different environments...

So I m stuck between the two, trying to find the best compromise...
Feel free to advise :-)

About your question, I would expect each package to come along with its own dependencies, and using it when importing : package A uses its own foo=1.0 and package B uses its own foo=2.0

• It s more or less the way I do it already in my sources : a lot of dependencies as submodules of my package. It works fine from source, as long as there is no conflicts. But I cant release like this, as it could mess up someones system pretty bad. ref : celeros rostful
• It s the way npm dependency management work, and its the reason why virtual environments have been created (so that you can have per-package dependencies, with setuptools entrypoints seamlessly using them) . dh-virtualenv provides something similar for pip dependencies from a debian package.
• python (webrelated) packages move much faster than ROS packages, so I think it's better if the web dependencies are not following ROS releases, at least until a package version is stable enough to make it into a system deb (when I can use rosdep to depend on it).

[NikolausDemmel]

Thanks for the explanation. I think I almost got what exactly it is you want to do. I have the following questions:

1. In the Python packages you are developing, do you actually have on ROS dependencies? Or is it purely PIP-installable dependencies?
2. If it is the latter, then if I understand you correctly, you would like to package your PIP package not just as a PIP package, but also as a catkin package (possibly with some ROS specific wrapper code) including all the PIP dependencies to make it self contained and releasable with bloom, right?
3. In the case where you have two ROS pyton-packages, lets call them a_pkg and b_pkg, which depend on foo=1.0 and foo=2.0 respectively, what would happen if in a third ROS python-package c_pkg, you do import a_pkg; import b_pkg. Is that something Python can deal with? I have a hard time imagining how this would work, given that a_pkg and b_pkg come both with distinct venvs containg the foo dependency (in different versions).

[asmodehn]

1. I have a few ros dependencies. mostly rospy, rosgraph, and other core packages. I m trying to keep them minimal (trying to keep maintenance simple). I have, I think, much more pip dependencies (python web related packages).
2. Yes. With or without ROS dependencies, releasing "robot usable" fully functional packages with bloom would be useful ( kind of "all 20 web pip packages merged into one awesome webserver as 1 ros package" ).
3. Python doesn't need to deal with it. The package manager (pip) does.
   c_pkg comes also with his virtual environment. Into this virtual environment, a_pkg is installed by pip as a dependency, along with foo==1.0. Then b_pkg is installed by pip as the other dependency, and then foo==2.0 overwrites foo==1.0. Not exactly sure about the details (v0.1 might never be installed, there might be warnings/errors, it might depend on how you specify it exactly and it's not a simple problem - see pip issue ), but the point is that there can be only one version of the package installed in the virtual environment in the end. So you know which version everyone is using, likely foo==2.0.
   FYI : Related pip issues for completeness :

• pip needs a dependency resolver pypa/pip#988
• Add a pip check command. pypa/pip#2492

3. Highlight the fact that a ROS python-package, who wants to depend on a ROS python package with virtualenv, has to have a virtualenv as well, because in the end some of the dependencies are just pip packages that wouldn't work with ROS out of the box.
   Eventually, when all python dependencies make it into their own independent ROS/system package, then the ROS python package could drop its virtualenv and become a usual ROS/system package, since he doesn't have any pip dependency anymore.

[asmodehn]

After quite a lot of painful trials and error, I think I finally managed to find a part of the solution...

I m working on a catkin extension https://github.com/asmodehn/catkin_pure_python that allows to retrieve pip packages into the workspace and define your current package as a package managed by pip ( and not default catkin ) but stills install it in the workspace.

This way one can work with pip dependencies, with latest python packages, and have other packages find them and import them. And this works from catkin with workspaces.

It works for me for the devel and the install space so far. I ll be using that for my development to find out how far this strategy can be used.
Debian packaging is another challenge for another time...

[brainwane]

Hello! Please try the beta of the new resolver in pip 20.2 and see whether that helps?

[brainwane]

pip 20.3 has the new dependency resolver on by default; please see the documentation on how to test and migrate in case it helps you address this problem.

[asmodehn]

@brainwane Thanks for the update.
For people arriving here nowadays, here is a small context heads-up related to this topic:

ROS v2 is now up and running with a different system to manage dependencies (including python packages), see https://design.ros2.org/articles/ament.html for design and https://github.com/ament/ament_cmake for implementation.

ROS v1 has a couple of tools addressing this problem in slightly different ways :

• the tool I mentioned before in this issue, not really active anymore, is still available there : https://github.com/pyros-dev/catkin_pip
• another similar tool, but with a different approach regarding pip, with a more active team : https://github.com/locusrobotics/catkin_virtualenv