Support for uv

[antirotor]

Changelog Description

uv is an extremely fast Python package installer and resolver, written in Rust, and designed as a drop-in replacement for pip and pip-tools workflows. This PR changes use of uv instead of Poetry to handle dependencies.

Additional info

Thanks to it's speed, we could use it to dynamically build environments for every process we run (ie. when running DCC with different Python, to avoid python dependency conflicts with main process).

Testing:

Better start with clean repo clone - but it should also work with existing venv pretty fine.

• all current command should work (create-env, install-runtime, ...)

[tweak-wtf]

i just checked this out and fiddled around a bit to get it running on my windows box. naively running ./tools/manage.ps1 create-env until it was working. had uv installed already the same way as in manage.ps1.

here's the changes: feature/use-uv...tweak-wtf:ayon-launcher:feature/use-uv-playground

it was picking up my system python so i added a python flag in uv venv that would restrict to 3.9.
then it was failing for not finding .venv/Scripts/pre-commit as it somehow didn't got installed from the compiled requirements.txt. so adding it to windows-requirements.txt did the trick.

i do get version warnings when running the tray because i think it's checking my system python but seems cosmetic. anyways, with cached dependencies the install takes 3s for me rn😅

EDIT: i realized the .python-version file only in the end. i think this would be better than adding the python flag to uv venv

[antirotor]

EDIT: i realized the .python-version file only in the end. i think this would be better than adding the python flag to uv venv

Ah, I've never get into that issue because I am always running pyenv so there's always .python-version. Maybe the next step would be to change https://github.com/ynput/ayon-dependencies-tool to make use of uv. I am wondering what speed gains could we find there.

[antirotor]

So there is another use case perfect for uv - we can solve some of the issues like this one or in general all cases where we now install pyside2/6 and otio to running python environment. To support offline studios, we need to distribute "cached" pip index with AYON somehow and build venv using uv on the fly. Problem with uv and it's caching is (if I understand it correctly) that uv is not caching wheels themselves, but unzipped content and there is no "populate cache" functionality currently.

[tweak-wtf]

So there is another use case perfect for uv - we can solve some of the issues like this one or in general all cases where we now install pyside2/6 and otio to running python environment. To support offline studios, we need to distribute "cached" pip index with AYON somehow and build venv using uv on the fly. Problem with uv and it's caching is (if I understand it correctly) that uv is not caching wheels themselves, but unzipped content and there is no "populate cache" functionality currently.

concerning the cache location, i fiddled around with $UV_CACHE_DIR in my ayon-comfyui repo. I let users define it from server settings.
https://github.com/tweak-wtf/ayon-comfyui/blob/a9f95b5c7c73cc7ff6a325d3073a1d92537778af/client/ayon_comfyui/tools/install_and_run_server_venv.ps1#L15C1-L18C2

seems to work fine. however u must have 1 box that's connected to the internet at least once that would actually do all the caching to the provided dir. all air-gapped clients would use this cache dir for building environments

[iLLiCiTiT]

I would mark this for review. It is not fully finished, but that can be done once reviewed.

[antirotor]

I've reverted the python 3.11 requirement so now it is on par with our current poetry version with the same dependencies and everything, just using uv

[antirotor]

it still needs some refactoring in shell scripts though

[iLLiCiTiT]

I'm not able to install runtime dependencies. Fails on missing toml which is in optional dependencies. My guess is that it does not install optional dependencies, which are not that much optional 🙂 .

[antirotor]

I'm not able to install runtime dependencies. Fails on missing toml which is in optional dependencies. My guess is that it does not install optional dependencies, which are not that much optional 🙂 .

Right, I was replacing it with tomli but it didn't occured to me that it has different import signature. I've returned the old toml module there so it works. Looking forward to remove it once we move to 3.11

[antirotor]

colorama was available in AYON launcher before this PR so existing dependency packages won't have it. Probably is not the only dependency missing after this PR...

but where it was coming from? I cannot find it in the original pyproject.toml. This might come from different resolving strategies of uv/poetry maybe.

[iLLiCiTiT]

but where it was coming from? I cannot find it in the original pyproject.toml. This might come from different resolving strategies of uv/poetry maybe.

I guess from something like sphinx or mypy... poetry.lock did have it (you can check in develop)

[iLLiCiTiT]

Using it for few days now without major issues, but I'm not running DCCs much so it does not mean much. Linux docker builds do build (even with pyside2). I didn't try macOs, can someone try that?

[MustafaJafar]

but where it was coming from? I cannot find it in the original pyproject.toml. This might come from different resolving strategies of uv/poetry maybe.

I checked the log of dependencies tool, it mentioned installing colorama then

- colorama
Found existing installation: colorama 0.4.6
Uninstalling colorama-0.4.6:
  Successfully uninstalled colorama-0.4.6

Well, I do have this module installed for the same python version used inside AYON.
I can import it normally in any py script but not in AYON.

[iLLiCiTiT]

I checked the log of dependencies tool, it mentioned installing colorama then

You have to upload the AYON launcher build to your AYON server, and use that version of AYON launcher in bundle for which you're creating the dependency package to avoid uninstall.

[MustafaJafar]

For reference #280

[iLLiCiTiT]

For reference #280

You could just push it to this branch, or add comment here.

[MustafaJafar]

I tried build launcher installer and got this error

Traceback (most recent call last):
  File "E:\Ynput\ayon-launcher\tools\build_post_process.py", line 548, in get_uv_packages
    result = subprocess.run(
  File "C:\.uv\python\cpython-3.9.13-windows-x86_64-none\lib\subprocess.py", line 528, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['uv', 'pip', '--exclude-editable', 'list', '--format=json']' returned non-zero exit status 2.

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "E:\Ynput\ayon-launcher\tools\build_post_process.py", line 934, in <module>
    main()
  File "E:\Ynput\ayon-launcher\tools\build_post_process.py", line 927, in main
    post_build_process(ayon_root, build_root)
  File "E:\Ynput\ayon-launcher\tools\build_post_process.py", line 624, in post_build_process
    store_base_metadata(build_root, build_content_root, ayon_version)
  File "E:\Ynput\ayon-launcher\tools\build_post_process.py", line 599, in store_base_metadata
    "python_modules": get_uv_packages(),
  File "E:\Ynput\ayon-launcher\tools\build_post_process.py", line 555, in get_uv_packages
    raise RuntimeError(f"Failed to get uv packages: {e}") from e
RuntimeError: Failed to get uv packages: Command '['uv', 'pip', '--exclude-editable', 'list', '--format=json']' returned non-zero exit status 2.    
Traceback (most recent call last):
  File "E:\Ynput\ayon-launcher\tools\build_post_process.py", line 934, in <module>
    main()
  File "E:\Ynput\ayon-launcher\tools\build_post_process.py", line 930, in main
    create_installer(ayon_root, build_root)
  File "E:\Ynput\ayon-launcher\tools\build_post_process.py", line 888, in create_installer
    metadata = get_build_metadata(build_root)
  File "E:\Ynput\ayon-launcher\tools\build_post_process.py", line 288, in get_build_metadata
*** All done in 29 secs. You will find AYON and build log in '.\build' directory.

[MustafaJafar]

I build the launcher, updated it in my bundle and attempted to build dependency package.
It failed and I've no idea what is the error about but it seems related to the launcher as I can find 1.4.4.dev0 in many places.

poetry.mixology.failure.SolveFailure: Because ayondeppackage depends on ayon (1.4.4.dev0) which doesn't match any versions, version solving failed.

[BigRoy]

I build the launcher, updated it in my bundle and attempted to build dependency package. It failed and I've no idea what is the error about but it seems related to the launcher as I can find 1.4.4.dev0 in many places.

poetry.mixology.failure.SolveFailure: Because ayondeppackage depends on ayon (1.4.4.dev0) which doesn't match any versions, version solving failed.

I have a local locked_requirements.json that lists "ayon": "1.4.4.dev0" which is generated through here through here.

So likely it's coming from this launcher itself being 1.4.4-dev and something converting that somehow?

[antirotor]

I build the launcher, updated it in my bundle and attempted to build dependency package.

if you built it without that --exclude-editable then this is the exact outcome

[antirotor]

I build the launcher, updated it in my bundle and attempted to build dependency package.

if you built it without that --exclude-editable then this is the exact outcome

just to bump up visibility on it - it was caused by wrong argument order fixed by @iLLiCiTiT in 64ed055