Skip to content

Wait for user session for preloaded disposables#757

Open
ben-grande wants to merge 3 commits intoQubesOS:mainfrom
ben-grande:preload-delay-gui-daemon
Open

Wait for user session for preloaded disposables#757
ben-grande wants to merge 3 commits intoQubesOS:mainfrom
ben-grande:preload-delay-gui-daemon

Conversation

@ben-grande
Copy link
Contributor

@ben-grande ben-grande commented Nov 26, 2025

With the GUI agent patch, it can start before the GUI daemon connects, allowing the user session to complete. Wait both services to guarantee no enabled user or system service tries to start after the preload is used.

Requires: QubesOS/qubes-gui-agent-linux#251
Requires: QubesOS/qubes-gui-agent-linux#255
Fixes: QubesOS/qubes-issues#9940
For: QubesOS/qubes-issues#1512

@ben-grande ben-grande force-pushed the preload-delay-gui-daemon branch from 7351532 to e28c131 Compare November 26, 2025 21:03
@ben-grande ben-grande force-pushed the preload-delay-gui-daemon branch 2 times, most recently from 22d069b to 4644fb9 Compare November 27, 2025 07:56
@codecov
Copy link

codecov bot commented Nov 27, 2025

Codecov Report

❌ Patch coverage is 75.34247% with 18 lines in your changes missing coverage. Please review.
✅ Project coverage is 70.13%. Comparing base (e23e32d) to head (9ecad91).

Files with missing lines Patch % Lines
qubes/ext/gui.py 80.39% 10 Missing ⚠️
qubes/vm/dispvm.py 0.00% 8 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main     #757      +/-   ##
==========================================
- Coverage   70.21%   70.13%   -0.09%     
==========================================
  Files          61       61              
  Lines       13942    13961      +19     
==========================================
+ Hits         9790     9792       +2     
- Misses       4152     4169      +17     
Flag Coverage Δ
unittests 70.13% <75.34%> (-0.09%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@ben-grande ben-grande marked this pull request as ready for review November 29, 2025 00:33
@ben-grande ben-grande force-pushed the preload-delay-gui-daemon branch from 4644fb9 to f66f7c3 Compare November 29, 2025 01:05
@ben-grande
Copy link
Contributor Author

It is taking a lot of time for the user session to complete... maybe there is something timing out?

QubesOS/qubes-gui-agent-linux#251 (comment)

Nov 29 00:51:20 dom0 qubesd[71112]: INFO: vm.disp457: Activating qube
Nov 29 00:51:24 dom0 qubesd[71112]: INFO: vm.disp457: Preload startup waiting 'qubes.WaitForRunningSystem' with '60' seconds timeout
Nov 29 00:51:24 dom0 qubesd[71112]: INFO: vm.disp457: Preload startup waiting 'qubes.WaitForSession' with '60' seconds timeout
Nov 29 00:51:36 dom0 qubesd[71112]: INFO: vm.disp457: Preload startup completed 'qubes.WaitForRunningSystem'
Nov 29 00:52:09 dom0 qubesd[71112]: INFO: vm.disp457: Preload startup completed 'qubes.WaitForSession'
  • 12 seconds for the system to be running.
  • 45 seconds for the user session to be running.

@ben-grande
Copy link
Contributor Author

Nov 29 01:49:22 dom0 qubesd[116914]: INFO: vm.disp4225: Preload startup log for 'qubes.WaitForSession': '(b'', b"+ set -eu\n++ qubesdb-read --default=True /qubes-gui-enabled\n+ test True = True\n++ qubesdb-read /default-user\n+ user=user\n+ '[' -e /var/run/qubes/qrexec-server.user.sock ']'\n+ sleep 0.1\n+ '[' -e /var/run/qubes/qrexec-server.user.sock ']'\n+ sleep 0.1\n+ '[' -e /var/run/qubes/qrexec-server.user.sock ']'\n+ sleep 0.1\n+ '[' -e /var/run/qubes/qrexec-server.user.sock ']'\n+ sleep 0.1\n+ '[' -e /var/run/qubes/qrexec-server.user.sock ']'\n+ sleep 0.1\n+ '[' -e /var/run/qubes/qrexec-server.user.sock ']'\n+ sleep 0.1\n+ '[' -e /var/run/qubes/qrexec-server.user.sock ']'\n+ sleep 0.1\n+ '[' -e /var/run/qubes/qrexec-server.user.sock ']'\n+ sleep 0.1\n+ '[' -e /var/run/qubes/qrexec-server.user.sock ']'\n+ sleep 0.1\n+ '[' -e /var/run/qubes/qrexec-server.user.sock ']'\n+ sleep 0.1\n+ '[' -e /var/run/qubes/qrexec-server.user.sock ']'\n+ sleep 0.1\n+ '[' -e /var/run/qubes/qrexec-server.user.sock ']'\n+ sleep 0.1\n+ '[' -e /var/run/qubes/qrexec-server.user.sock ']'\n+ sleep 0.1\n+ '[' -e /var/run/qubes/qrexec-server.user.sock ']'\n+ sleep 0.1\n+ '[' -e /var/run/qubes/qrexec-server.user.sock ']'\n+ sleep 0.1\n+ '[' -e /var/run/qubes/qrexec-server.user.sock ']'\n+ systemctl --user --wait --quiet is-system-running\n+ exit 0\n")'

So, 1.5s waiting for the sock, not much, compared to the 46s the call took, so most of the time has to be systemctl --user --wait --quiet is-system-running.

@ben-grande
Copy link
Contributor Author

ben-grande commented Nov 29, 2025

So, 1.5s waiting for the sock, not much, compared to the 46s the call took, so most of the time has to be systemctl --user --wait --quiet is-system-running.

This is very strange because the long time doesn't happen when calling the script manually, only when calling from qubesd.

The following is very fast (0.02-1.5s) (do not preload):

time qvm-run -p --dispvm=default-dvm-gui --service qubes.WaitForSession
time qvm-run -p --dispvm=default-dvm-gui 'time /etc/qubes-rpc/qubes.WaitForSession'
time qvm-run -p --dispvm=default-dvm-gui 'time systemctl --user --wait is-system-running'

So why is qubesd variant so slow?

@marmarek
Copy link
Member

maybe add systemd-analyze blame or systemd-analyze plot somewhere there? (see also other options to this tool)

@qubesos-bot
Copy link

qubesos-bot commented Nov 29, 2025

OpenQA test summary

Complete test suite and dependencies: https://openqa.qubes-os.org/tests/overview?distri=qubesos&version=4.3&build=2026020704-4.3&flavor=pull-requests

Test run included the following:

New failures, excluding unstable

Compared to: https://openqa.qubes-os.org/tests/overview?distri=qubesos&version=4.3&build=2026020304-devel&flavor=update

  • system_tests_network

  • system_tests_guivm_gui_interactive

    • collect_logs: wait_serial (wait serial expected)
      # wait_serial expected: qr/JrqaX-\d+-/...

    • collect_logs: Failed (test died + timed out)
      # Test died: command 'curl --form upload=@journalctl.log --form upn...

    • collect_logs: wait_serial (wait serial expected)
      # wait_serial expected: qr/lqE1a-\d+-/...

  • system_tests_basic_vm_qrexec_gui_xfs

    • TC_20_NonAudio_whonix-gateway-18-pool: test_000_start_shutdown (error)
      subprocess.CalledProcessError: Command 'qubes.WaitForSession' retur...
  • system_tests_gui_tools

    • qubesmanager_manager: unnamed test (unknown)
    • qubesmanager_manager: Failed (test died)
      # Test died: no candidate needle with tag(s) 'qube-manager-dom0-log...

Failed tests

7 failures
  • system_tests_network

  • system_tests_guivm_gui_interactive

    • collect_logs: wait_serial (wait serial expected)
      # wait_serial expected: qr/JrqaX-\d+-/...

    • collect_logs: Failed (test died + timed out)
      # Test died: command 'curl --form upload=@journalctl.log --form upn...

    • collect_logs: wait_serial (wait serial expected)
      # wait_serial expected: qr/lqE1a-\d+-/...

  • system_tests_basic_vm_qrexec_gui_xfs

    • TC_20_NonAudio_whonix-gateway-18-pool: test_000_start_shutdown (error)
      subprocess.CalledProcessError: Command 'qubes.WaitForSession' retur...
  • system_tests_gui_tools

    • qubesmanager_manager: unnamed test (unknown)
    • qubesmanager_manager: Failed (test died)
      # Test died: no candidate needle with tag(s) 'qube-manager-dom0-log...

Fixed failures

Compared to: https://openqa.qubes-os.org/tests/166096#dependencies

31 fixed
  • system_tests_network

  • system_tests_pvgrub_salt_storage

    • system_tests: Fail (unknown)
      Tests qubes.tests.integ.grub failed (exit code 1), details reported...

    • system_tests: Failed (test died)
      # Test died: Some tests failed at qubesos/tests/system_tests.pm lin...

    • TC_41_HVMGrub_fedora-42-xfce: test_000_standalone_vm (error)
      qubes.exc.QubesVMError: Cannot connect to qrexec agent for 120 seco...

    • TC_41_HVMGrub_fedora-42-xfce: test_010_template_based_vm (error)
      qubes.exc.QubesVMError: Cannot connect to qrexec agent for 120 seco...

  • system_tests_gui_interactive

    • collect_logs: wait_serial (wait serial expected)
      # wait_serial expected: qr/m~uU4-\d+-/...

    • collect_logs: Failed (test died + timed out)
      # Test died: command 'curl --form upload=@journalctl.log --form upn...

  • system_tests_network_ipv6

  • system_tests_network_updates

    • system_tests: Fail (unknown)
      Tests qubes.tests.integ.dom0_update failed (exit code 1), details r...

    • system_tests: Failed (test died)
      # Test died: Some tests failed at qubesos/tests/system_tests.pm lin...

    • TC_00_Dom0Upgrade_whonix-gateway-18: test_000_update_salt (failure)
      ?[0;31mDuration: 139139.863 ms?[0;0m... ?[0;31m Changes: ?[0;0m

  • system_tests_dispvm

    • system_tests: Fail (unknown)
      Tests qubes.tests.integ.dispvm failed (exit code 1), details report...

    • system_tests: Failed (test died)
      # Test died: Some tests failed at qubesos/tests/system_tests.pm lin...

    • TC_20_DispVM_whonix-workstation-18: test_015_preload_race_more (error)
      raise TimeoutError from exc_val... TimeoutError

  • system_tests_guivm_vnc_gui_interactive

    • guivm_manager: unnamed test (unknown)

    • guivm_manager: Failed (test died)
      # Test died: no candidate needle with tag(s) 'menu-qubes-tools' mat...

    • guivm_manager: wait_serial (wait serial expected)
      # wait_serial expected: qr/yp0HG-\d+-/...

  • system_tests_audio

    • system_tests: Fail (unknown)
      Tests qubes.tests.integ.audio failed (exit code 1), details reporte...

    • system_tests: Failed (test died)
      # Test died: Some tests failed at qubesos/tests/system_tests.pm lin...

    • TC_20_AudioVM_PipeWire_whonix-workstation-18: test_260_audio_mic_enabled_switch_audiovm (failure)
      AssertionError: Failed to get mic attach/detach status!

  • system_tests_whonix

    • whonixcheck: fail (unknown)
      Whonixcheck for anon-whonix failed...

    • whonixcheck: wait_serial (wait serial expected)
      # wait_serial expected: qr/D8rIe-\d+-/...

    • whonixcheck: Failed (test died + timed out)
      # Test died: command 'qvm-run -ap whonix-gateway-18 'LC_ALL=C whon...

Unstable tests

Details

Performance Tests

Performance degradation:

No issues

Remaining performance tests:

No remaining performance tests

@marmarek
Copy link
Member

  • TC_00_Backup: test_000_basic_backup (error)
    qubes.exc.QubesValueError: Qube does not support the RPC(s) 'qubes....

This should not fail restoring a backup... (not sure if this PR is to blame, but it's likely)

return False
supported = True
missing_services = []
for service in ["qubes.WaitForRunningSystem", "qubes.WaitForSession"]:
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just supporting qubes.WaitForSession RPC doesn't guarantee it supports late gui-daemon connection. I'll add some explicit feature about that.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed. This code remains the same, it changes only on vm/dispvm.py.

@ben-grande
Copy link
Contributor Author

  • TC_00_Backup: test_000_basic_backup (error)
    qubes.exc.QubesValueError: Qube does not support the RPC(s) 'qubes....

This should not fail restoring a backup... (not sure if this PR is to blame, but it's likely)

This was fixed.

@ben-grande
Copy link
Contributor Author

maybe add systemd-analyze blame or systemd-analyze plot somewhere there? (see also other options to this tool)

QubesOS/qubes-gui-agent-linux#251 (comment)

But when looking at the logs:

journalctl --user

I don't know why qubes-notification-agent fails to start the first time even if dom0 user session is logged in.

@ben-grande ben-grande force-pushed the preload-delay-gui-daemon branch from f66f7c3 to 41714fd Compare December 1, 2025 12:03
@ben-grande
Copy link
Contributor Author

Hum, what is the chance that OpenQA ran without QubesOS/qubes-notification-proxy@7b82428? I will run the integration tests but it seems to be working correctly.

qvm-features-request was not triggered when installing the gui-agent package with rpm -Uvh.

ben-grande added a commit to ben-grande/qubes-core-qrexec that referenced this pull request Dec 1, 2025
By default, it resolves the link path literally, which causes problem if
other commands do not run on the same directory the link is at.

For: QubesOS/qubes-core-admin#757
For: QubesOS/qubes-notification-proxy#13
For: QubesOS/qubes-issues#9940
For: QubesOS/qubes-issues#1512
@ben-grande
Copy link
Contributor Author

  • Technically, I can lower the nogui mean on long runs by setting guivm=None

QubesOS/qubes-gui-agent-linux#255


Apparently, I removed qubes.WaitForSession when rebasing... adding again.

@ben-grande ben-grande force-pushed the preload-delay-gui-daemon branch from ceabe8e to d341b15 Compare January 8, 2026 15:00
@ben-grande
Copy link
Contributor Author

ben-grande commented Jan 8, 2026

openQArun PR_LABEL=openqa-pending TEST=system_tests_dispvm,system_tests_dispvm_perf TEST_TEMPLATES=debian-13-xfce

@ben-grande ben-grande force-pushed the preload-delay-gui-daemon branch from d341b15 to 11f85ed Compare January 9, 2026 00:40
ben-grande added a commit to ben-grande/qubes-core-qrexec that referenced this pull request Jan 9, 2026
Or if GUID of the client can be found on the server. This script will be
replicated to qubes-core-agent-linux.

For: QubesOS/qubes-notification-proxy#13
For: QubesOS/qubes-gui-agent-linux#251
For: QubesOS/qubes-core-admin#757
For: QubesOS/qubes-issues#1512
For: QubesOS/qubes-issues#9940
Fixes: QubesOS/qubes-issues#10443
ben-grande added a commit to ben-grande/qubes-core-agent-linux that referenced this pull request Jan 9, 2026
@ben-grande
Copy link
Contributor Author

ben-grande commented Jan 9, 2026

openQArun PR_LABEL=openqa-pending TEST=system_tests_dispvm,system_tests_dispvm_perf TEST_TEMPLATES=debian-13-xfce

@ben-grande
Copy link
Contributor Author

ben-grande commented Jan 9, 2026

openQArun PR_LABEL=openqa-pending TEST=system_tests_dispvm,system_tests_dispvm_perf TEST_TEMPLATES=debian-13-xfce

@ben-grande
Copy link
Contributor Author

ben-grande commented Jan 10, 2026 via email

@ben-grande ben-grande force-pushed the preload-delay-gui-daemon branch from 11f85ed to 7b42702 Compare January 10, 2026 16:24
@ben-grande
Copy link
Contributor Author

ben-grande commented Jan 10, 2026

openQArun PR_LABEL=openqa-pending TEST=system_tests_dispvm,system_tests_dispvm_perf TEST_TEMPLATES=debian-13-xfce

@ben-grande
Copy link
Contributor Author

ben-grande commented Jan 10, 2026

I made a mistake of removing guivm when it should have one... but this led to find some slow behavior when creating a qube:

Having a GUIVM slows down the creation of a qube (app.add_new_vm() or dispvm.create_on_disk()) by 400ms. Does not slow down usage of preloaded disposable, but still, something worth checking out on qubes.ext.gui.

@ben-grande
Copy link
Contributor Author

ben-grande commented Jan 12, 2026

Having a GUIVM slows down the creation of a qube (app.add_new_vm() or dispvm.create_on_disk()) by 400ms. Does not slow down usage of preloaded disposable, but still, something worth checking out on qubes.ext.gui.

It is qubesappmenusext/__init__ create_on_disk(), it calls await self.update_appemnus which calls qubes.UpdateAppMenusFor+QUBE or qvm-appmenus --update --quiet --force -- QUBE. Some places (setting properties) uses asyncio.ensure_future(self.update_appmenus(vm). This also affects cleanup:

It takes >400ms to create, >300ms to remove.

With the GUI agent patch, it can start before the GUI daemon connects,
allowing the user session to complete. Wait both services to guarantee
no enabled user or system service tries to start after the preload is
used.

Requires: QubesOS/qubes-gui-agent-linux#251
Requires: QubesOS/qubes-gui-agent-linux#255
Fixes: QubesOS/qubes-issues#9940
For: QubesOS/qubes-issues#1512
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Start GUI session in VM before GUI daemon connects

3 participants