Skip to content

Add atlasopenmagic DID finder#1292

Open
ponyisi wants to merge 9 commits intodevelopfrom
atlasopenmagic
Open

Add atlasopenmagic DID finder#1292
ponyisi wants to merge 9 commits intodevelopfrom
atlasopenmagic

Conversation

@ponyisi
Copy link
Collaborator

@ponyisi ponyisi commented Jan 31, 2026

Fixes #1306

Adds an atlasopenmagic DID finder to the backend (will need a frontend update for proper support in the client). Gives the list of URLs reported by the atlasopenmagic package for a given release/dataset[/skim] combination.

@codecov
Copy link

codecov bot commented Jan 31, 2026

Codecov Report

❌ Patch coverage is 93.54839% with 2 lines in your changes missing coverage. Please review.
✅ Project coverage is 87.13%. Comparing base (e7748f1) to head (bb8eedb).

Files with missing lines Patch % Lines
...sopenmagic/src/did_finder_atlasopenmagic/celery.py 93.54% 1 Missing and 1 partial ⚠️
Additional details and impacted files
@@             Coverage Diff             @@
##           develop    #1292      +/-   ##
===========================================
+ Coverage    87.07%   87.13%   +0.05%     
===========================================
  Files           96       97       +1     
  Lines         3389     3420      +31     
  Branches       399      406       +7     
===========================================
+ Hits          2951     2980      +29     
- Misses         367      368       +1     
- Partials        71       72       +1     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@BenGalewsky
Copy link
Contributor

When I deploy this I get:

/opt/servicex/.venv/lib/python3.14/site-packages/billiard/pool.py:1855: SyntaxWarning: 'return' in a 'finally' block
  return self._send_ack(
/opt/servicex/.venv/lib/python3.14/site-packages/billiard/connection.py:349: SyntaxWarning: 'return' in a 'finally' block
  return f
/opt/servicex/.venv/lib/python3.14/site-packages/billiard/connection.py:351: SyntaxWarning: 'return' in a 'finally' block
  return self._get_more_data(ov, maxsize)
/opt/servicex/.venv/lib/python3.14/site-packages/celery/concurrency/asynpool.py:400: SyntaxWarning: 'return' in a 'finally' block
  return remove(fd)
Traceback (most recent call last):
  File "/opt/servicex/.venv/bin/celery", line 8, in <module>
    sys.exit(main())
             ~~~~^^
  File "/opt/servicex/.venv/lib/python3.14/site-packages/celery/__main__.py", line 15, in main
    sys.exit(_main())
             ~~~~~^^
  File "/opt/servicex/.venv/lib/python3.14/site-packages/celery/bin/celery.py", line 236, in main
    return celery(auto_envvar_prefix="CELERY")
  File "/opt/servicex/.venv/lib/python3.14/site-packages/click/core.py", line 1157, in __call__
    return self.main(*args, **kwargs)
           ~~~~~~~~~^^^^^^^^^^^^^^^^^
  File "/opt/servicex/.venv/lib/python3.14/site-packages/click/core.py", line 1078, in main
    rv = self.invoke(ctx)
  File "/opt/servicex/.venv/lib/python3.14/site-packages/click/core.py", line 1688, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
                           ~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^
  File "/opt/servicex/.venv/lib/python3.14/site-packages/click/core.py", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
           ~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/servicex/.venv/lib/python3.14/site-packages/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
  File "/opt/servicex/.venv/lib/python3.14/site-packages/click/decorators.py", line 33, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/opt/servicex/.venv/lib/python3.14/site-packages/celery/bin/base.py", line 135, in caller
    return f(ctx, *args, **kwargs)
  File "/opt/servicex/.venv/lib/python3.14/site-packages/celery/bin/worker.py", line 348, in worker
    worker = app.Worker(
        hostname=hostname, pool_cls=pool_cls, loglevel=loglevel,
    ...<4 lines>...
        quiet=ctx.obj.quiet,
        **kwargs)
  File "/opt/servicex/.venv/lib/python3.14/site-packages/celery/worker/worker.py", line 98, in __init__
    self.setup_instance(**self.prepare_args(**kwargs))
    ~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/servicex/.venv/lib/python3.14/site-packages/celery/worker/worker.py", line 131, in setup_instance
    self.on_init_blueprint()
    ~~~~~~~~~~~~~~~~~~~~~~^^
  File "/opt/servicex/.venv/lib/python3.14/site-packages/celery/apps/worker.py", line 116, in on_init_blueprint
    self._custom_logging = self.setup_logging()
                           ~~~~~~~~~~~~~~~~~~^^
  File "/opt/servicex/.venv/lib/python3.14/site-packages/celery/apps/worker.py", line 180, in setup_logging
    return self.app.log.setup(
           ~~~~~~~~~~~~~~~~~~^
        self.loglevel, self.logfile,
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        redirect_stdouts=False, colorize=colorize, hostname=self.hostname,
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    )
    ^
  File "/opt/servicex/.venv/lib/python3.14/site-packages/celery/app/log.py", line 63, in setup
    handled = self.setup_logging_subsystem(
        loglevel, logfile, colorize=colorize, hostname=hostname,
    )
  File "/opt/servicex/.venv/lib/python3.14/site-packages/celery/app/log.py", line 118, in setup_logging_subsystem
    get_multiprocessing_logger(),
    ~~~~~~~~~~~~~~~~~~~~~~~~~~^^
  File "/opt/servicex/.venv/lib/python3.14/site-packages/celery/utils/log.py", line 270, in get_multiprocessing_logger
    return util.get_logger()
           ~~~~~~~~~~~~~~~^^
  File "/opt/servicex/.venv/lib/python3.14/site-packages/billiard/util.py", line 131, in get_logger
    logging._acquireLock()
    ^^^^^^^^^^^^^^^^^^^^
AttributeError: module 'logging' has no attribute '_acquireLock'

@BenGalewsky
Copy link
Contributor

I did update the deployment template to apply resource limit and request settings as we've done to all other pods

@BenGalewsky
Copy link
Contributor

Can you provide suggestions on how to test this?

enabled: true
image: sslhep/servicex-did-finder-atlasopenmagic
pullPolicy: Always
tag: develop
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It will once PR is accepted

uses: actions/setup-python@v6.2.0
with:
python-version: ${{ matrix.python-version }}
python-version: |
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why are we removing the template arguments here?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So I would like this DID finder to use Python 3.14; poetry will complain if it can't set up the appropriate release, so we need both Python 3.14 and 3.10 in the test environment (3.10 for the other servicex containers). Since we pin to specific Python versions by poetry, there is no value in iterating over a matrix of python versions; rather we just need to make all relevant python versions available at once.

@MattShirley
Copy link
Collaborator

MattShirley commented Feb 3, 2026

When I deploy this I get:

/opt/servicex/.venv/lib/python3.14/site-packages/billiard/pool.py:1855: SyntaxWarning: 'return' in a 'finally' block
  return self._send_ack(
/opt/servicex/.venv/lib/python3.14/site-packages/billiard/connection.py:349: SyntaxWarning: 'return' in a 'finally' block
  return f
/opt/servicex/.venv/lib/python3.14/site-packages/billiard/connection.py:351: SyntaxWarning: 'return' in a 'finally' block
  return self._get_more_data(ov, maxsize)
/opt/servicex/.venv/lib/python3.14/site-packages/celery/concurrency/asynpool.py:400: SyntaxWarning: 'return' in a 'finally' block
  return remove(fd)
Traceback (most recent call last):
  File "/opt/servicex/.venv/bin/celery", line 8, in <module>
    sys.exit(main())
             ~~~~^^
  File "/opt/servicex/.venv/lib/python3.14/site-packages/celery/__main__.py", line 15, in main
    sys.exit(_main())
             ~~~~~^^
  File "/opt/servicex/.venv/lib/python3.14/site-packages/celery/bin/celery.py", line 236, in main
    return celery(auto_envvar_prefix="CELERY")
  File "/opt/servicex/.venv/lib/python3.14/site-packages/click/core.py", line 1157, in __call__
    return self.main(*args, **kwargs)
           ~~~~~~~~~^^^^^^^^^^^^^^^^^
  File "/opt/servicex/.venv/lib/python3.14/site-packages/click/core.py", line 1078, in main
    rv = self.invoke(ctx)
  File "/opt/servicex/.venv/lib/python3.14/site-packages/click/core.py", line 1688, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
                           ~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^
  File "/opt/servicex/.venv/lib/python3.14/site-packages/click/core.py", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
           ~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/servicex/.venv/lib/python3.14/site-packages/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
  File "/opt/servicex/.venv/lib/python3.14/site-packages/click/decorators.py", line 33, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/opt/servicex/.venv/lib/python3.14/site-packages/celery/bin/base.py", line 135, in caller
    return f(ctx, *args, **kwargs)
  File "/opt/servicex/.venv/lib/python3.14/site-packages/celery/bin/worker.py", line 348, in worker
    worker = app.Worker(
        hostname=hostname, pool_cls=pool_cls, loglevel=loglevel,
    ...<4 lines>...
        quiet=ctx.obj.quiet,
        **kwargs)
  File "/opt/servicex/.venv/lib/python3.14/site-packages/celery/worker/worker.py", line 98, in __init__
    self.setup_instance(**self.prepare_args(**kwargs))
    ~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/servicex/.venv/lib/python3.14/site-packages/celery/worker/worker.py", line 131, in setup_instance
    self.on_init_blueprint()
    ~~~~~~~~~~~~~~~~~~~~~~^^
  File "/opt/servicex/.venv/lib/python3.14/site-packages/celery/apps/worker.py", line 116, in on_init_blueprint
    self._custom_logging = self.setup_logging()
                           ~~~~~~~~~~~~~~~~~~^^
  File "/opt/servicex/.venv/lib/python3.14/site-packages/celery/apps/worker.py", line 180, in setup_logging
    return self.app.log.setup(
           ~~~~~~~~~~~~~~~~~~^
        self.loglevel, self.logfile,
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        redirect_stdouts=False, colorize=colorize, hostname=self.hostname,
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    )
    ^
  File "/opt/servicex/.venv/lib/python3.14/site-packages/celery/app/log.py", line 63, in setup
    handled = self.setup_logging_subsystem(
        loglevel, logfile, colorize=colorize, hostname=hostname,
    )
  File "/opt/servicex/.venv/lib/python3.14/site-packages/celery/app/log.py", line 118, in setup_logging_subsystem
    get_multiprocessing_logger(),
    ~~~~~~~~~~~~~~~~~~~~~~~~~~^^
  File "/opt/servicex/.venv/lib/python3.14/site-packages/celery/utils/log.py", line 270, in get_multiprocessing_logger
    return util.get_logger()
           ~~~~~~~~~~~~~~~^^
  File "/opt/servicex/.venv/lib/python3.14/site-packages/billiard/util.py", line 131, in get_logger
    logging._acquireLock()
    ^^^^^^^^^^^^^^^^^^^^
AttributeError: module 'logging' has no attribute '_acquireLock'

I also get this error when setting the image tag to atlasopenmagic.

@ponyisi
Copy link
Collaborator Author

ponyisi commented Feb 9, 2026

@BenGalewsky @MattShirley logging issues should be fixed by latest push

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Create ATLASMagic DID Finder

3 participants