YAML Configuration Generation

[Relm-Arrowny]

Currently, the Bluesky beamline-specific device configurations are defined within a Python script located in the dodal package. Any modification to a device's name, PV prefix, or configuration parameters necessitates editing this Python file, committing the change to dodal, and performing a full software release cycle. This process is cumbersome, introduces unnecessary dependencies on core library releases for deployment configuration changes and complicates beamline commissioning or maintenance.

Goals

Acceptance criteria :

Configuration Structure & Separation

• Clear separation of concerns - separate beamline device definitions away from python codes that create the instance, dodal contain only device classes and no beamline-specific configuration (PVs, names, parameters)
• A clearly defined configuration layer - specific device properties declaratively in YAML. This includes the device's Python class, its PV prefix, instantiation arguments, configuration and initialisation parameters using property setter.
• Each device should be defined in its own YAML with one-to-one mapping, each logical device instance (e.g., i10apple2) must be defined in its own dedicated YAML file.
• An aggregation mechanism, such as an all.yml file, capable of easily including, importing, or temporarily excluding individual or specific device configuration files using comment mark for deployment.
• support object injection or wire-up in YAML file, i.e. a new object reuses existing objects like IOC.
• support property setter/getter after object instantiation

Instantiation & Consumption

• Develop a config loader or object creator utility that reads the aggregated configuration, dynamically resolves the Python class from dodal, and creates the instantiated device objects making them available to device manger.
• The implementation should not only limited to device hardware objects, it should support any object creation.
• The object creation process is a 2 stage process: 1st is object instantiation and registration with device manager, 2nd is property setter so different objects can be injected or wired-up together, i.e. IOC. The property setting should support both compulsory and optional property checking.

Deployment & Maintainability

• Remove dodal configuration dependences. Any change to a PV prefix or device parameter or adding a new device requires only updating the relevant YAML files and restarting the BlueAPI, with no modification or redeployment of dodal.
• Create clear documentation and a schema or example file for the YAML format, detailing the required structure, device types, and configuration parameters.

Tasks

base on the goals, deduce the tasks required. check them off once they're done

• Schema Definition: Finalize the devices.yaml format to support nested dependencie
• Handle Hierarchy: Define how to represent complex devices that contain sub-devices
• Class Mapping template: Build a Python template that maps YAML strings
• Instantiation Logic: Develop the "loader" that reads the YAML and instantiate the devices with the provided parameters
• Pydantic: Use Pydantic to validate the YAML file.
• Mock Mode Support: Enable a flag in the YAML for testing without beamline hardware.

Related Resources

any images/gifs, mockups/wireframes that would aid the development should be listed here
MockUp : link to mockup
Coda doc: link to coda
Additional context
Possible Structure:

beamline: i10
globals:
  config_server: "https://daq-config.diamond.ac.uk"
  lookup_dir: "/dls_sw/i10/software/gda/.../lookupTables/"

# References to individual device files
includes:
  - configs/mirrors.yaml
  - configs/id_downstream.yaml
  - configs/id_upstream.yaml
  - configs/slits.yaml

Each devices:

devices:
  - name: first_mirror
    type: PiezoMirror
    module: dodal.devices.i10
    args:
      prefix: "{PREFIX.beamline_prefix}-OP-COL-01:"

  - name: switching_mirror
    type: PiezoMirror
    module: dodal.devices.i10
    args:
      prefix: "{PREFIX.beamline_prefix}-OP-SWTCH-01:"
    properties:
      <property_name>: property_value

- name: pgm
    type: PlaneGratingMonochromator
    description: "I10 Plane Grating Monochromator"
    args:
      prefix: "{PREFIX.beamline_prefix}-OP-PGM-01:"
      grating: "I10Grating"
      grating_pv: "NLINES2"

[Relm-Arrowny]

@DiamondLightSource/smg please have a quick look and see if this capture what we wanted for configure file, feel free to edit the issue.

[Villtord]

so this yaml file is going to be a part of sm-bluesky package?

[Relm-Arrowny]

so this yaml file is going to be a part of sm-bluesky package?

I think the idea is, it will live here for now and I think we will have blueapi pick it up as an import(not 100% sure how exactly how it going to work)

[fajinyuan]

This will need the library for dependency injection at https://pypi.org/project/dependency-injector/

This is not required as device manager gives the same functions.

[fajinyuan]

You can find a good description on this at https://python-dependency-injector.ets-labs.org/introduction/di_in_python.html

[DominicOram]

Apologies to jump in. Undecided whether we would use this mechanism in MX or not but happy it's being worked on. There are some things that the current format gives you that it would be a shame to lose:

• dodal connect ixx will try and instantiate and connect all the devices associated with the beamline and print which ones can't connect. We've found this to be incredibly useful in catching things like typos in PV names etc. It would be good for this system to provide a similar utility. We may be able to do it as part of Create CI pipelines for offline testing of BlueAPI configuration blueapi#1206 though?
• At the moment, if I want to make a change to a dodal device it's very easy for me to see which beamlines use it and therefore work out who I need to discuss the change with to ensure compatibility. Moving these files into the xx-bluesky repos means that it's now a lot harder to find this out. I think as part of this we should make a utility for being able to do this search. Otherwise we will either end up with situations where people are afraid to make any changes in dodal for fear of breaking other people's beamlines or where people will accidentally break things without realising

[oliwenmandiamond]

I agree with Dom, it should remain in dodal so that configuration for beamlines is under one roof so easily searchable and know there's no breaking changes from a chain of repos. I think the idea was that we would experiment with playing around with this idea in our repo as not everyone was on board with this originally but seems the wind has changed on this.

I think it might be worth us moving this issue to dodal so that we can get wider input and discussion going. As discussed in our standup today, we don't think a library for dependency injection is needed because DeviceManager should manage all of the dependencies for us.

[fajinyuan]

Hi @DominicOram your comments on this is very welcomed. We would like everyone to add their comments to this. as @oliwenmandiamond already stated, this is in sm-bluesky just during developing and testing period. Once we developed a solution we will do a demo to other teams to see if this can be improved and moved to dodal so everyone can use it. The good news I was told this morning the new device manager already support Inversion Of Control.

This change or development must be backward compatible - that is to say it should continue to support 'dodal connect ixx' via object instantiation. We might have to add extra processes for this command to work.

[DominicOram]

Thanks @fajinyuan! My preference would be that if we think it will end up in dodal then we should do the development/testing in there too so that:

• Everyone can easily see the direction and help if needed
• It saves the job of having to at some point have a huge PR that we will have to collectively review and merge
• The history is more easily found, it's hopefully easier for later people to find out why decisions were made on the structure of the solution

We could maybe do it on a specific dev branch in dodal if that helps?

[Relm-Arrowny]

Seem like no one is unhappy this goes to dodal anymore, shall I move this issue there and get some more feedback before we make a start?

[fajinyuan]

Yes please @Relm-Arrowny