docs: Add DiracX tasks architecture explanation #714
Conversation
chrisburr
force-pushed
the
docs/tasks-architecture
branch
from
a0c73b0 to
fb198da
Compare
|
|
||
| ### Sizing | ||
|
|
||
| Currently we foresee the need for three sizes of tasks: |
There was a problem hiding this comment.
Do you plan a task system to be aware of these categories?
| - **Standalone tasks:** The task performs its work and then returns. For example synchronising the IAM to DiracX configuration queries IAM and then updates the DiracX CS. | ||
| - **Spawning tasks:** The task creates additional tasks. For example, many cron-triggered tasks for transformations will spawn a task per transformation. | ||
| - **Batching tasks:** This task groups together the work of many smaller tasks. For example, it is more efficient to clean many jobs at once so many individual cleaning tasks are batched together for execution. | ||
| - **Callback tasks:** These tasks run in response to several parent tasks finishing successfully. Removing a user from the configuration store after deleting all of the objects owned by them. |
There was a problem hiding this comment.
I am not entirely sure to understand the Callback tasks concept.
IIUC, you would have 2-3 independent tasks finishing successfully and that would spawn a "child" task, is that correct?
So the main difference between Spawning tasks and Callback tasks is:
- A
Spawning taskspawns a new task by itself - A
Callback taskis spawned by the task queue system itself monitoring the tasks.
There was a problem hiding this comment.
Functionally they're very similar, the difference is that the callback needs to make sure it only runs once.
|
|
||
| Occasionally tasks must be scheduled to run at some point in the future, most commonly to retry a task after a delay. For example if a resource is banned corresponding tasks can use an exponential backoff strategy to avoid running excessively. | ||
|
|
||
| ## What is a task? |
There was a problem hiding this comment.
Minor suggestion but you define triggering mechanism the same way as the type of tasks. For instance:
- scheduled tasks
- spawning tasks
I would potentially move the What is a task? section defining the type of tasks before the triggering mechanism to define:
- standalone task
- spawning task
- batching task
- callback task (I wonder if it's a type of task, it seems to be a triggering mechanism like
reactiveto me but I am not sure to understand it correctly)
Then I would explain how they can be triggered:
- cron-based triggering
- reactive triggering (include callback?)
- scheduled triggering
| - **Medium:** Medium memory, medium CPU. Single task per thread. For example, bulk submitting jobs for a transformation. | ||
| - **Large:** Large amounts of memory, multiple CPU cores. Single task per thread. For example, generating reporting data. | ||
|
|
||
| Tasks should be designed to be lightweight, especially in terms of memory usage, using techniques such as streaming database responses. The vast majority of tasks should be in the **Small** category. The **Large** tasks should be reserved for infrequent, time insensitive activity (e.g. reporting). |
There was a problem hiding this comment.
I imagine the size of a task is defined statically and not dynamically based on input data for instance?
| ## What is a task? | ||
|
|
||
| Tasks are async Python functions which have extremely low overhead, allowing for many tasks to be spawned for even cheap operations. | ||
| Tasks can be executed in four different ways: |
There was a problem hiding this comment.
Are these concepts mutually exclusive?
Conceptually:
- it seems that a standalone task cannot be anything else.
- A batching task could also be a spawning task (I don't see any use case but we might want to decide whether we allow such a combination).
There was a problem hiding this comment.
They're not mutually exclusive.
| - **Realtime:** Tasks which are expected to be executed immediately. If there is ever a backlog of tasks in this queue the available worker pool is likely undersized for the installation. Examples of realtime tasks would be user input in DiracX Web or running the job optimizers. | ||
| - **Normal:** Tasks which should generally run immediately but for which there is a less strict latency requirement. If there is regularly a backlog in this queue the available worker pool is likely undersized for the installation. Examples include submitting jobs for transformations or periodically polling external services. | ||
| - **Background:** Tasks which have no specific latency requirements. Having a backlog of tasks in response to operational activity is expected and not directly indicative of an issue provided the backlog is not growing without bound. Examples include pending requests for jobs or data management activities from the transformation system. |
There was a problem hiding this comment.
Well here I am sure the priorities are mutually exclusive
|
|
||
| Several levels of locking can be applied to support the reentrancy requirements of tasks: | ||
|
|
||
| - **Task level locking:** Tasks can be configured to prevent simultaneous execution. For example, most cron-style tasks are configured to ensure only a single instance is running at any given time. |
There was a problem hiding this comment.
If a given task tries to acquire a lock that is already held, I guess it follows the "soft failure" mode and is retried later, right?
|
|
||
| ## Broker | ||
|
|
||
| The state of the broker should be ephemeral and recreated with each update. Any persistent state should be stored in the standard MySQL database's used by DiracX. This requirement is imposed to: |
There was a problem hiding this comment.
Very minor suggestion (because it could be another SQL implementation in the future + this document is focused on definitions/requirements rather than implementation).
| The state of the broker should be ephemeral and recreated with each update. Any persistent state should be stored in the standard MySQL database's used by DiracX. This requirement is imposed to: | |
| The state of the broker should be ephemeral and recreated with each update. Any persistent state should be stored in the standard SQL database's used by DiracX. This requirement is imposed to: |
| - Reduce the complexity of reasoning about updates which may change details of the broker's internal state. | ||
| - Improve performance by removing the need to ensure every action is flushed to persistent storage. | ||
|
|
||
| Upon first start, the broker is populated with the cron-style tasks as well as any pending reactive tasks that have been persisted in MySQL. The tasks which were persisted are those eligible for the dead letter queue. |
There was a problem hiding this comment.
Same here?
| Upon first start, the broker is populated with the cron-style tasks as well as any pending reactive tasks that have been persisted in MySQL. The tasks which were persisted are those eligible for the dead letter queue. | |
| Upon first start, the broker is populated with the cron-style tasks as well as any pending reactive tasks that have been persisted in SQL. The tasks which were persisted are those eligible for the dead letter queue. |
|
|
||
| ### Resource utilisation | ||
|
|
||
| To ensure the stability of the system, all workers should be configured to enforce memory and CPU limits. If these limits are exceeded the DiracX task system must be able to detect and report the issue, however in the case of small workers such detection may be unreliable due to technical constraints in async environments. |
There was a problem hiding this comment.
Are workers organized into separate pools based on resource capacity?
Or do they have identical resource allocation sized for large tasks?
How do we plan to connect task queues to workers?
3 participants
This PR contains the requirements we have for DiracX tasks and is intentionally separate from the implementation. Me at @chaen think this can be made to cover all of the use cases DiracX will have but we're open to feedback in case we missed something.