Remove background thread

[marcelldls]

Currently the backend creates a background thread and run_coroutine_threadsafe is used as the mechanism to schedule work in this background eventloop.

Moving to a single eventloop has the following benefits:

• Solving the An error occurred: <asyncio.locks.Lock object at 0x7fa65fa5e450 [locked]> is bound to a different event loop which can occur on put requests
• An ability to run multiple transports simultanously from a single thread
• An expected performance increase
• Dropping the inclusion of softioc AsyncioDispatcher which is interdependency between transports

To solve:

• fixing the tests
• adding some kind of performance testing?
• serving an ioc terminal shell?

@GDYendell Creating a draft in-case this needs to be picked up but also for some thoughts?

[marcelldls]

I did have a go testing a Rest API + IOC (mutli transport FastCS device) in a multi-threaded implementation (https://github.com/DiamondLightSource/FastCS/tree/multi-trans-threaded) vs a single event loop - against a serial device with a single client:

Single eventloop:

---------------------------------------------- benchmark 'test-ca-get': 1 tests ---------------------------------------------
Name (time in us)          Min         Max      Mean    StdDev    Median      IQR  Outliers  OPS (Kops/s)  Rounds  Iterations
-----------------------------------------------------------------------------------------------------------------------------
test_ca_get           165.0540  1,920.9450  226.4467  199.2780  179.1805  53.4310       3;6        4.4161      86           1
-----------------------------------------------------------------------------------------------------------------------------

--------------------------------------------- benchmark 'test-ca-put': 1 tests ---------------------------------------------
Name (time in us)          Min       Max      Mean    StdDev    Median       IQR  Outliers  OPS (Kops/s)  Rounds  Iterations
----------------------------------------------------------------------------------------------------------------------------
test_ca_put           355.1140  880.8100  468.5281  109.6180  434.3490  116.5380      18;6        2.1343      88           1
----------------------------------------------------------------------------------------------------------------------------

------------------------------------ benchmark 'test-rest-get': 1 tests ------------------------------------
Name (time in ms)        Min     Max    Mean  StdDev  Median     IQR  Outliers       OPS  Rounds  Iterations
------------------------------------------------------------------------------------------------------------
test_rest_get         1.1292  4.7000  1.4125  0.2761  1.3594  0.2187     31;19  707.9836     429           1
------------------------------------------------------------------------------------------------------------

------------------------------------- benchmark 'test-rest-put': 1 tests ------------------------------------
Name (time in ms)        Min      Max    Mean  StdDev  Median     IQR  Outliers       OPS  Rounds  Iterations
-------------------------------------------------------------------------------------------------------------
test_rest_put         1.3367  23.7540  1.8726  1.6800  1.6514  0.2766      7;20  534.0216     405           1
-------------------------------------------------------------------------------------------------------------

Multithreaded:

---------------------------------------------- benchmark 'test-ca-get': 1 tests ----------------------------------------------
Name (time in us)          Min          Max      Mean    StdDev    Median      IQR  Outliers  OPS (Kops/s)  Rounds  Iterations
------------------------------------------------------------------------------------------------------------------------------
test_ca_get           160.9580  17,599.5540  205.0493  287.7912  171.5760  49.5490    24;412        4.8769    6194           1
------------------------------------------------------------------------------------------------------------------------------

----------------------------------------------- benchmark 'test-ca-put': 1 tests ----------------------------------------------
Name (time in us)          Min          Max      Mean    StdDev    Median       IQR  Outliers  OPS (Kops/s)  Rounds  Iterations
-------------------------------------------------------------------------------------------------------------------------------
test_ca_put           343.7850  37,594.9940  502.2014  945.2674  445.8790  123.6257    15;101        1.9912    2887           1
-------------------------------------------------------------------------------------------------------------------------------

------------------------------------ benchmark 'test-rest-get': 1 tests ------------------------------------
Name (time in ms)        Min     Max    Mean  StdDev  Median     IQR  Outliers       OPS  Rounds  Iterations
------------------------------------------------------------------------------------------------------------
test_rest_get         1.0128  4.2176  1.2461  0.1763  1.1997  0.1706    163;34  802.5187     995           1
------------------------------------------------------------------------------------------------------------

------------------------------------ benchmark 'test-rest-put': 1 tests ------------------------------------
Name (time in ms)        Min     Max    Mean  StdDev  Median     IQR  Outliers       OPS  Rounds  Iterations
------------------------------------------------------------------------------------------------------------
test_rest_put         1.3639  9.2658  1.7262  0.6479  1.6370  0.2226      9;14  579.3186     730           1
------------------------------------------------------------------------------------------------------------

I ran the tests a few times and they are consistent, on my machine. It seems that the multithreaded approach does add significant tail latency to "ca clients" which supports the approach in this PR

Also, I found no significant difference for a single client when adding additional transports

[GDYendell]

I haven't looked at the changes yet, but nice work with the benchmarking, that is really useful.

[marcelldls]

I haven't looked at the changes yet, but nice work with the benchmarking, that is really useful.

Still needs some work. Im not sure I see this as becoming part of the test suite but it might be useful in dev or to track drift

[codecov]

Codecov Report

Attention: Patch coverage is 80.90909% with 21 lines in your changes missing coverage. Please review.

Project coverage is 87.77%. Comparing base (cf891fe) to head (27acd34).
Report is 1 commits behind head on main.

Files with missing lines	Patch %	Lines
src/fastcs/launch.py	82.35%	6 Missing ⚠️
src/fastcs/transport/graphQL/graphQL.py	25.00%	3 Missing ⚠️
src/fastcs/transport/tango/adapter.py	66.66%	3 Missing ⚠️
src/fastcs/transport/graphQL/adapter.py	66.66%	2 Missing ⚠️
src/fastcs/transport/rest/adapter.py	66.66%	2 Missing ⚠️
src/fastcs/transport/rest/rest.py	33.33%	2 Missing ⚠️
src/fastcs/transport/tango/dsr.py	84.61%	2 Missing ⚠️
src/fastcs/transport/adapter.py	83.33%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main      #98      +/-   ##
==========================================
- Coverage   88.08%   87.77%   -0.32%     
==========================================
  Files          31       31              
  Lines        1309     1341      +32     
==========================================
+ Hits         1153     1177      +24     
- Misses        156      164       +8     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[marcelldls]

Here are the latest benchmarks (running EPICS + REST + Tango together):

----------------------------------------------------- benchmark 'test-ca': 4 tests ----------------------------------------------------
Name (time in us)                   Mean                 Min                    Max            Outliers  OPS (Kops/s)            Rounds
---------------------------------------------------------------------------------------------------------------------------------------
test_ca_get                     220.8249 (1.0)      166.3830 (1.0)       1,047.8750 (1.25)          5;5        4.5285 (1.0)          83
test_ca_get_loaded_baseline     236.3295 (1.07)     173.0630 (1.04)      2,516.3090 (3.00)         3;29        4.2314 (0.93)        567
test_ca_put                     516.5574 (2.34)     400.6250 (2.41)        838.4920 (1.0)          19;2        1.9359 (0.43)         86
test_ca_get_loaded_request      696.0231 (3.15)     195.4340 (1.17)     32,072.8860 (38.25)       12;63        1.4367 (0.32)        224
---------------------------------------------------------------------------------------------------------------------------------------

---------------------------------------------- benchmark 'test-rest': 4 tests ----------------------------------------------
Name (time in ms)                   Mean               Min               Max            Outliers       OPS            Rounds
----------------------------------------------------------------------------------------------------------------------------
test_rest_get_loaded_baseline     1.9697 (1.0)      1.6496 (1.0)      4.8783 (1.44)        21;16  507.7032 (1.0)         270
test_rest_get                     2.0336 (1.03)     1.7746 (1.08)     3.3918 (1.0)         42;24  491.7436 (0.97)        328
test_rest_put                     2.1435 (1.09)     1.8912 (1.15)     5.2598 (1.55)        18;19  466.5302 (0.92)        297
test_rest_get_loaded_request      4.5069 (2.29)     1.9771 (1.20)     9.9291 (2.93)         29;4  221.8820 (0.44)        136
----------------------------------------------------------------------------------------------------------------------------

---------------------------------------------------- benchmark 'test-tango': 4 tests -----------------------------------------------------
Name (time in us)                        Mean                 Min                    Max            Outliers         OPS            Rounds
------------------------------------------------------------------------------------------------------------------------------------------
test_tango_get                       710.8007 (1.0)      574.7050 (1.0)       1,207.7270 (1.0)        124;31  1,406.8641 (1.0)         680
test_tango_get_loaded_baseline       783.0947 (1.10)     673.3800 (1.17)     25,766.2440 (21.33)        3;65  1,276.9847 (0.91)        832
test_tango_put                       875.9329 (1.23)     635.3760 (1.11)      1,737.2210 (1.44)        315;4  1,141.6399 (0.81)        859
test_tango_get_loaded_request      4,734.4724 (6.66)     657.0640 (1.14)     17,598.7140 (14.57)        33;2    211.2168 (0.15)        125
------------------------------------------------------------------------------------------------------------------------------------------

I believe Tango suffers under loading since we are having to run it in a separate thread as it wont take a running event loop

[marcelldls]

@GDYendell ready

[GDYendell]

@marcelldls there is a conflict after the other merges.

[marcelldls]

@GDYendell Looking a bit neater now?