IDriverFactory extension does not yet work correctly

[rowo360]

Description
I made an attempt to activate a IDriverFactory extension using TestCentric-Gui 2.0.0-beta5 - unfortunately without success. While trying to get it activated, I searched for the missing detail and also looked into the TestCentric source code and log files. Finally I came to the conclusion that there's a gap in the TestCentric agent for this use case, so I raised this bug. But please review my attempt in general, if I'm on the right track.

Overview
The IDriverFactory extension is successfully detected by the TestCentric GUI. It's listed in the extension dialog with the state enabled. However the agent fails to load the IDriverFactory and there's an error message in the GUI while loading a test.

Exception callstack:

IDriverFactory implementation

• The DriverFactory implements interface TestCentric.Engine.Extensibility.IDriverFactory and is basically empty for the start
• It contains no path in the ExtensionAttribute as proposed by the exception message (because it doesn't make any difference)
• The resulting assembly is located in a folder 'MyTestCentricAddin'
• In the TestCentricGUI folder the '*.addin' file is extended with the path to the addin folder
  ../MyTestCentricAddin/*.dll
• With these steps the new addin is already successfully detected by the TestCentricGUI
• Next I added also an '*.addin' file for the agent. It's located in the folder:
  TestCentric.Extension.Net462PluggableAgent.2.5.0\tools\agent
  And contains a similar line:
  ../../../MyTestCentricAddin/*.dll

My assumption was that these steps would be sufficient to enable the extension also for the agent.

Observations
I'm adding some observations from analyzing this issue - it might be helpful.
I activated the detailed logging and compared the log file from the agent with the log file from the GUI:

Here's the essential part from the GUI log file:
21:45:39.493 Info [ 1] ServiceManager: Initializing ExtensionService
21:45:39.493 Info [ 1] ExtensionManager: Initializing ExtensionPoints...
21:45:39.493 Info [ 1] ExtensionManager: Assembly: testcentric.engine
21:45:39.493 Info [ 1] ExtensionManager: Assembly: TestCentric.Engine.Api
21:45:39.493 Info [ 1] ExtensionManager: Found Path=/TestCentric/Engine/TypeExtensions/ITestEventListener, Type=TestCentric.Engine.ITestEventListener
21:45:39.493 Info [ 1] ExtensionManager: Found Path=/TestCentric/Engine/TypeExtensions/IService, Type=TestCentric.Engine.Services.IService
21:45:39.493 Info [ 1] ExtensionManager: Found Path=/TestCentric/Engine/TypeExtensions/IAgentLauncher, Type=TestCentric.Engine.Extensibility.IAgentLauncher
21:45:39.493 Info [ 1] ExtensionManager: Found Path=/TestCentric/Engine/TypeExtensions/IDriverFactory,
...

And here's the log file from the agent:
21:45:40.497 Debug [ 5] DriverService: Creating ExtensionManager
21:45:40.497 Debug [ 5] DriverService: Initializing ExtensionManager
21:45:40.497 Info [ 5] ExtensionManager: Initializing ExtensionPoints...
21:45:40.497 Info [ 5] ExtensionManager: Assembly: TestCentric.Agent.Core
21:45:40.497 Info [ 5] ExtensionManager: Found Path=/NUnit/Engine/NUnitV2Driver, Type=TestCentric.Engine.Extensibility.IFrameworkDriver
21:45:40.497 Info [ 5] ExtensionManager: Initializing Extensions...
...

From my point of view the essential difference is the set of assemblies which are used by the ExtensionManager to get the list of available extension points. The GUI uses the assembly TestCentric.Engine.Api whereas the agent only uses the assembly TestCentric.Agent.Core. So overall the agent is not aware of the extension point IDriverFactory and cannot load the extension properly.

I also had a quick look into the code base and noticed the construction of the ExtensionManager here:
TestCentric.Agent.Core\Drivers\DriverService.cs
And yes indeed, here only the TestCentric.Agent.Core assemby is passed in.
As a final step I extended this line by passing in the TestCentric.Engine.Api assembly additionally and by that the extension gets successfuly loaded and also be invoked during test execution.
However I have no clue if this is a suitable fix at all and if it fits into the general concepts, it was just an attempt. But overall I hope that these observations help to narrow down this issue.

Environment

• TestCentric Version 2.0.0-beta5
• OS: Windows 11
• Target Runtime for your test assembly: .NET Framework 4.8

[CharliePoole]

You are correct. Most extensions are intended to be loaded by the engine itself in the same process as the GUI. The drive extension is currently the only exception. It really should not be considered an "engine" extension but an agent extension. In beta5, we don't currently yet that extension and I probably should have documented that somewhere.

From the information you give, I can't locate the code you modified. Are you working from the current master or the source code for the beta5 release? The code is constantly changing and the current master has changed a great deal since beta5. If you can post further details I'll comment further. That said, if such a simple change makes your extension load and run, I would suggest using it but watching out for changes in the future.

I'm renaming this issue for use in tracking the fact that the IDriverFactory extension does not yet work.

[rowo360]

Thanks for the explanation and the details about the IDriverFactory extension. So I seen now, that this extension point is a little different than the other extension points and it's OK to add special handling for it.

I started to work on beta5 release, but finally switched to the current master. This issue exists on both builds. However I was a little inaccurate by talking about the TestCentricGUI, because my fix is located in the project TestCentric.Agent.Core. And I would call it more a 'proof-of-concept': if we pass in the TestCentric.Engine.Api assembly to the ExtensionManager, the Manager is capable to extract the extension point IDriverFactory - that will work fine!

I'm working with screenshots now, because I did not manage to push to the server:
(File - TestCentric.Agent.Core\src\TestCentric.Agent.Core\Drivers\DriverService.cs)

Finally there's one more challenging question in my mind:
I actually started by implementing an extension using the NUnit interface: NUnit.Engine.Extensibility.IDriverFactory. But after some attempts I switched to use the TestCentric interface: TestCentric.Engine.Extensibility.IDriverFactory and got better progress with that one. But how's the long term vision? Is the TestCentric IDriverFactory only supported? Or will be NUnit IDriverFactory supported sometime in the future (for example: existing NUnit IDriverFactory implementations used with NUnit.ConsoleRunner)?

[CharliePoole]

TestCentric.Agent.Core is a common file shared by all pluggable TC agents, i.e. all TC agents as they are all pluggable. :-)

In beta5, only TestCentric agents are supported. In the current master, I'm working one at a time to get those agents defined by the NUnit api supported. Once that happens, I'll remove the TC api interface definitions.

As noted, the IDriverFactory extension point is unique because it isn't really used by the engine itself but only by agents. In future there could be more such extension points.

This could be a good area for you to work on but it's not quite ready to be worked. There are a few preliminary things to do, which I'll send your way separately.

[rowo360]

I'm adding some further comments about this bug which I notice while debugging. It's not with the intention to work on this issue right away - there's some code restructuring ongoing - but to help once this issue is picked up again:

The essential part is still the ExtensionManager in the DriverService class which fails to retrieve a IDriverFactory extension. The key difference to the Engine/GUI is that a new instance of the ExtensionManager class is created here without the help of the ExtensionService class. That's due to the fact that the Agent.Core is independent of the Engine and cannot access the existing/initialized ExtensionService.

So, one option might be to initialize the new ExtensionManager class properly (identically how the 'ExtensionService' do the initialization). Here's a code snippet:

extensionManager.FindExtensionPoints(ENGINE_ASSEMBLY, ENGINE_API_ASSEMBLY);
extensionManager.FindExtensions(ENGINE_DIRECTORY);

However this code doesn't work yet, it fails with an exception that the assembly "NUnit.Engine.API" cannot be found (Agent.Core process!)

Another option would be not to create a new ExtensionManager, but to use the existing, initialized one from the engine. For example by passing the IExtensionManager/IDriverFactory as an argument to the agent. Similar like this it's done for ITestEventListener:
TestEngineResult Run(ITestEventListener listener, TestFilter filter);

Passing it either as an argument in a method or in the constructor. But obviously this would be an API change which causes quite a lot adaptations.

[CharliePoole]

I'm in the process of making changes to the NUnit version of ExtensionManager, which will eventually reflect back to TestCentric. The new ExtensionManager is parameterized specifically for the purpose of allowing it to be used in various contexts, not just by the engine.

Eventually, I'd like to use the NUnit.Extensibility package in place of TestCentric.Extensibility, but I think the simplest intermediate step will be to first port these changes back to TestCentric and then work on this issue, or possibly do that as part of this issue.

It seems like I should move this one to TestCentric.Agent.Core unless you see a reason to keep it here.

[rowo360]

Yes, that's fine to move this issue to TestCentric.Agent.Core!
Or starting with the other idea: keep all issues in TestCentric.GUI centrally and add a TestCentric.Agent.Core label (or similar).

[CharliePoole]

I'm OK in starting out in TestCentric.Gui, but they have to end up in the correct repo when they are finished. That's because the software that generates release notes for GitHub looks in the repository where the release is done.

Moving them when I start work is optional but a bit cleaner since I don't have to remember to move them after they close.