Assembly loading move to AssemblyLoadContext for net8 and higher

[SimonCropp]

Version 6 changes the assembly loading behavior to using the new AssemblyLoadContext when executing under net8 and higher.

This can have some subtle effects on how types are compared. For example if assemblies are being loaded via Assembly.LoadFrom or AssemblyLoadContext.Default then those assemblies (and the contained types) will not be comparable to the assemblies and types loaded using nunits AssemblyLoadContext.

This will often manifest in plugin based frameworks where assemblies on disk as scanned for known types

For example the below will pass on version 5.2 but fail on version 6.0

var assembly = Assembly.LoadFrom(file);
var type = assembly.GetType("Class1");
ClassicAssert.IsTrue(typeof(Class1) == type);

The fix is to move assembly loading over to using either:

• nunits AssemblyLoadContext:

// can be cached statically
var context = AssemblyLoadContext.GetLoadContext(Assembly.GetExecutingAssembly())!;
var assembly = context.LoadFromAssemblyPath(file);

• or combining AssemblyName.GetAssemblyName with Assembly.Load

var name = AssemblyName.GetAssemblyName(file);
var assembly = Assembly.Load(name);

Tests

//All these work on V5.2
[TestFixture]
public class Tests
{
    string file = Path.GetFullPath("ClassLibrary.dll");

    //fails on 6.0
    [Test]
    public void WithLoadFrom()
    {
        var assembly = Assembly.LoadFrom(file);
        var type = assembly.GetType("Class1");
        ClassicAssert.IsTrue(typeof(Class1) == type);
    }

    [Test]
    public void WithGetAssemblyName()
    {
        var name = AssemblyName.GetAssemblyName(file);
        var assembly = Assembly.Load(name);
        var type = assembly.GetType("Class1");
        ClassicAssert.IsTrue(typeof(Class1) == type);
    }

    [Test]
    public void WithGetLoadContext()
    {
        var context = AssemblyLoadContext.GetLoadContext(Assembly.GetExecutingAssembly())!;
        var assembly = context.LoadFromAssemblyPath(file);
        var type = assembly.GetType("Class1");
        ClassicAssert.IsTrue(typeof(Class1) == type);
    }

    //fails on 6.0
    [Test]
    public void WithDefaultContext()
    {
        var context = AssemblyLoadContext.Default!;
        var assembly = context.LoadFromAssemblyPath(file);
        var type = assembly.GetType("Class1");
        ClassicAssert.IsTrue(typeof(Class1) == type);
    }
}

Full Repro

https://github.com/SimonCropp/NunitAdaptorAssemblyLoadingRepro

[SimonCropp]

my assumption is that this is an undocumented breaking change. so i am not expecting a fix.

this issue is intended as a reference issue for the release notes.

corresponding release notes PR: nunit/docs#1116

if my assumption is incorrect, and a code change can be made to restore the v5 behavior, then this can be re-purposed or closed

[OsirisTerje]

Thanks @SimonCropp !

@CharliePoole I assume this change is in the Engine, since we now use version 3.21.0.

Adapter version 4.6.0 - 5.2.0 used engine 3.18.1

[CharliePoole]

@OsirisTerje This is essentially the behavior we observed with nunit/nunit3-vs-adapter#1348, which was closed without action. AwsomeAssertions was attempting to load assemblies in the default AssemlyLoadContext, contrary to the engine's requirement of loading it in our own AssemblyLoadContext. This isn't a behavior we can readily change in V3, which allows direct user control over whether or not a separate process (also, for .NET Framework a separate AppDomain) in running tests. I'd like to revisit this in V4, so I'll transfer this one to the engine and tag it as V4.

@SimonCropp Your four test cases make a nice summary of the current status, I'll add them to my tests.

@OsirisTerje One ironic note, since this is arising in the adapter, is that we could probably safely use the default context in that case, since we believe that VS takes care of loading things for us anyway. We may want to experiment with this approach or even with not using the engine at all for your V7.

[Methuselah96]

I'm coming here after trying to upgrade NUnit3TestAdapter to v6. Below are some test cases that are passed for me when using NUnit3TestAdapter v5, but are now failing for me with NUnit3TestAdapter v6. I'm trying to use CSScript and I'm not sure of a workaround. Is this not considered a bug?

using CSScriptLib;
using Microsoft.CodeAnalysis.CSharp.Scripting;
using Microsoft.CodeAnalysis.Scripting;

namespace TestCase;

public class Point
{
    public int X { get; set; }
    public int Y { get; set; }
}

[TestFixture]
public class DynamicEvaluationTests
{
    [Test]
    public void TestCSScript()
    {
        dynamic script = CSScript.Evaluator.LoadCode(
            @"using TestCase;
             public class Script
             {
                 public Point CreatePoint(int x, int y)
                 {
                     return new Point { X = x, Y = y };
                 }
             }"
        );
        Point result = script.CreatePoint(1, 2);
        result.X.ShouldBe(1);
        result.Y.ShouldBe(2);
    }

    [Test]
    public async Task TestRoslyn()
    {
        var result = await CSharpScript.EvaluateAsync<Point>(
            "new Point { X = 1, Y = 2 }",
            ScriptOptions.Default.WithReferences(typeof(Point).Assembly).WithImports("TestCase")
        );
        result.X.ShouldBe(1);
        result.Y.ShouldBe(2);
    }
}

`TestCSScript` exception

Microsoft.CSharp.RuntimeBinder.RuntimeBinderException : Cannot implicitly convert type 'TestCase.Point' to 'TestCase.Point'
at CallSite.Target(Closure, CallSite, Object)
at System.Dynamic.UpdateDelegates.UpdateAndExecute1[T0,TRet](CallSite site, T0 arg0)
at TestCase.DynamicEvaluationTests.TestCSScript() (file:///Users/nathanbierema/Documents/ParagonCore/servers/libraries/GeometryLibrary/GeometryLibraryTests/LoggingTests.cs#L29,0)
at System.Reflection.MethodBaseInvoker.InterpretedInvoke_Method(Object obj, IntPtr* args)
at System.Reflection.MethodBaseInvoker.InvokeWithNoArgs(Object obj, BindingFlags invokeAttr)
at CallSite.Target(Closure, CallSite, Object)
at System.Dynamic.UpdateDelegates.UpdateAndExecute1[T0,TRet](CallSite site, T0 arg0)
at TestCase.DynamicEvaluationTests.TestCSScript() (file:///Users/nathanbierema/Documents/ParagonCore/servers/libraries/GeometryLibrary/GeometryLibraryTests/LoggingTests.cs#L29,0)
at System.Reflection.MethodBaseInvoker.InterpretedInvoke_Method(Object obj, IntPtr* args)
at System.Reflection.MethodBaseInvoker.InvokeWithNoArgs(Object obj, BindingFlags invokeAttr)

`TestRoslyn` exception

System.ArgumentException : Cannot bind to the target method because its signature is not compatible with that of the delegate type.
at System.Reflection.RuntimeMethodInfo.CreateDelegateInternal(Type delegateType, Object firstArgument, DelegateBindingFlags bindingFlags)
at System.Reflection.MethodInfo.CreateDelegate[T]()
at Microsoft.CodeAnalysis.Scripting.ScriptBuilder.Build[T](Compilation compilation, DiagnosticBag diagnostics, Boolean emitDebugInformation, CancellationToken cancellationToken)
at Microsoft.CodeAnalysis.Scripting.ScriptBuilder.CreateExecutor[T](ScriptCompiler compiler, Compilation compilation, Boolean emitDebugInformation, CancellationToken cancellationToken)
at Microsoft.CodeAnalysis.Scripting.Script`1.GetExecutor(CancellationToken cancellationToken)
at Microsoft.CodeAnalysis.Scripting.Script`1.RunAsync(Object globals, Func`2 catchException, CancellationToken cancellationToken)
at Microsoft.CodeAnalysis.Scripting.Script`1.RunAsync(Object globals, CancellationToken cancellationToken)
at Microsoft.CodeAnalysis.CSharp.Scripting.CSharpScript.RunAsync[T](String code, ScriptOptions options, Object globals, Type globalsType, CancellationToken cancellationToken)
at Microsoft.CodeAnalysis.CSharp.Scripting.CSharpScript.EvaluateAsync[T](String code, ScriptOptions options, Object globals, Type globalsType, CancellationToken cancellationToken)
at TestCase.DynamicEvaluationTests.TestRoslyn() (file:///Users/nathanbierema/Documents/ParagonCore/servers/libraries/GeometryLibrary/GeometryLibraryTests/LoggingTests.cs#L37,0)
at NUnit.Framework.Internal.TaskAwaitAdapter.GenericAdapter`1.GetResult()
at NUnit.Framework.Internal.AsyncToSyncAdapter.Await[TResult](TestExecutionContext context, Func`1 invoke)
at NUnit.Framework.Internal.AsyncToSyncAdapter.Await(TestExecutionContext context, Func`1 invoke)
at NUnit.Framework.Internal.Commands.TestMethodCommand.RunTestMethod(TestExecutionContext context)
at NUnit.Framework.Internal.Commands.TestMethodCommand.Execute(TestExecutionContext context)
at NUnit.Framework.Internal.Execution.SimpleWorkItem.<>c__DisplayClass3_0.<PerformWork>b__0()
at NUnit.Framework.Internal.ContextUtils.<>c__DisplayClass1_0`1.<DoIsolated>b__0(Object _)
at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
at NUnit.Framework.Internal.ContextUtils.DoIsolated(ContextCallback callback, Object state)
at NUnit.Framework.Internal.ContextUtils.DoIsolated[T](Func`1 func)
at NUnit.Framework.Internal.Execution.SimpleWorkItem.PerformWork()
at System.Reflection.RuntimeMethodInfo.CreateDelegateInternal(Type delegateType, Object firstArgument, DelegateBindingFlags bindingFlags)
at System.Reflection.MethodInfo.CreateDelegate[T]()
at Microsoft.CodeAnalysis.Scripting.ScriptBuilder.Build[T](Compilation compilation, DiagnosticBag diagnostics, Boolean emitDebugInformation, CancellationToken cancellationToken)
at Microsoft.CodeAnalysis.Scripting.ScriptBuilder.CreateExecutor[T](ScriptCompiler compiler, Compilation compilation, Boolean emitDebugInformation, CancellationToken cancellationToken)
at Microsoft.CodeAnalysis.Scripting.Script`1.GetExecutor(CancellationToken cancellationToken)
at Microsoft.CodeAnalysis.Scripting.Script`1.RunAsync(Object globals, Func`2 catchException, CancellationToken cancellationToken)
at Microsoft.CodeAnalysis.Scripting.Script`1.RunAsync(Object globals, CancellationToken cancellationToken)
at Microsoft.CodeAnalysis.CSharp.Scripting.CSharpScript.RunAsync[T](String code, ScriptOptions options, Object globals, Type globalsType, CancellationToken cancellationToken)
at Microsoft.CodeAnalysis.CSharp.Scripting.CSharpScript.EvaluateAsync[T](String code, ScriptOptions options, Object globals, Type globalsType, CancellationToken cancellationToken)
at TestCase.DynamicEvaluationTests.TestRoslyn() (file:///Users/nathanbierema/Documents/ParagonCore/servers/libraries/GeometryLibrary/GeometryLibraryTests/LoggingTests.cs#L37,0)
at NUnit.Framework.Internal.TaskAwaitAdapter.GenericAdapter`1.GetResult()
at NUnit.Framework.Internal.AsyncToSyncAdapter.Await[TResult](TestExecutionContext context, Func`1 invoke)
at NUnit.Framework.Internal.AsyncToSyncAdapter.Await(TestExecutionContext context, Func`1 invoke)
at NUnit.Framework.Internal.Commands.TestMethodCommand.RunTestMethod(TestExecutionContext context)
at NUnit.Framework.Internal.Commands.TestMethodCommand.Execute(TestExecutionContext context)
at NUnit.Framework.Internal.Execution.SimpleWorkItem.<>c__DisplayClass3_0.<PerformWork>b__0()
at NUnit.Framework.Internal.ContextUtils.<>c__DisplayClass1_0`1.<DoIsolated>b__0(Object _)
at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
at NUnit.Framework.Internal.ContextUtils.DoIsolated(ContextCallback callback, Object state)
at NUnit.Framework.Internal.ContextUtils.DoIsolated[T](Func`1 func)
at NUnit.Framework.Internal.Execution.SimpleWorkItem.PerformWork()

[CharliePoole]

@Methuselah96 In order to give your report the fullest attention, it will be helpful if you can report it separately, rather than piggybacking on a different bug with somewhat different circumstances. Here are a few guidelines for doing so...

If the bug occurs when using the console runner and/or engine 3.21.0 directly, report it in this repo. If it works when using the engine directly but not under NUnit3TestAdapter, then report it there.

Since you are using dynamic script evaluation, which is not something I've seen used in tests before, please indicate where it has worked before, i.e. what versions of the tools, to give us a starting point. This will be useful whether you report it as an engine bug or an adapter bug.

Your examples are great and will help either team in working on the issue.

[Methuselah96]

Okay, thanks, I've never used NUnit.Console and I'm getting different exceptions when I try to do so, so I opened an issue in the adapter repo here, although I still suspect it's related. For context, I piggybacked here since my issue seemed most similar to this closed issue and I didn't want to create a bunch of new issues if they were all likely related.

[OsirisTerje]

@Methuselah96 Don't worry about creating new issues. It is easier for us to organize when they are separate. Added this one to a parent issue under the adapter, so just create your own, and since you have repro code, can you create a repro project to go with it, and upload that - we have one more thing to check :-)