feat: support 'same-as-agent' model option for legacy evaluators

[Chibionos]

Summary

• Add support for the same-as-agent model configuration in legacy LLM-based evaluators
• When an evaluator specifies same-as-agent as its model, it now resolves to the actual model from agent.json settings
• Added logging for model resolution and evaluator creation to improve visibility

Changes

• Updated EvaluatorFactory.create_evaluator() to accept agent_model parameter
• Updated _create_legacy_evaluator_internal() to pass agent_model to LLM-based evaluators
• Updated _create_legacy_llm_as_judge_evaluator() to resolve same-as-agent to actual model
• Updated _create_legacy_trajectory_evaluator() to resolve same-as-agent to actual model
• Added _get_agent_model() method to runtime to load model from agent.json
• Added INFO-level logging for model resolution and evaluator creation
• Fixed error message in trajectory evaluator (was incorrectly saying "LLM evaluator")

Test plan

• Tested with calculator_same_as_agent example containing evaluators with "model": "same-as-agent"
• Verified both LLM-as-judge and Trajectory evaluators resolve model correctly
• Verified logging shows model resolution and creation messages

🤖 Generated with Claude Code

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[akshaylive]

IMO it's better to do something like this:

from typing import Protocol, runtime_checkable


@runtime_checkable
class LLMAgentProtocol(Protocol):
    def get_agent_model(self) -> str:
        ...

And switch the implementation of _get_agent_model to:

def _get_agent_model(self, runtime: UiPathRuntimeProtocol) -> str | None:
    if isinstance(runtime, LLMAgentProtocol):
        return runtime.get_agent_model()
    else:
        return None

That way, react agent can implement that method and it should work seamlessly.

[Chibionos]

IMO it's better to do something like this:

from typing import Protocol, runtime_checkable


@runtime_checkable
class LLMAgentProtocol(Protocol):
    def get_agent_model(self) -> str:
        ...

And switch the implementation of _get_agent_model to:

def _get_agent_model(self, runtime: UiPathRuntimeProtocol) -> str | None:
    if isinstance(runtime, LLMAgentProtocol):
        return runtime.get_agent_model()
    else:
        return None

That way, react agent can implement that method and it should work seamlessly.

Good catch implemented your Protocol-based pattern.

[akshaylive]

I don't think the factory should contain the model. It's a runtime concept. I.e; the factory can create different runtimes with different LLMs based on the entrypoint, so I'd rather keep it in runtime. This is why runtime: UiPathRuntimeProtocol has to be piped everywhere but it should be doable, right?

Also, please create a unit test to test the change :)

[mjnovice]

nit: I would suggest passing in agent_model_settings object instead, so that @mathurk @AAgnihotry do not have to do double work :)

[mjnovice]

what happens when we specify a custom model settings cc - @mathurk @AAgnihotry

[mathurk]

lgtm