generated from amazon-archives/__template_Apache-2.0
-
Notifications
You must be signed in to change notification settings - Fork 15
Open
Labels
enhancementNew feature or requestNew feature or request
Description
Problem Statement
Similar to Output-based evaluators( OutputEvaluator, TrajectoryEvaluator), the trace-based evaluators should be able to take expected_* parameters into considerations for evaluation.
Trace-based evaluators:
- faithfulness_evaluator
- goal_success_rate_evaluator.py
- harmfulness_evaluator.py
- helpfulness_evaluator.py
- response_relevance_evaluator.py
- tool_parameter_accuracy_evaluator.py
- tool_selection_accuracy_evaluator.py
Notes
- As of now,
expected_outputandexpected_trajectoryare the "expected similar" output, not the "exact" outputs. The criteria is further tuned in therubric. We need to consider what the expected behavior is for trace-based evaluators.
Proposed Solution
No response
Use Case
N/A
Alternatives Solutions
No response
Additional Context
No response
Reactions are currently unavailable
Metadata
Metadata
Assignees
Labels
enhancementNew feature or requestNew feature or request
Type
Projects
Status
Intake