Skip to content

Comments

optimize CEL rule evaluation performance#716

Open
YakirOren wants to merge 7 commits intomainfrom
optimize/http-event-creation
Open

optimize CEL rule evaluation performance#716
YakirOren wants to merge 7 commits intomainfrom
optimize/http-event-creation

Conversation

@YakirOren
Copy link
Contributor

@YakirOren YakirOren commented Feb 10, 2026

Summary

  • Reorder CreateEventFromRequest to check packet direction before expensive HTTP parsing
  • Eliminate full StructEvent copy in MakeHttpEvent by mutating in-place
  • Extract isPrivateIP helper, removing inline IIFE and redundant net.ParseIP closures
  • Replace map-based GetPacketDirection with switch statement

Summary by CodeRabbit

  • New Features

    • Added parse.basename() to extract filenames from paths.
    • New context-based evaluation methods for rules and expressions to support efficient repeated evaluations.
  • Refactor

    • Pre-computed expression indexing by event type to speed rule evaluation.
  • Tests

    • Added tests covering basename edge cases.
  • Chores

    • Ensure rules are initialized when registered or updated for consistent behavior.

@coderabbitai
Copy link

coderabbitai bot commented Feb 10, 2026

📝 Walkthrough

Walkthrough

Refactors CEL evaluation to separate context creation from evaluation, adds context-aware evaluation APIs, indexes rule expressions by event type, exposes parse.basename, updates rule manager to build and reuse evalContext, and adjusts rule factory to initialize rules.

Changes

Cohort / File(s) Summary
CEL Evaluation Core
pkg/rulemanager/cel/cel.go, pkg/rulemanager/cel/cel_interface.go
Made eval-context creation public (CreateEvalContext(utils.K8sEvent)), added context-based evaluation APIs (EvaluateRuleWithContext, EvaluateExpressionWithContext), kept wrapper methods for prior signatures, and extended the RuleEvaluator interface.
CEL Library Extensions
pkg/rulemanager/cel/libraries/parse/parse.go, pkg/rulemanager/cel/libraries/parse/parselib.go, pkg/rulemanager/cel/libraries/parse/parsing_test.go
Added parse.basename(string) -> string overload with implementation and cost estimate; added unit tests covering edge cases.
Rule Types & Initialization
pkg/rulemanager/types/v1/types.go
Added ExpressionsByEventType map[utils.EventType][]RuleExpression to Rule and Init() method to precompute/group expressions by event type (json/yaml excluded).
Rule Factory
pkg/rulemanager/rulecreator/factory.go
Ensure rules are initialized (Init()) when registered, synced, or updated.
Rule Manager / Evaluation Flow
pkg/rulemanager/rule_manager.go
Refactored evaluation to create a single evalContext per event and pass it to CEL context-based evaluators; use ExpressionsByEventType for per-event expressions; updated helper signatures to accept evalContext; removed obsolete per-event-type helpers.

Sequence Diagram(s)

mermaid
sequenceDiagram
participant RM as RuleManager
participant EV as EnrichedEvent
participant EC as EvalContext
participant CEL as CEL Evaluator
participant Prog as CEL Program/Cache
RM->>EV: receive event
RM->>EC: CreateEvalContext(event.Event)
RM->>RM: lookup rule.ExpressionsByEventType[eventType]
RM->>CEL: EvaluateRuleWithContext(evalContext, expressions)
CEL->>Prog: compile or fetch cached program
Prog->>CEL: program
CEL->>CEL: evaluate program against evalContext
CEL-->>RM: boolean result / error

Estimated code review effort

🎯 4 (Complex) | ⏱️ ~45 minutes

Possibly related PRs

Suggested reviewers

  • matthyx

Poem

🐰
I built a little context tree, all snug and neat,
One eval for many hops, no repeated feat.
Basename nibbles paths down to the core,
Rules grouped by event — I hop and explore! ✨

🚥 Pre-merge checks | ✅ 2 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (2 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title 'optimize CEL rule evaluation performance' directly aligns with the main changes: context-aware CEL evaluation refactoring, pre-indexing rules by event type, and new evaluation methods to eliminate per-event scanning.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing touches
  • 📝 Generate docstrings
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Post copyable unit tests in a comment
  • Commit unit tests in branch optimize/http-event-creation

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands and usage tips.

@YakirOren YakirOren changed the title optimize HTTP event creation optimize CEL rule evaluation performance Feb 12, 2026
@matthyx matthyx self-assigned this Feb 12, 2026
@YakirOren YakirOren force-pushed the optimize/http-event-creation branch from 4cca818 to 9a69a13 Compare February 12, 2026 17:23
@matthyx matthyx marked this pull request as ready for review February 13, 2026 16:01
Copy link

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Caution

Some comments are outside the diff and can’t be posted inline due to platform limitations.

⚠️ Outside diff range comments (1)
pkg/rulemanager/rulecreator/factory.go (1)

71-80: ⚠️ Potential issue | 🟠 Major

Pre-existing bug: CreateRulePolicyRulesByEventType appends to the slice it's iterating over.

This isn't part of the current PR changes, but this method iterates over rules and conditionally appends to rules in the same loop (line 75), which will cause duplicate entries and potentially an infinite loop (in Go, range evaluates the length once, so it won't be infinite, but duplicates will still be added).

Proposed fix
 func (r *RuleCreatorImpl) CreateRulePolicyRulesByEventType(eventType utils.EventType) []typesv1.Rule {
-	rules := r.CreateRulesByEventType(eventType)
-	for _, rule := range rules {
+	allRules := r.CreateRulesByEventType(eventType)
+	var rules []typesv1.Rule
+	for _, rule := range allRules {
 		if rule.SupportPolicy {
 			rules = append(rules, rule)
 		}
 	}
-
 	return rules
 }
🧹 Nitpick comments (4)
pkg/rulemanager/cel/libraries/parse/parse.go (1)

31-41: Consider using path.Base from the standard library.

The stdlib path.Base handles additional edge cases (e.g., trailing slashes, empty strings, "." paths) and is well-tested. Your current implementation returns "" for a trailing-slash input like "/usr/bin/", whereas path.Base would return "bin".

If the empty-string-on-trailing-slash behavior is intentional for your CEL rules, this is fine as-is — just worth documenting that choice. Otherwise, path.Base is a drop-in replacement here.

♻️ If stdlib behavior is acceptable
-import (
-	"strings"
-
-	"github.com/google/cel-go/common/types"
-	"github.com/google/cel-go/common/types/ref"
-	"github.com/kubescape/node-agent/pkg/rulemanager/cel/libraries/celparse"
-)
+import (
+	"path"
+
+	"github.com/google/cel-go/common/types"
+	"github.com/google/cel-go/common/types/ref"
+	"github.com/kubescape/node-agent/pkg/rulemanager/cel/libraries/celparse"
+)
 func (l *parseLibrary) basename(path ref.Val) ref.Val {
 	s, ok := path.Value().(string)
 	if !ok {
 		return types.MaybeNoSuchOverloadErr(path)
 	}
-	idx := strings.LastIndex(s, "/")
-	if idx == -1 {
-		return types.String(s)
-	}
-	return types.String(s[idx+1:])
+	return types.String(path.Base(s))
 }

Note: there would be a name collision between the path parameter and the path import — rename the parameter (e.g., val) if you go this route.

pkg/rulemanager/cel/libraries/parse/parsing_test.go (1)

45-64: Consider adding a test for the empty-string edge case.

An empty string input (parse.basename('')) would exercise the LastIndex == -1 branch and return "". It's a plausible real-world input (e.g., unset field) worth covering.

pkg/rulemanager/rule_manager.go (1)

373-386: getUniqueIdAndMessage: only the last error is returned.

If the message evaluation (line 374) fails and uniqueID evaluation (line 378) succeeds, err is overwritten to nil and the message error is silently lost. The function would return an empty message with no error indication. This appears to be a pre-existing pattern, but now that both calls use the same eval context, it's worth noting.

Optional: propagate the first error
 func (rm *RuleManager) getUniqueIdAndMessage(evalContext map[string]any, rule typesv1.Rule) (string, string, error) {
 	message, err := rm.celEvaluator.EvaluateExpressionWithContext(evalContext, rule.Expressions.Message)
 	if err != nil {
 		logger.L().Error("RuleManager - failed to evaluate message", helpers.Error(err))
+		return "", "", fmt.Errorf("failed to evaluate message: %w", err)
 	}
 	uniqueID, err := rm.celEvaluator.EvaluateExpressionWithContext(evalContext, rule.Expressions.UniqueID)
 	if err != nil {
 		logger.L().Error("RuleManager - failed to evaluate unique ID", helpers.Error(err))
+		return "", "", fmt.Errorf("failed to evaluate unique ID: %w", err)
 	}
pkg/rulemanager/types/v1/types.go (1)

36-48: Good design: pre-indexed expressions avoid per-event scanning.

The json:"-" yaml:"-" tags and the pointer-receiver Init() are correct. One subtle point: if Init() is ever not called, ExpressionsByEventType remains nil. In Go, indexing a nil map returns the zero value (empty slice) without panicking, so rules would be silently skipped rather than producing an error. The factory currently calls Init() in all mutation paths (RegisterRule, SyncRules, UpdateRule), ensuring all rules in circulation are initialized. However, CreateRuleByID and CreateRuleByName can return an empty Rule{} on failed lookup; consider adding a defensive nil-check in the consumer (rule_manager.go) to guard against future code paths.

Optional: lazy-init guard in the lookup

In rule_manager.go where you look up expressions:

+		if rule.ExpressionsByEventType == nil {
+			logger.L().Warning("Rule not initialized, skipping", helpers.String("rule", rule.ID))
+			continue
+		}
		ruleExpressions := rule.ExpressionsByEventType[eventType]
		if len(ruleExpressions) == 0 {

Copy link
Contributor

@matthyx matthyx left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

did you run a Bench to see the improvements? maybe we should commit it too...

Signed-off-by: Yakir Oren <yakiroren@gmail.com>
Signed-off-by: Yakir Oren <yakiroren@gmail.com>
Signed-off-by: Yakir Oren <yakiroren@gmail.com>
Signed-off-by: Yakir Oren <yakiroren@gmail.com>
Signed-off-by: Yakir Oren <yakiroren@gmail.com>
…n CEL evaluation

Signed-off-by: Yakir Oren <yakiroren@gmail.com>
Signed-off-by: Yakir Oren <yakiroren@gmail.com>
@YakirOren YakirOren force-pushed the optimize/http-event-creation branch from f65bd00 to 03d2b02 Compare February 17, 2026 16:54
Copy link

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Caution

Some comments are outside the diff and can’t be posted inline due to platform limitations.

⚠️ Outside diff range comments (1)
pkg/rulemanager/cel/cel.go (1)

132-154: ⚠️ Potential issue | 🟠 Major

Unchecked type assertion and discarded errors in CreateEvalContext.

Two concerns in this now-public method:

  1. Line 140: event.(utils.CelEvent) will panic if the event doesn't implement CelEvent and no converter is registered. Since this is now a public API (previously private), the risk surface is wider.
  2. Lines 138, 140: Both xcel.NewObject return values discard the error. If object creation fails, obj will be a zero value and downstream evaluation will produce confusing errors.

Consider using the comma-ok pattern for the type assertion and at least logging or returning an error from CreateEvalContext (which would require a signature change), or adding a defensive nil-check on obj.

Suggested defensive fix (minimal, no signature change)
 	if converter, exists := c.eventConverters[eventType]; exists {
-		obj, _ = xcel.NewObject(converter(event))
+		var err error
+		obj, err = xcel.NewObject(converter(event))
+		if err != nil {
+			logger.L().Warning("CEL: failed to create object from converted event", helpers.Error(err))
+		}
 	} else {
-		obj, _ = xcel.NewObject(event.(utils.CelEvent))
+		celEvent, ok := event.(utils.CelEvent)
+		if !ok {
+			logger.L().Warning("CEL: event does not implement CelEvent", helpers.String("eventType", string(eventType)))
+			return map[string]any{"eventType": string(eventType)}
+		}
+		var err error
+		obj, err = xcel.NewObject(celEvent)
+		if err != nil {
+			logger.L().Warning("CEL: failed to create object", helpers.Error(err))
+		}
 	}
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@pkg/rulemanager/cel/cel.go` around lines 132 - 154, The CreateEvalContext
function currently panics on a failed type assertion and discards errors from
xcel.NewObject; change it to use the comma-ok form when asserting
event.(utils.CelEvent) and capture the error returned by xcel.NewObject in both
branches. If the type assertion fails and no converter exists, fall back to
calling xcel.NewObject(event) (or set obj to event) but handle the returned
error: log it via the CEL receiver's logger (e.g., c.logger) or set obj to a
safe fallback (nil or the raw event) instead of ignoring the error; ensure you
also guard the "http" assignment by checking obj != nil. Optionally, for a
stricter fix, change CreateEvalContext signature to return (map[string]any,
error) and propagate xcel.NewObject errors upward. Use symbol references
CreateEvalContext, c.eventConverters, xcel.NewObject, utils.CelEvent, and the
"http" evalContext key to locate and update the code.
🧹 Nitpick comments (1)
pkg/rulemanager/rule_manager.go (1)

373-386: getUniqueIdAndMessage only returns the error from the UniqueID evaluation, silently dropping the message evaluation error.

Line 374 evaluates the message expression and logs the error, but err is then overwritten at line 378 by the uniqueID evaluation. If the message evaluation fails but the uniqueID evaluation succeeds, err will be nil and the caller won't know the message failed. This appears to be a pre-existing issue, but now is a good time to address it.

Suggested fix
 func (rm *RuleManager) getUniqueIdAndMessage(evalContext map[string]any, rule typesv1.Rule) (string, string, error) {
 	message, err := rm.celEvaluator.EvaluateExpressionWithContext(evalContext, rule.Expressions.Message)
 	if err != nil {
 		logger.L().Error("RuleManager - failed to evaluate message", helpers.Error(err))
+		return "", "", err
 	}
 	uniqueID, err := rm.celEvaluator.EvaluateExpressionWithContext(evalContext, rule.Expressions.UniqueID)
 	if err != nil {
 		logger.L().Error("RuleManager - failed to evaluate unique ID", helpers.Error(err))
 	}
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@pkg/rulemanager/rule_manager.go` around lines 373 - 386,
getUniqueIdAndMessage currently overwrites the error from evaluating the message
with the error from evaluating the UniqueID, potentially returning nil when the
message evaluation failed; update getUniqueIdAndMessage to capture both
evaluation errors from rm.celEvaluator.EvaluateExpressionWithContext (for
rule.Expressions.Message and rule.Expressions.UniqueID), log each failure as you
already do, and return a non-nil error if either evaluation failed (e.g., return
the first non-nil error or wrap both errors together) while still hashing
uniqueID via hashStringToMD5 before returning; ensure the returned err reflects
any failure from evaluating message or uniqueID so callers of
getUniqueIdAndMessage see the correct error.
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.

Outside diff comments:
In `@pkg/rulemanager/cel/cel.go`:
- Around line 132-154: The CreateEvalContext function currently panics on a
failed type assertion and discards errors from xcel.NewObject; change it to use
the comma-ok form when asserting event.(utils.CelEvent) and capture the error
returned by xcel.NewObject in both branches. If the type assertion fails and no
converter exists, fall back to calling xcel.NewObject(event) (or set obj to
event) but handle the returned error: log it via the CEL receiver's logger
(e.g., c.logger) or set obj to a safe fallback (nil or the raw event) instead of
ignoring the error; ensure you also guard the "http" assignment by checking obj
!= nil. Optionally, for a stricter fix, change CreateEvalContext signature to
return (map[string]any, error) and propagate xcel.NewObject errors upward. Use
symbol references CreateEvalContext, c.eventConverters, xcel.NewObject,
utils.CelEvent, and the "http" evalContext key to locate and update the code.

---

Nitpick comments:
In `@pkg/rulemanager/rule_manager.go`:
- Around line 373-386: getUniqueIdAndMessage currently overwrites the error from
evaluating the message with the error from evaluating the UniqueID, potentially
returning nil when the message evaluation failed; update getUniqueIdAndMessage
to capture both evaluation errors from
rm.celEvaluator.EvaluateExpressionWithContext (for rule.Expressions.Message and
rule.Expressions.UniqueID), log each failure as you already do, and return a
non-nil error if either evaluation failed (e.g., return the first non-nil error
or wrap both errors together) while still hashing uniqueID via hashStringToMD5
before returning; ensure the returned err reflects any failure from evaluating
message or uniqueID so callers of getUniqueIdAndMessage see the correct error.

@matthyx matthyx moved this to Needs Reviewer in KS PRs tracking Feb 23, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: Needs Reviewer

Development

Successfully merging this pull request may close these issues.

2 participants