Avoid uninstalling and re-installing service components on policy change #11740

ycombinator · 2025-12-11T02:10:07Z

What does this PR do?

This PR identifies Service Runtime components with only their input type; the output ID is not longer used.

Why is it important?

Service Runtime components are intended to be kept running (via a service) for as long as possible. We should only install/start or uninstall/stop them if they are being explicitly added or removed, respectively, from the component model. If only their configuration is being updated, we should keep the component running.

If a component's ID changes between the last and current component models, Elastic Agent will ask the component's service to uninstall and then reinstall itself. Prior to this PR, service components' ID were determined by their input type and output ID. Therefore, if a service component's output were changed, it would cause the service to be uninstall and then reinstalled. This is undesirable behavior, as services should be kept running as long as possible.

With the changes in this PR, we no longer consider the output ID when generating service components' IDs. If a service component's output is changed, it's ID remains the same between the last and current component models. Elastic Agent does not uninstall and reinstall the component's service but simply passes the configuration change to it (which it was doing prior to this PR anyway).

Checklist

I have read and understood the pull request guidelines of this project.
My code follows the style guidelines of this project
I have commented my code, particularly in hard-to-understand areas
~~I have made corresponding changes to the documentation~~
~~I have made corresponding change to the default configuration files~~
I have added tests that prove my fix is effective or that my feature works
I have added an entry in ./changelog/fragments using the changelog tool
~~I have added an integration test or an E2E test~~

Disruptive User Impact

None.

How to test this PR locally

Policy reassign does not uninstall/reinstall Endpoint

Using the Fleet UI, create three Agent policies:
- default: containing only the system integration
- tp-es: containing the Elastic Defend integration, with tamper protection enabled, and using the Elasticsearch output.
- tp-ls: containing the Elastic Defend integration, with tamper protection enabled, and using the Logstash output. Note that you will need to create the Logstash output in Fleet > Settings.
Enroll an Elastic Agent in the tp-es policy and verify the agent is healthy and shipping data.
Assign the Agent to the tp-ls policy.
Check the Agent logs and make sure the Endpoint component is not being uninstalled and reinstalled. Concretely, check that there is no log entry for uninstall endpoint service.
Check the Endpoint logs (located under /opt/Elastic/Endpoint/state/log/ on Linux) and make sure that Endpoint has connected to Logstash (or has attempted to and failed if there is no actual Logstash endpoint listening).

Removing Endpoint from policy uninstalls Endpoint

Assign the Agent to the default policy.
Check the Agent logs and make sure the Endpoint component is stopped and uninstalled. Concretely, check that there is a log entry for stopping endpoint service runtime, followed by uninstall endpoint service, followed by Stopped: endpoint service runtime.

Related issues

Resolves Cannot successfully change output type or name in tamper protected agent polices that contain Elastic Defend #11266

Questions to ask yourself

How are we going to support this in production?
How are we going to measure its adoption?
How are we going to debug this?
What are the metrics I should take care of?
...

mergify · 2025-12-11T02:10:47Z

This pull request does not have a backport label. Could you fix it @ycombinator? 🙏
To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

backport-./d./d is the label that automatically backports to the 8./d branch. /d is the digit
backport-active-all is the label that automatically backports to all active branches.
backport-active-8 is the label that automatically backports to all active minor branches for the 8 major.
backport-active-9 is the label that automatically backports to all active minor branches for the 9 major.

elasticmachine · 2025-12-12T14:54:58Z

Pinging @elastic/elastic-agent-control-plane (Team:Elastic-Agent-Control-Plane)

cmacknz · 2025-12-15T18:32:34Z

Dropping the output ID makes sense to me, but I think if you do a policy assignment the input ID also changes, which would cause an uninstall and reinstall.

Does the policy reassignment case work properly for tamper protected agents? Or is this only handling the case where the output is changed within the same policy?

ycombinator · 2025-12-16T17:57:51Z

Dropping the output ID makes sense to me, but I think if you do a policy assignment the input ID also changes, which would cause an uninstall and reinstall.

As discussed in today's meeting, the component ID uses the input type (not input ID) so, with the change in this PR, we should still prevent an uninstall and reinstall.

ycombinator · 2025-12-16T17:59:36Z

Does the policy reassignment case work properly for tamper protected agents? Or is this only handling the case where the output is changed within the same policy?

~~I will test this along with the upgrade scenario.~~

Yes, it does. I forgot that that's how I'd tested this PR to begin with. I even added manual testing steps to the PR's description. 🤦

I will test the upgrade scenario next.

ycombinator · 2025-12-17T08:54:42Z

I will test the upgrade scenario next.

Okay, so upgrading works in that nothing is broken after upgrade. Here's the output of elastic-agent status before, during, and after the upgrade:

Before

┌─ fleet
│  └─ status: (HEALTHY) Connected
└─ elastic-agent
   ├─ status: (HEALTHY) Running
   ├─ info
   │  ├─ id: a4d2bed3-5c18-42d0-bc55-cc838f2a43a6
   │  ├─ version: 9.2.0-SNAPSHOT
   │  └─ commit: 07d5c3a34e486f81130d7177678c547944dbb84c
   └─ endpoint-default
      ├─ status: (HEALTHY) Healthy: communicating with endpoint service
      ├─ endpoint-default
      │  ├─ status: (HEALTHY) Applied policy 'defend' (Defend policy rev. 1. Agent policy rev. 21.)
      │  └─ type: OUTPUT
      └─ endpoint-default-a92a4f1c-8773-43ce-8e97-062fc2cebb1f
         ├─ status: (HEALTHY) Applied policy 'defend' (Defend policy rev. 1. Agent policy rev. 21.)
         └─ type: INPUT

During

┌─ fleet
│  └─ status: (STARTING)
├─ elastic-agent
│  ├─ status: (HEALTHY) Running
│  ├─ info
│  │  ├─ id: 6c6af8b5-7e88-42d0-8d71-70afbcfa39f1
│  │  ├─ version: 9.3.0
│  │  └─ commit: 7026d24f297cdcd6826da98ec3c39ea7f0c59c19
│  └─ endpoint
│     ├─ status: (HEALTHY) Healthy: communicating with endpoint service
│     ├─ endpoint
│     │  ├─ status: (HEALTHY) Applied policy 'defend' (Defend policy rev. 1. Agent policy rev. 21.)
│     │  └─ type: OUTPUT
│     └─ endpoint-a92a4f1c-8773-43ce-8e97-062fc2cebb1f
│        ├─ status: (HEALTHY) Applied policy 'defend' (Defend policy rev. 1. Agent policy rev. 21.)
│        └─ type: INPUT
└─ upgrade_details
   ├─ target_version: 9.3.0
   ├─ state: UPG_WATCHING
   ├─ action_id: 349b43f0-973b-4b40-ab21-ee082debcfaf
   └─ metadata

After

┌─ fleet
│  └─ status: (HEALTHY) Connected
└─ elastic-agent
   ├─ status: (HEALTHY) Running
   ├─ info
   │  ├─ id: 6c6af8b5-7e88-42d0-8d71-70afbcfa39f1
   │  ├─ version: 9.3.0
   │  └─ commit: 7026d24f297cdcd6826da98ec3c39ea7f0c59c19
   └─ endpoint
      ├─ status: (HEALTHY) Healthy: communicating with endpoint service
      ├─ endpoint
      │  ├─ status: (HEALTHY) Applied policy 'defend' (Defend policy rev. 1. Agent policy rev. 21.)
      │  └─ type: OUTPUT
      └─ endpoint-a92a4f1c-8773-43ce-8e97-062fc2cebb1f
         ├─ status: (HEALTHY) Applied policy 'defend' (Defend policy rev. 1. Agent policy rev. 21.)
         └─ type: INPUT```

Notice that the ID of the Endpoint component after the upgrade changes to `endpoint`.  Specifically, the ID no longer has the output ID, `default`, as the suffix.

cmacknz · 2025-12-17T21:43:37Z

What's the right additional testing to add for this? It looks correct by inspection, but I think we should add something proving it does what we think.

If we could get a simulated policy reassignment (input id: key change) and output change as a test of just the component model that's probably the lightest thing we could do.

If policy reassignment always fails without this change, that's a test case that is missing from the endpoint integration tests, so we could add that. I also don't think we have a test that does an upgrade with defend installed but that is even heavier, though we have had lots of problems around this discovered in the field so maybe worth it.

ycombinator · 2025-12-22T05:21:33Z

If we could get a simulated policy reassignment (input id: key change) and output change as a test of just the component model that's probably the lightest thing we could do.

Added a unit test case for input ID change to TestComponentUpdateDiff in cf89151.

A unit test that tests just changing the output already exists in TestComponentUpdateDiff:

elastic-agent/internal/pkg/agent/application/coordinator/coordinator_test.go

Line 208 in eb1cd7f

name: "just-change-output",

But I added additional assertions for that test case to make sure no components were added, removed, or updated in the component model as a result of such a change: 4e352a7#diff-21de8fb82a5ebccaa0ca3afb0cb8bfbe10d5b8f920021785c2d485ee35a3556cR225-R227

go.mod

ycombinator · 2025-12-22T07:10:24Z

testing/integration/ess/endpoint_security_test.go

+	require.NotZero(t, secondPID)
+
+	// Assert that endpoint process IDs haven't changed across policy reassignment
+	require.Equal(t, firstPID, secondPID)


Is checking that Endpoint hasn't restarted good enough for this test? Or should we try to check that Endpoint has not be uninstalled/reinstalled — that might be more involved and require looking at the Endpoint service's logs in the service manager.

ycombinator · 2025-12-22T07:11:21Z

If policy reassignment always fails without this change, that's a test case that is missing from the endpoint integration tests, so we could add that.

Added in 8a1a8c3

ycombinator · 2025-12-22T07:15:33Z

I also don't think we have a test that does an upgrade with defend installed...
Looks like we do have some integration tests around upgrading Endpoint:

elastic-agent/testing/integration/ess/endpoint_security_test.go

Line 65 in eb1cd7f

func TestUpgradeAgentWithTamperProtectedEndpoint_DEB(t *testing.T) {

elastic-agent/testing/integration/ess/endpoint_security_test.go

Line 99 in eb1cd7f

func TestUpgradeAgentWithTamperProtectedEndpoint_RPM(t *testing.T) {

go.mod

…essful policy reassignment

elasticmachine · 2025-12-23T22:09:50Z

💔 Build Failed

Buildkite Build
Commit: aa3188b

Failed CI Steps

History

💔 Build #32608 failed e2c6373
💔 Build #32552 failed cdb0764
💔 Build #32541 failed ea763b6
💔 Build #32539 failed cf4ba02
💔 Build #32534 failed d2aa5d2

cc @ycombinator

mergify bot assigned ycombinator Dec 11, 2025

ycombinator changed the title ~~Avoid stopping and stopping service components on policy change~~ Avoid starting and stopping service components on policy change Dec 12, 2025

ycombinator force-pushed the service-component-avoid-stop-start branch from 84a4523 to 1951fec Compare December 12, 2025 14:53

ycombinator marked this pull request as ready for review December 12, 2025 14:54

ycombinator requested a review from a team as a code owner December 12, 2025 14:54

ycombinator requested review from blakerouse and michel-laterman December 12, 2025 14:54

ycombinator added the Team:Elastic-Agent-Control-Plane Label for the Agent Control Plane team label Dec 12, 2025

ycombinator changed the title ~~Avoid starting and stopping service components on policy change~~ Avoid uninstalling and re-installing service components on policy change Dec 16, 2025

cmacknz added the backport-9.3 Automated backport to the 9.3 branch label Dec 17, 2025

ycombinator force-pushed the service-component-avoid-stop-start branch from d7bf91b to e892133 Compare December 22, 2025 04:48

ycombinator added 12 commits December 22, 2025 10:19

Add UsesCommandRuntime and UsesServiceRuntime methods on Component

f089efb

Use new methods

84d06ca

Add test case for only output being changed on service component

016a485

Implement logic to not remove and add same service component

773d5b6

Adding CHANGELOG fragment

1847522

Improve comment

22476ec

Fix logic location

bee109a

Update unit test

33132c3

Update service component naming

d8111d6

Refactor: extract logic into helper method

91d9918

Relocate unit test and add lots of cases

1099766

Remove unnecessary code

d0f2c2b

ycombinator mentioned this pull request Dec 22, 2025

Added ReassignAgentToPolicy method to Kibana Fleet client elastic/elastic-agent-libs#380

Merged

3 tasks

ycombinator commented Dec 22, 2025

View reviewed changes

go.mod Outdated Show resolved Hide resolved

ycombinator commented Dec 22, 2025

View reviewed changes

Add integration test: TestPolicyReassignWithTamperProtectedEndpoint

8a1a8c3

ycombinator force-pushed the service-component-avoid-stop-start branch from 13f5ff1 to 8a1a8c3 Compare December 22, 2025 07:13

ycombinator requested a review from cmacknz December 22, 2025 07:15

Update replace in go.mod

92469fe

ycombinator commented Dec 22, 2025

View reviewed changes

go.mod Outdated Show resolved Hide resolved

ycombinator added 18 commits December 22, 2025 13:01

Running mage notice

d2aa5d2

Bump up context timeout and use for entire test

174ca59

Define fixture

2fc4cd5

Fix syntax errors

cf4ba02

Fix installOpts

ea763b6

Only cleanup Endpoint using first policy's uninstall token until succ…

cdb0764

…essful policy reassignment

Clarify log message

e254cf6

Upgrade endpoint package version

e2c6373

Use exec.CommandContext and separate out args

3810c78

Compare Endpoint policy IDs

ca68c31

Use agentID from enrollment response

1517b94

Install Elastic Defend in second policy

03be197

Add endpoint cleanup after reassigning policy

cd8b284

Fixing log messages

eecd554

Give Endpoint time to receive reassigned policy

289435b

Updating dependency version

2ef136a

Adding log statements

a296eb1

Remove replace

aa3188b

Avoid uninstalling and re-installing service components on policy change #11740

Are you sure you want to change the base?

Avoid uninstalling and re-installing service components on policy change #11740

Conversation

ycombinator commented Dec 11, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What does this PR do?

Why is it important?

Checklist

Disruptive User Impact

How to test this PR locally

Policy reassign does not uninstall/reinstall Endpoint

Removing Endpoint from policy uninstalls Endpoint

Related issues

Questions to ask yourself

Uh oh!

mergify bot commented Dec 11, 2025

Uh oh!

elasticmachine commented Dec 12, 2025

Uh oh!

cmacknz commented Dec 15, 2025

Uh oh!

ycombinator commented Dec 16, 2025

Uh oh!

ycombinator commented Dec 16, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ycombinator commented Dec 17, 2025

Before

During

After

Uh oh!

cmacknz commented Dec 17, 2025

Uh oh!

ycombinator commented Dec 22, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

ycombinator Dec 22, 2025

Choose a reason for hiding this comment

Uh oh!

ycombinator commented Dec 22, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ycombinator commented Dec 22, 2025

Uh oh!

Uh oh!

elasticmachine commented Dec 23, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

💔 Build Failed

Failed CI Steps

History

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

ycombinator commented Dec 11, 2025 •

edited

Loading

ycombinator commented Dec 16, 2025 •

edited

Loading

ycombinator commented Dec 22, 2025 •

edited

Loading

ycombinator commented Dec 22, 2025 •

edited

Loading

elasticmachine commented Dec 23, 2025 •

edited

Loading