Gather Kubernetes data about pod volumes and add K8s context to filesystem metrics #613

ciprianfocsaneanu · 2026-01-15T14:57:17Z

No description provided.

…ystem metrics

- Added VPCIndex in-memory index for IP metadata lookups across VPCs, subnets, peered networks, and service ranges - Periodic refresh from cloud provider with configurable interval - gRPC endpoint GetIPsInfo enriches IPs with VPC metadata (region, zone, cloud service detection via domain) - Improved zone/region detection with kube nodes info

…ystem metrics

…ude kube-system

cmd/agent/daemon/pipeline/storage_info_provider.go

ciprianfocsaneanu · 2026-01-19T09:02:10Z

Tested on a GCP cluster with 2 pods, one requesting filesystem mode and one with block. Works as expected with logging only mode

time=2026-01-19T09:00:42.531Z level=info source=storage_pipeline.go:59 msg="collected 16 block device metrics" component=ctrl
time=2026-01-19T09:00:42.531Z level=info source=controller.go:155 msg="[BlockDevice] name=sda path=/dev/sda size=107374182400 type=SSD partition_of= holders=[] is_virtual=false raid_level= read_iops=0.02 write_iops=66.30 read_throughput=68 write_throughput=640136 read_latency_ms=0.00 write_latency_ms=0.62 util=0.02" writer=block_device
time=2026-01-19T09:00:42.531Z level=info source=controller.go:155 msg="[BlockDevice] name=sda3 path=/dev/sda3 size=2147483648 type=SSD partition_of=sda holders=[] is_virtual=false raid_level= read_iops=0.00 write_iops=0.00 read_throughput=0 write_throughput=0 read_latency_ms=0.00 write_latency_ms=0.00 util=0.00" writer=block_device
time=2026-01-19T09:00:42.531Z level=info source=controller.go:155 msg="[BlockDevice] name=sda5 path=/dev/sda5 size=2147483648 type=SSD partition_of=sda holders=[] is_virtual=false raid_level= read_iops=0.00 write_iops=0.00 read_throughput=0 write_throughput=0 read_latency_ms=0.00 write_latency_ms=0.00 util=0.00" writer=block_device
time=2026-01-19T09:00:42.531Z level=info source=controller.go:155 msg="[BlockDevice] name=sda6 path=/dev/sda6 size=512 type=SSD partition_of=sda holders=[] is_virtual=false raid_level= read_iops=0.00 write_iops=0.00 read_throughput=0 write_throughput=0 read_latency_ms=0.00 write_latency_ms=0.00 util=0.00" writer=block_device
time=2026-01-19T09:00:42.531Z level=info source=controller.go:155 msg="[BlockDevice] name=sda8 path=/dev/sda8 size=16777216 type=SSD partition_of=sda holders=[] is_virtual=false raid_level= read_iops=0.00 write_iops=0.00 read_throughput=0 write_throughput=0 read_latency_ms=0.00 write_latency_ms=0.00 util=0.00" writer=block_device
time=2026-01-19T09:00:42.531Z level=info source=controller.go:155 msg="[BlockDevice] name=sda9 path=/dev/sda9 size=512 type=SSD partition_of=sda holders=[] is_virtual=false raid_level= read_iops=0.00 write_iops=0.00 read_throughput=0 write_throughput=0 read_latency_ms=0.00 write_latency_ms=0.00 util=0.00" writer=block_device
time=2026-01-19T09:00:42.531Z level=info source=controller.go:155 msg="[BlockDevice] name=sda11 path=/dev/sda11 size=8388608 type=SSD partition_of=sda holders=[] is_virtual=false raid_level= read_iops=0.00 write_iops=0.00 read_throughput=0 write_throughput=0 read_latency_ms=0.00 write_latency_ms=0.00 util=0.00" writer=block_device
time=2026-01-19T09:00:42.531Z level=info source=controller.go:155 msg="[BlockDevice] name=sda2 path=/dev/sda2 size=16777216 type=SSD partition_of=sda holders=[] is_virtual=false raid_level= read_iops=0.00 write_iops=0.00 read_throughput=0 write_throughput=0 read_latency_ms=0.00 write_latency_ms=0.00 util=0.00" writer=block_device
time=2026-01-19T09:00:42.531Z level=info source=controller.go:155 msg="[BlockDevice] name=sda4 path=/dev/sda4 size=16777216 type=SSD partition_of=sda holders=[] is_virtual=false raid_level= read_iops=0.00 write_iops=0.00 read_throughput=0 write_throughput=0 read_latency_ms=0.00 write_latency_ms=0.00 util=0.00" writer=block_device
time=2026-01-19T09:00:42.531Z level=info source=controller.go:155 msg="[BlockDevice] name=sda10 path=/dev/sda10 size=512 type=SSD partition_of=sda holders=[] is_virtual=false raid_level= read_iops=0.00 write_iops=0.00 read_throughput=0 write_throughput=0 read_latency_ms=0.00 write_latency_ms=0.00 util=0.00" writer=block_device
time=2026-01-19T09:00:42.531Z level=info source=controller.go:155 msg="[BlockDevice] name=sda12 path=/dev/sda12 size=33554432 type=SSD partition_of=sda holders=[] is_virtual=false raid_level= read_iops=0.00 write_iops=0.00 read_throughput=0 write_throughput=0 read_latency_ms=0.00 write_latency_ms=0.00 util=0.00" writer=block_device
time=2026-01-19T09:00:42.531Z level=info source=controller.go:155 msg="[BlockDevice] name=dm-0 path=/dev/dm-0 size=2087714816 type=SSD partition_of= holders=[] is_virtual=true raid_level= read_iops=0.00 write_iops=0.00 read_throughput=0 write_throughput=0 read_latency_ms=0.00 write_latency_ms=0.00 util=0.00" writer=block_device
time=2026-01-19T09:00:42.531Z level=info source=controller.go:155 msg="[BlockDevice] name=sdc path=/dev/sdc size=1073741824 type=SSD partition_of= holders=[] is_virtual=false raid_level= read_iops=0.00 write_iops=0.00 read_throughput=0 write_throughput=0 read_latency_ms=0.00 write_latency_ms=0.00 util=0.00" writer=block_device
time=2026-01-19T09:00:42.531Z level=info source=controller.go:155 msg="[BlockDevice] name=sda1 path=/dev/sda1 size=102917717504 type=SSD partition_of=sda holders=[] is_virtual=false raid_level= read_iops=0.02 write_iops=66.30 read_throughput=68 write_throughput=640136 read_latency_ms=0.00 write_latency_ms=0.62 util=0.02" writer=block_device
time=2026-01-19T09:00:42.531Z level=info source=controller.go:155 msg="[BlockDevice] name=sda7 path=/dev/sda7 size=512 type=SSD partition_of=sda holders=[] is_virtual=false raid_level= read_iops=0.00 write_iops=0.00 read_throughput=0 write_throughput=0 read_latency_ms=0.00 write_latency_ms=0.00 util=0.00" writer=block_device
time=2026-01-19T09:00:42.531Z level=info source=controller.go:155 msg="[BlockDevice] name=sdb path=/dev/sdb size=1073741824 type=SSD partition_of= holders=[] is_virtual=false raid_level= read_iops=0.00 write_iops=0.00 read_throughput=0 write_throughput=0 read_latency_ms=0.00 write_latency_ms=0.00 util=0.00" writer=block_device
time=2026-01-19T09:00:42.535Z level=info source=storage_pipeline.go:78 msg="collected 4 filesystem metrics" component=ctrl
time=2026-01-19T09:00:42.535Z level=info source=controller.go:172 msg="[Filesystem] mount=/ devices=[/dev/root] type=ext2 options=[ro relatime] total=2068168704 used=1282899968 total_inodes=63744 used_inodes=14659 namespace=<nil> pod=<nil> pod_uid=<nil> pvc=<nil> pv=<nil> storage_class=<nil>" writer=filesystem
time=2026-01-19T09:00:42.535Z level=info source=controller.go:172 msg="[Filesystem] mount=/usr/share/oem devices=[/dev/sda8] type=ext4 options=[ro nosuid nodev noexec relatime] total=11513856 used=24576 total_inodes=4096 used_inodes=11 namespace=<nil> pod=<nil> pod_uid=<nil> pvc=<nil> pv=<nil> storage_class=<nil>" writer=filesystem
time=2026-01-19T09:00:42.535Z level=info source=controller.go:172 msg="[Filesystem] mount=/mnt/stateful_partition devices=[/dev/sda1] type=ext4 options=[rw nosuid nodev noexec relatime] total=101203873792 used=7969947648 total_inodes=6258720 used_inodes=84435 namespace=<nil> pod=<nil> pod_uid=<nil> pvc=<nil> pv=<nil> storage_class=<nil>" writer=filesystem
time=2026-01-19T09:00:42.535Z level=info source=controller.go:172 msg="[Filesystem] mount=/var/lib/kubelet/pods/d9d84e01-0fb6-4953-a657-531d06697678/volumes/kubernetes.io~csi/pvc-760c59ea-df87-4ebe-bce2-9cce69b80a67/mount devices=[/dev/sdb] type=ext4 options=[rw relatime] total=1020702720 used=24576 total_inodes=65536 used_inodes=11 namespace=default pod=test-pod-gcp pod_uid=d9d84e01-0fb6-4953-a657-531d06697678 pvc=test-pvc-gcp pv=pvc-760c59ea-df87-4ebe-bce2-9cce69b80a67 storage_class=standard-rwo" writer=filesystem
time=2026-01-19T09:00:42.535Z level=info source=storage_info_provider.go:327 msg="CollectPodVolumeMetrics: requesting pod volumes for node gke-ciprian-01-19-default-pool-ca7196f7-w42p"
time=2026-01-19T09:00:42.537Z level=info source=storage_info_provider.go:334 msg="CollectPodVolumeMetrics: received 2 volumes from controller"
time=2026-01-19T09:00:42.537Z level=info source=storage_pipeline.go:105 msg="collected 2 pod volume metrics" component=ctrl
time=2026-01-19T09:00:42.537Z level=info source=controller.go:205 msg="[PodVolume] ns=default pod=test-pod-gcp-block pod_uid=62d21c62-f011-4cb0-bbea-cc03f9f24f59 controller=Pod/test-pod-gcp-block container=test volume=data mount= mode=Block pvc=test-pvc-gcp-block requested_size=1073741824 pv=pvc-2f7e7ae2-bbe2-410b-a109-571ce3298b98 storage_class=standard-rwo csi_driver=pd.csi.storage.gke.io csi_volume_handle=projects/engineering-test-353509/zones/us-central1-a/disks/pvc-2f7e7ae2-bbe2-410b-a109-571ce3298b98 device_path=/dev/xvda" writer=pod_volume
time=2026-01-19T09:00:42.537Z level=info source=controller.go:205 msg="[PodVolume] ns=default pod=test-pod-gcp pod_uid=d9d84e01-0fb6-4953-a657-531d06697678 controller=Pod/test-pod-gcp container=test volume=data mount=/data mode=Filesystem pvc=test-pvc-gcp requested_size=1073741824 pv=pvc-760c59ea-df87-4ebe-bce2-9cce69b80a67 storage_class=standard-rwo csi_driver=pd.csi.storage.gke.io csi_volume_handle=projects/engineering-test-353509/zones/us-central1-a/disks/pvc-760c59ea-df87-4ebe-bce2-9cce69b80a67 device_path=<nil>" writer=pod_volume
time=2026-01-19T09:00:42.596Z level=info source=storage_pipeline.go:128 msg="collected node stats summary" component=ctrl
time=2026-01-19T09:00:42.596Z level=info source=controller.go:189 msg="[NodeStats] node=gke-ciprian-01-19-default-pool-ca7196f7-w42p image_fs_size=101203873792 image_fs_used=3871612928 container_fs_size=101203873792 container_fs_used=3871612928" writer=node_stats

patrickpichler · 2026-01-20T07:06:56Z

cmd/agent/daemon/pipeline/controller_test.go

 		fsWriter := ctrl.filesystemMetricsWriter.(*mockFilesystemMetricsWriter)

-		ctrl.collectStorageMetrics()
+		ctrl.collectStorageMetrics(context.Background())


Nit: use t.Context() instead

ciprianfocsaneanu and others added 15 commits January 15, 2026 16:56

Gather Kubernetes data about pod volumes and add K8s context to files…

730eee0

…ystem metrics

Take into account block volumes as well

9e0b52e

Add logging only mode

ec21bd3

Gather Kubernetes data about pod volumes and add K8s context to files…

72f6763

…ystem metrics

Fix merge

fdf6ca9

Merge branch 'main' into add-k8s-metadata-to-storage

0ce475c

Add debugging logs

0eb39ba

Filter pod volume metrics to only include PVC-backed volumes and excl…

9a4d281

…ude kube-system

Add RBAC permissions for PVCs and PVs in controller ClusterRole

870343c

Add CSI Volume handle to K8sPodVolumeMetric

44e25ed

Deduplicate filesystem metrics

8fe64a0

Add warning log about PV not found in index for PVC

7ff8bac

Improve logs

d88cf61

Improve logs

45d4176

patrickpichler reviewed Jan 19, 2026

View reviewed changes

cmd/agent/daemon/pipeline/storage_info_provider.go Outdated Show resolved Hide resolved

ciprianfocsaneanu added 8 commits January 19, 2026 11:32

Simplify FilesystemMetric and add in-tree volume source support

448fd2b

Remove logging only mode

520df3a

Use existing context

38926cd

Use local tables

f0254ae

Remove local suffix

0f84814

Merge branch 'main' into add-k8s-metadata-to-storage

ef1c2bc

Undo wrong change of removing metrics client start

fb2d1a1

Add k8s_pod_volume_metrics to E2E test

9e68ea2

ciprianfocsaneanu requested a review from samuelvl January 20, 2026 06:19

patrickpichler reviewed Jan 20, 2026

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Gather Kubernetes data about pod volumes and add K8s context to filesystem metrics #613

Gather Kubernetes data about pod volumes and add K8s context to filesystem metrics #613

ciprianfocsaneanu commented Jan 15, 2026

Uh oh!

Uh oh!

ciprianfocsaneanu commented Jan 19, 2026

Uh oh!

patrickpichler Jan 20, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Gather Kubernetes data about pod volumes and add K8s context to filesystem metrics #613

Are you sure you want to change the base?

Gather Kubernetes data about pod volumes and add K8s context to filesystem metrics #613

Conversation

ciprianfocsaneanu commented Jan 15, 2026

Uh oh!

Uh oh!

ciprianfocsaneanu commented Jan 19, 2026

Uh oh!

patrickpichler Jan 20, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants