perf-prof is a comprehensive Linux system-level analysis tool for long-term performance monitoring with low overhead, broad compatibility, and high reliability.
perf-prof is a user-space performance profiling tool built on top of libperf, libtraceevent, and libbpf. It provides real-time analysis without writing event data to disk, processing everything in memory and discarding it immediately after use.
- In-Memory Processing: Events are processed in memory and discarded immediately - no persistent storage overhead
- Broad Compatibility: Works with older Linux kernels (requires perf_event support)
- User-Space Implementation: Safe execution with rapid iteration capability
- Modular Architecture: 30+ specialized profilers for different analysis scenarios
- Low Overhead: Kernel-level filtering reduces data transfer to user space
- Real-Time Analysis: Process events as they occur with immediate feedback
# Install required dependencies
yum install -y xz-devel elfutils-libelf-devel
# Optional: Install eBPF dependencies
yum install -y llvm bpftool# Clone the repository
git clone https://github.com/OpenCloudOS/perf-prof.git
cd perf-prof
# Build the project
make
# Verbose build
make V=1
# Clean build artifacts
make clean# Using CROSS_COMPILE
make CROSS_COMPILE=aarch64-linux-gnu-
# Using LLVM
make LLVM=1# List all profilers
./perf-prof -h
# List all tracepoint events
./perf-prof list# Profile CPU usage at 997Hz with call graph
./perf-prof profile -F 997 -g
# Generate flame graph
./perf-prof profile -F 997 -g --flame-graph cpu.folded
# Profile user-space only on specific CPU
./perf-prof profile -F 997 -C 0-3 --exclude-kernel --than 30# Detect kernel memory leaks
./perf-prof kmemleak --alloc "kmem:kmalloc//ptr=ptr/size=bytes_alloc/stack/" \
--free "kmem:kfree//ptr=ptr/" --order -m 128 -g# Monitor task states (R, S, D, T, I)
./perf-prof task-state -i 1000
# Analyze scheduling delay
./perf-prof rundelay --than 4ms -i 1000# Trace specific events
./perf-prof trace -e sched:sched_wakeup,sched:sched_switch -i 1000
# Trace with filtering
./perf-prof trace -e "sched:sched_wakeup/prio<10/" -i 1000
perf-prof provides 30+ specialized profilers organized by category:
- profile - CPU performance sampling analysis
- oncpu - Monitor processes running on CPU
- kmemleak - Memory leak detection
- kmemprof - Memory allocation profiling
- page-faults - Page fault tracking
- task-state - Process state monitoring (R, S, D, T, I)
- rundelay - Scheduling run delay analysis
- sched-migrate - Process migration monitoring
- blktrace - Block device I/O tracking
- kvm-exit - KVM exit latency analysis
- kvmmmu - KVM MMU mapping observation
- hwstat - Hardware state monitoring (cycles, IPC)
- llcstat - Last-level cache monitoring
- tlbstat - dTLB monitoring
- ldlat-loads - Intel load latency counting
- ldlat-stores - Intel store instruction counting
- split-lock - x86 split lock detection
- hrtimer - High-resolution conditional sampling
- irq-off - Interrupt disabled detection
- watchdog - Hard/soft lock detection
- sql - SQL aggregation analysis with SQLite
- top - Key-value statistical analysis
- multi-trace - Multi-event relationship analysis
- syscalls - System call latency analysis
- expr - Expression-based event processing
- trace - Event tracking
- list - List tracepoint events
- expr - Expression-based test tool
- usdt - User Statically Defined Tracing
- breakpoint - Kernel/user space hardware breakpoints
- kcore - Read kernel memory
- misc - Miscellaneous tracking
perf-prof follows a three-layer event selection specification:
# List all events
./perf-prof list
# Filter by category
./perf-prof list | grep -E "^(sched:|kmem:|timer:|irq:)"# View event fields
./perf-prof trace -e sched:sched_wakeup help
# Multiple events
./perf-prof trace -e sched:sched_wakeup,sched:sched_switch helpEVENT: sys:name[/filter/ATTR/ATTR/.../]
kprobe:func[/filter/ATTR/ATTR/.../]
kretprobe:func[/filter/ATTR/ATTR/.../]
uprobe:func@"file"[/filter/ATTR/ATTR/.../]
uretprobe:func@"file"[/filter/ATTR/ATTR/.../]
# Numeric comparison
./perf-prof trace -e "sched:sched_wakeup/pid>1000/"
./perf-prof trace -e "sched:sched_wakeup/prio<10/"
# String matching
./perf-prof trace -e 'sched:sched_wakeup/comm=="java"/'
./perf-prof trace -e 'sched:sched_wakeup/comm~"pyth*"/'
# Logical combinations
./perf-prof trace -e "sched:sched_wakeup/pid>1000 && prio<10/"stack # Enable call stack
alias=str # Event alias
max-stack=int # Max stack depth
key=EXPR # Event key for correlation
top-by=EXPR # Sort field
comm=EXPR # Process name display
ptr=EXPR # Pointer field
size=EXPR # Size field
num=EXPR # Number distribution field# profiler help with examples
./perf-prof trace -h
./perf-prof task-state -h
# Event help with field information
./perf-prof trace -e sched:sched_wakeup help
./perf-prof kmemleak --alloc kmem:kmalloc --free kmem:kfree help- Main Options Reference - Complete command-line options
- profile - CPU performance analysis
- task-state - Process state monitoring
- multi-trace - Multi-event analysis
- sql - SQL aggregation analysis
- top - Key-value statistics
- kmemleak - Memory leak detection
- kvm-exit - KVM exit analysis
- blktrace - Block device I/O tracking
- trace - Event tracing
- Event Filtering - Trace event filter syntax
- Expressions - Expression language reference
- README - 中文版 - Chinese README
# Run all tests
cd tests
pytest
# Run specific test file
pytest test_profile.py
# Run with custom runtime and memory leak check
pytest --runtime=20 --memleak-check=2000perf-prof/
├── *.c # Core profiler modules (30+ profilers)
├── lib/ # Base libraries (libperf, libtraceevent, libbpf)
├── arch/ # Architecture-specific code
├── bpf-skel/ # BPF skeleton programs
├── filter/ # Event filters (BPF, tracepoint, PMU)
├── sqlite/ # SQLite amalgamation source code and extension modules
├── include/ # Included header files
├── tests/ # Test suite
└── docs/ # Documentation
Monitoring Framework:
monitor.c/h- Core frameworktep.c/h- Trace event parsertrace_helpers.c/h- Trace event utilitiesstack_helpers.c/h- Stack traversal and symbol resolution
Profiling Units:
- Each profiler is an independent
.cfile - Registered via
PROFILER_REGISTER()macro - Supports
init,deinit,interval,read,samplecallbacks
Event Source → Filters → Ring Buffer → Sort → Profiler → Output
- Create source file
new_profiler.c - Implement
profilerstructure with required callbacks - Define
name,desc,argc,option - Register with
PROFILER_REGISTER() - Add to
Buildfile:perf-prof-y += new_profiler.o - Add test in
tests/directory
We welcome contributions! Please follow these guidelines:
- Fork the repository
- Create a feature branch
- Write tests for new functionality
- Ensure all tests pass
- Submit a pull request
This project is licensed under GPLv2. See LICENSE for details.
- Follow Linux kernel coding style
- Write clear, maintainable code
- Include tests for new features
- Document public APIs and interfaces
Built with components from the Linux kernel:
- libperf
- libtraceevent
- libbpf
- Additional utility libraries