Re: [PATCH v7 00/14] perf build: Reduce build time by nearly half

[Namhyung Kim]

On Mon, May 18, 2026 at 08:46:24AM -0700, Ian Rogers wrote:
> This patch series refactors Kbuild internals, BPF skeleton generation,
> Python AST pre-computation, and foundational tooling dependencies across
> the perf tool build system. By eliminating umbrella target synchronization
> barriers, decoupling static library prerequisites, parallelizing single-core
> script generators, and eradicating redundant feature checks, this series
> unlocks absolute theoretical peak multi-core concurrency during Kbuild startup.
> 
> On a 28-core build workstation (make -j28 all from scratch), clean build
> latency improves by over 44%:
> 
>   Before:
>     real    0m29.006s
>     user    2m46.019s
>     sys     0m30.610s
> 
>   After:
>     real    0m16.091s
>     user    2m40.135s
>     sys     0m25.740s
> 
> Saving 12.9 full seconds time per clean build. Furthermore, nothing to
> build incremental builds are improved by nearly 7x:
> 
>   Before:
>     real    0m11.528s
>     user    0m9.633s
>     sys     0m6.965s
> 
>   After:
>     real    0m1.717s
>     user    0m1.682s
>     sys     0m0.960s
> 
> Summary of Patches:
> 
> 1: Fast-Path Feature Detection
>   - Refactors test-clang-bpf-co-re.bin and test-bpftool-skeletons.bin feature
>     checks to group shell pipelines within curly braces and redirect both stdout
>     and stderr to .make.output before touching $@ purely upon success
>     (> $(@:.bin=.make.output) 2>&1 && touch $@). Grouping the pipeline ({ cmd1 | cmd2; })
>     ensures that compiler stderr is successfully captured in .make.output rather
>     than escaping to the parent shell. This perfectly matches standard Kbuild
>     feature check conventions and ensures the target files are touched on disk
>     purely upon success, allowing Kbuild to cache positive detections and avoid
>     continuous sub-make re-evaluations during incremental builds. Adds
>     test-bpftool-skeletons.bin to the clean FILES list and explicit source
>     prerequisite test-clang-bpf-co-re.c.

I think patch 1 can be separated and needs Ack/Review from BPF folks.

> 
> 2-4: Flattening Umbrella Prepare Barriers
>   - builtin-trace embedded inclusions and pmu-events generation are completely
>     decoupled from the sequential "prepare" umbrella target, eliminating Make
>     AST double-parsing overhead and unchoking parallel compilation barriers.
> 
> 5-7: Decoupling & Pre-generating BPF Skeletons
>   - BPF skeleton rules are extracted out of Makefile.perf into bpf_skel.mak.
>   - Decouples bpftool bootstrap from top-level static libbpf dependencies,
>     attaching bpf-skel-prepare directly to the umbrella prepare target. This
>     allows Make to pre-compile bpftool and dump vmlinux.h in the background at
>     build startup, removing the 7-second serialization bottleneck before BPF
>     object compilation.
>   - Ensures benchmark skeleton intermediate .bpf.o files are cleanly removed
>     during make clean, and adds bpf-skel-prepare to .PHONY.
> 
> 8-9: Foundational Linkage Optimization
>   - Moves static libsymbol library prerequisites out of the prepare step.
>   - Eliminates redundant libbpf sub-make feature checks during static builds.
> 
> 10-11: jevents.py Concurrency & Deduplication
>   - Splits the massive 2.8 MB big_c_string literal out of pmu-events.c into a
>     dedicated pmu-events-string.c compilation unit. This slices C compilation
>     latency in half by compiling string and struct tables simultaneously across
>     separate CPU cores while preserving zero dynamic ELF relocations. Adds
>     pmu-events-string.c to .gitignore, declares extern const char big_c_string[];
>     locally inside output_string_file and output_file when split to prevent linkage
>     conflicts with empty-pmu-events.c, defers file closures to ensure identical
>     timestamps, and uses canonical Make 4.0 @: dependency chaining.
>   - Pre-populates jevents.py JSON ASTs and metric formulas in parallel across
>     all available CPU cores using ProcessPoolExecutor (accelerating Python
>     execution by 11x, from 3.3s down to ~290ms). Moves _init_worker to top-level
>     scope to ensure clean pickling under spawn multiprocessing start methods.
> 
> 12: Out-of-Tree Incremental Rebuild Fix
>   - Prefixes SCRIPTS (perf-archive, perf-iostat) with $(OUTPUT) to prevent
>     Make from continuously re-executing script installation rules on already
>     built out-of-tree builds.
> 
> 13-14: AST Parsing Optimization & Shell Fork Eradication
>   - Converts ZENS, ARMS, and INTELS in pmu-events/Build from recursive assignment
>     (=) to simply expanded assignment (:=) and replaces model_name/vendor_name
>     with pure GNU Make string functions. This guarantees Make executes directory
>     probing shell forks exactly once during AST parsing and evaluates path macros
>     purely in memory, completely eradicating over 7,800 redundant sub-processes
>     during out-of-tree build evaluation.
>   - Converts llvm-config shell queries in Makefile.config from recursive assignment
>     (=) to simply expanded assignment (:=). This eliminates ~185 redundant sub-processes
>     that were previously executed across object compilation dependency checks.
> 
> Changes since v6:
>   - Rebase/resend as last series failed to apply by Sashiko.
> 
> Ian Rogers (14):
>   tools build: Fix feature checks to touch target files on success
>   perf trace beauty: Make beauty generated C code standalone .o files
>   perf build: Decouple pmu-events from prepare umbrella target
>   perf build: Remove empty archheaders target
>   perf build: Move BPF skeleton generation out of Makefile.perf
>   perf build: Encapsulate vmlinux.h and bpftool in bpf_skel.mak
>   perf build: Pre-generate BPF skeleton tooling during umbrella prepare
>     phase
>   perf build: Move libsymbol dependency out of prepare step
>   perf build: Remove redundant libbpf feature check for static builds
>   perf pmu-events: Split big_c_string storage into standalone
>     compilation unit
>   perf pmu-events: Parallelize JSON and metric pre-computation in
>     jevents.py
>   perf build: Prefix SCRIPTS with output directory to fix continuous
>     rebuilds
>   perf pmu-events: Convert recursive shell assignments and macros to
>     Make built-ins
>   perf build: Convert llvm-config shell queries to simply expanded
>     variables

Reviewed-by: Namhyung Kim <namhyung@xxxxxxxxxx>

Thanks,
Namhyung