Re: [PATCH perf-tools-next v9 4/6] perf ftrace: Support display of inlined functions in function graph tracer

[Aaron Tomlin]

On Thu, Oct 22, 2026 at 02:53:54PM -0700, Namhyung Kim wrote:
> On Tue, Oct 20, 2026 at 10:33:35PM -0400, Aaron Tomlin wrote:
> > The Linux kernel's function graph tracer operates at the machine
> > instruction level via compiler instrumentation
> > (-fpatchable-function-entry), recording entry and return events for
> > physical function calls. Consequently, functions inlined by the compiler
> > are invisible in the resulting call graph, as no discrete call or return
> > instructions are emitted for them. While technically expected, this
> > omission frequently obscures the logical execution flow when analysing
> > kernel subsystems heavily reliant upon inlining.
> > 
> > Address this by introducing the --inline option to perf ftrace for the
> > function_graph tracer.
> > 
> > When the kernel is configured with CONFIG_FUNCTION_GRAPH_RETADDR=y, the
> > funcgraph-retaddr trace option records the caller's return address on
> > each function entry, manifested in the trace stream as a comment
> > (e.g. /* <-irq_work_queue_on+0x81/0xa0 */). By interrogating DWARF debug
> > information from the kernel image (vmlinux) utilising libdw, perf ftrace
> > resolves this return address to its inlined callchain and synthesises the
> > intermediate inlined frames directly into the streamed call graph.
> > 
> > Key aspects of this implementation:
> > 
> >     1.  Inlined frame presentation
> > 
> >         Synthesised inlined functions are rendered with an explicit
> >         /* (inline) */ annotation at both entry and exit:
> > 
> >             # tracer: function_graph
> >             #
> >             # CPU               FUNCTION CALLS
> >             # |                 |   |   |   |
> >              6) activate_task() {
> >              6)   enqueue_task() {
> >              6)     enqueue_task_fair() {
> >              6)       update_curr_eevdf() { /* (inline) */
> >              6)         update_curr() {
> >              6)           update_se() {
> >              6)             cgroup_account_cputime() { /* (inline) */
> >              6)               cpuacct_charge();
> >              6)             } /* cgroup_account_cputime (inline) */
> >              6)           }
> >              6)           dl_server_update() {
> >              6)             update_curr_dl_se();
> >              6)         }
> >              6)       } /* update_curr_eevdf (inline) */
> >              6)       enqueue_hierarchy() { /* (inline) */
> >              6)         update_curr() {
> >              6)           update_se();
> >              6)         }
> >              6)         enqueue_entity() {
> >              6)           account_entity_enqueue() { /* (inline) */
> >              6)             account_llc_enqueue();
> >              6)           } /* account_entity_enqueue (inline) */
> >              6)         }
> >              6)       } /* enqueue_hierarchy (inline) */
> >              6)       place_entity() {
> >              6)         avg_vruntime();
> >              6)       }
> >              6)       __enqueue_entity();
> >              6)       dl_server_start();
> >              6)       hrtick_update() { /* (inline) */
> >              6)         hrtick_active() { /* (inline) */
> >              6)           hrtimer_active();
> >              6)         } /* hrtick_active (inline) */
> >              6)       } /* hrtick_update (inline) */
> >              6)       hrtick_start_fair();
> >              6)     psi_enqueue() { /* (inline) */
> >              6)       psi_task_change() {
> >              6)         psi_flags_change();
> >              6)         psi_group_change() {
> >              6)           record_times();
> >              6)         }
> >              6)       }
> >              6)     } /* psi_enqueue (inline) */
> >              6)   }
> >              6) }
> > 
> >         This affords immediate visual clarity to the reader upon
> >         entering an inlined scope, without necessitating a search for
> >         the closing delimiter.
> > 
> >     2.  Call graph reconstruction and per-task stack tracking
> > 
> >         Maintains a per-task synthetic call stack tracking real and
> >         inlined frames across context switches, synthesising entry and
> >         exit delimiters to accurately reconstruct the execution
> >         hierarchy. Tasks are tracked in a hash table keyed by PID, with
> >         per-CPU active bindings updated upon context-switch banners or
> >         verbose line records. Inlined frames are structured strictly
> >         root-to-leaf and validated against the active stack to prevent
> >         un-traced callers from attaching orphan inlines. Bare closing
> >         braces without trailing function names (the default when
> >         funcgraph-tail is disabled) are correctly matched against
> >         preceding frames on the stack. Context column alignment,
> >         including task comm-pid delimiters in verbose mode, is preserved
> >         when generating prefixes for synthesised inline frames.
> >         Furthermore, stack depth saturation guards ensure that inlined
> >         frame depth tracking remains clamped, indentation cannot drift
> >         even under deep call trees exceeding FTRACE_MAX_STACK, and
> >         overflow depth is automatically recovered upon caller detection.
> > 
> >     3.  Option handling and diagnostics
> > 
> >         Adds -k or --vmlinux to designate an explicit debug image, and
> >         provides clear diagnostic guidance should --inline be invoked on
> >         a kernel lacking CONFIG_FUNCTION_GRAPH_RETADDR. Automatically
> >         enables return address comment filtering unless explicitly
> >         requested.
> > 
> >     4.  Automated verification
> > 
> >         Expands the unit test suite in tools/perf/tests/ftrace.c to
> >         validate DWARF inline symbol resolution, comment filtering
> >         toggles, bare closing braces handling, task context-switch
> >         isolation and migration, and stack saturation/overflow recovery.
> > 
> > Assisted-by: Antigravity:gemini-3.1-pro
> > Signed-off-by: Aaron Tomlin <atomlin@xxxxxxxxxxx>
> > ---
> >  tools/perf/Documentation/perf-ftrace.txt |   17 +-
> >  tools/perf/builtin-ftrace.c              |   76 +-
> >  tools/perf/tests/ftrace.c                |  914 ++++++++++++++++++-
> >  tools/perf/util/ftrace.c                 | 1059 +++++++++++++++++++++-
> >  tools/perf/util/ftrace.h                 |   14 +
> >  5 files changed, 2059 insertions(+), 21 deletions(-)
> 
> I think it's too big and you can split into pieces like:
> 
>  1. add the basic task stack management code
>  2. support context switch
>  3. support inline functions
>  4. add test code
> 
> Thanks,
> Namhyung

Hi Namhyung,

Acknowledged.

Kind regards,
-- 
Aaron Tomlin