Re: [PATCH perf-tools-next v2 0/4] perf ftrace: Support inlined functions and display enhancements in function graph tracer

[Aaron Tomlin]

On Wed, Oct 07, 2026 at 04:24:48PM -0700, Namhyung Kim wrote:
> Hello,
> 
> Interesting work!

Thank you Namhyung.

> On Tue, Oct 06, 2026 at 07:27:52PM -0400, Aaron Tomlin wrote:
> > The Linux kernel's function graph tracer operates at the machine
> > instruction level via compiler instrumentation
> > (-fpatchable-function-entry), recording entry and return events for
> > physical function calls. Consequently, functions inlined by the compiler
> > are invisible in the resulting call-graph, as no discrete call or return
> > instructions are emitted for them. While technically expected, this
> > omission frequently obscures the logical execution flow when analysing
> > kernel subsystems heavily reliant upon inlining (such as the scheduler,
> > memory management, locking primitives, and RCU).
> > 
> > When the kernel is configured with CONFIG_FUNCTION_GRAPH_RETADDR=y, the
> > funcgraph-retaddr trace option records the caller's return address on each
> > function entry, manifested in the trace stream as a comment
> > (e.g. /* <-wake_up_new_task+0x1d1/0x3e0 */). By interrogating DWARF debug
> > information from the kernel image (vmlinux) utilising libdw, perf ftrace
> > resolves this return address to its inlined callchain and synthesises the
> > intermediate inlined frames directly into the streamed call-graph.
> > 
> > Synthesised inlined functions are rendered with an explicit /* (inline) */
> > annotation at both entry and exit:
> > 
> >      # CPU  DURATION              FUNCTION CALLS
> >      # |     |   |                 |   |   |   |
> >       1)   0.856 us |  mutex_unlock(lock=0xffffffffbc28bfa0);
> >       0)            |  wake_up_new_task(p=0xffff8b1f183bac80) {
> >       0)            |    rb_irq_work_queue() { /* (inline) */
> >       0)            |      rb_wakeups() { /* (inline) */
> >       0)   1.374 us |        housekeeping_any_cpu(type=3);
> >       0)            |        arch_irq_work_raise() {
> >       0)            |          apic_wait_icr_idle() { /* (inline) */
> >       0)            |            x2apic_send_IPI_self(vector=246) {
> >       0)            |              instr_sysvec_irq_work() { /* (inline) */
> >       0)            |                irq_enter_rcu() {
> >       0)            |                  instr_sysvec_irq_work() { /* (inline) */
> >       0)   0.673 us |                    irqtime_account_irq();
> >       0)            |                  } /* instr_sysvec_irq_work (inline) */
> >       0)   1.697 us |                }
> 
> Let me read this output..
> 
> It seems housekeeping_any_cpu() is called from the inlined chain:
> 
>   wake_up_new_task()
>     -> rb_irq_work_queue()
>       -> rb_wakeups()
>         -> housekeeping_any_cpu()
> 
> But I think the actual call chain would seem like:
> 
>   wake_up_new_task()
>     -> rb_wakeups()
>       -> rb_irq_work_queue()
>         -> housekeeping_any_cpu()
> 
> And I don't know how wake_up_new_task() called rb_wakeups() directly.

That resulted in the inverted nesting of rb_irq_work_queue() and
rb_wakeups(), as well as the unnatural parentage under wake_up_new_task().

In the actual implementation, DWARF inline resolution via libdw traverses
function DIEs from parent to child (using cu_walk_functions_at()), so
synthesised inlined frames are always ordered strictly root-to-leaf. In
this scenario, rb_wakeups() correctly precedes rb_irq_work_queue().

In the next iteration, I will update the commit log in with a genuine,
reproducible trace from the kernel (e.g. wake_up_new_task() ->
activate_task() [inline] -> enqueue_task(), as exercised in the test suite)
to avoid any confusion.

> Also how do you know if arch_irq_work_raise() is called within the same
> inlined call-chain?  It seems rb_irq_work_queue() indeed calls the
> function after housekeeping_any_cpu():
> 
>   rb_irq_work_queue()
>     -> irq_work_queue_on()
>       -> __irq_work_queue_local()
>         -> irq_work_raise()
> 
> I guess you compare the callchain with the previous one and merge the
> call if they match.  Then it would support partial match and close some
> parents correctly like when other function is called from rb_wakeups(),
> for example.

Yes. Each CPU maintains a call-stack tracking both real and synthesised
inlined frames. When a function record arrives with a return address comment:

    1.  We resolve the inlined call chain from DWARF information and locate
        the enclosing real caller on the stack

    2.  We determine the longest common prefix between the currently active
        inlined frames above that caller and the newly resolved inlined
        chain

    3.  Any active inlined frames beyond the common prefix are closed and
        popped by emitting synthesised closing delimiters: "} /* <func> (inline) */"

    4.  Any new inlined frames from the common prefix index onwards are
        opened and pushed onto the stack

Consequently, when sibling calls are made within the same inlined parent
(or when a subsequent call branches off an intermediate inlined ancestor
like rb_wakeups()), shared ancestors are preserved while only the divergent
frames are unwound and replaced. When the enclosing real function exits,
any remaining open inlined frames above it are closed as well.

This partial matching and stack unwinding behaviour will also validated by
the unit tests in tools/perf/tests/ftrace.c.

> > This series decomposes the feature into a modular, bisectable progression
> > of four patches:
> > 
> >     1.  Stack frame optimisation in __cmd_ftrace()
> > 
> >         Replaces the 4096-byte stack array in __cmd_ftrace() with dynamic
> >         heap allocation alongside struct strbuf streaming, reducing the
> >         stack frame footprint by over 96% (down to ~160 bytes) while
> >         ensuring complete line-delimited records and robust EOF handling.
> > 
> >     2.  Return address comment filtering
> > 
> >         Introduces ftrace_parse_retaddr() and ftrace_filter_retaddr() with
> >         --filter-retaddr and --graph-opts filter-retaddr to strip caller
> >         comments from trace output, accompanied by an initial unit test
> >         suite in tools/perf/tests/ftrace.c.
> 
> I'm curious why do we need options as we already have 'retaddr' graph-option
> to enable it.  I think it should work the same as 'retaddr' is not
> given, no?  Otherwise, we could add 'noretaddr' instead.
> 
> Also I think it's only useful with the --inline option.  If so, we could
> handle that automatically when it's given (and 'retaddr' is not given).

That is a very good point. The only reason return address comment filtering
is required in the first place is because --inline needs the kernel's
funcgraph-retaddr option enabled behind the scenes in tracefs. Without the
caller's return address and instruction offset, DWARF cannot resolve the
inlined call chain. However, leaving those raw comments (/* <-caller+offset
*/) in the stream creates considerable visual clutter when the user is
primarily interested in the reconstructed inlined hierarchy.

Handling this automatically is much cleaner:

    1.  When --inline is specified

        perf ftrace enables funcgraph-retaddr in the kernel for symbol
        resolution, while automatically filtering the comments from the
        rendered output

    2.  If the user explicitly passes --graph-opts retaddr alongside --inline

        We retain the return address comments in the output as requested

    3.  When --inline is not specified

        The kernel's funcgraph-retaddr option remains disabled by default
        unless --graph-opts retaddr is supplied, in which case the comments
        are shown without filtering

For the next iteration, I will drop --graph-opts filter-retaddr and
--filter-retaddr, folding the filtering logic directly into the --inline
implementation so that it behaves automatically based on whether
--graph-opts retaddr is present.


Kind regards,
-- 
Aaron Tomlin