Re: [PATCH 02/12] perf debuginfo: Fetch debuginfo keyed by build ID using debuginfod

[Ian Rogers]

On Wed, Sep 16, 2026 at 12:02 PM Arnaldo Carvalho de Melo
<acme@xxxxxxxxxx> wrote:
>
> On Wed, Sep 16, 2026 at 10:59:31AM -0700, Ian Rogers wrote:
> > On Wed, Sep 16, 2026 at 4:48 AM Arnaldo Carvalho de Melo <acme@xxxxxxxxxx> wrote:
> > >
> > > perf already uses debuginfod to fetch source files when annotating
> > > (via probe-finder.c) and 'perf probe' has open_from_debuginfod(),
> > > which queries debuginfo keyed by build ID when a module's debuginfo
> > > isn't found locally, largely the same thing this adds; eventually
> > > that one could be moved over to the new helper.  For the analysis
> > > tools there was no way to obtain the debuginfo for a DSO in a
> > > profile when it isn't available locally under the name the DSO was
> > > opened with, for instance the vmlinux for the kernel a profile was
> > > recorded on when processing it on another machine, or after the
> > > kernel and its debuginfo package got upgraded in between.
> > >
> > > Add debuginfo__find_build_id(), that uses the debuginfod client to
> > > locate a debuginfo file keyed by the build ID, checking its local
> > > cache first and then querying the servers in DEBUGINFOD_URLS, and
> > > debuginfo__new_build_id(), that opens the DWARF in the file it finds.
>
> > When do we have a build ID but not a DSO? The current intent is that
>
> Trying to parse this: Before we had a PERF_RECORD_MMAP with a dso name
> that we, at the end, when enabled, would look for a build-id to add to
> the perf.data headers, now we get both in the PERF_RECORD_MMAP3, right?

I believe the dso_id allows look up of a DSO before having its name.

> > dso__debuginfo hide these complexities. We currently don't purge DSOs
>
> From the cache, right, and that is a problem, we need to do that LRU you
> mention, to not have a ever growing cache.

Yeah, I think we can periodically scan the machine's DSOs, and if the
reference count is one and some LRU metric is met, we can drop the
reference count to 0 and remove it from the DSOs array. Since the
debuginfo is owned by the DSO, it gets cleaned up as well.

> There is a recent patch from someone at Uber about another aspect of
> this, the symtabs loaded in memory for resolving symbols are not in any
> way constrained, we go on loading, not purging, hope that patch gets
> resubmitted addressing the sashiko reviews that were acknowledged.

I've missed that one, but it sounds good.

> > and the first call to dso__debuginfo should trigger loading with the
> > DSO owning the debuginfo. With this change we now have a duplication
> > of DSO's data, keyed by build ID and mapping directly to the
> > debuginfo. I can't see a performance or efficiency gain and we could
> > potentially implement some kind of LRU mechanism for DSOs, which would
> > benefit memory usage for things like perf top.
>
> > > The debuginfod client fails when DEBUGINFOD_URLS isn't set even when
> > > what it wants is in its local cache, and the distro setup scripts that
> > > populate it from /etc/debuginfod don't reach cron jobs, systemd services
> > > and other environments that don't source the profile scripts, so also
> > > set it from the .urls files in /etc/debuginfod when not set.
>
> > I found the explanation above hard to follow. Are we setting an
> > environment variable because debuginfod isn't respecting its /etc
> > setup?
>
> IIRC what I saw was the debuginfod client not finding things in the
> cache when that DEBUGINFO_URLS variable wasn't set, so setting it
> doesn't mean to ask for downloads necessarily, but to use what is
> already cached locally.

Perhaps we need to get this fixed in libdebuginfod. Or should the
local connection to debuginfod automatically scan the /etc directory.
It sounds like an elfutils bug that we may need to work around, but we
should probably also raise the issue with elfutils.

> > > That has
> > > to happen before any thread that can call getenv() is started, as
> > > setenv() is not thread safe, and there are getenv()s outside the fetch
> > > lock: libdebuginfod reads DEBUGINFOD_URLS in every debuginfod_begin(),
> > > which perf also does in build-id.c, probe-event.c and probe-finder.c,
> > > so do it from symbol__init(), on the single-threaded setup, and not
> > > lazily from the fetch path.
> > >
> > > Querying servers, possibly third party ones, sends off-box the build
> > > IDs of the binaries being analysed and a fetch can take a while, so
> > > this is opt-out: on by default, off with --no-debuginfod, with
> > > core.debuginfod=false, per tool with report.debuginfod and
> > > top.debuginfod, and, since users that set buildid.dir to /dev/null
> > > (e.g. Linus) or otherwise turn the local build-id cache off clearly
> > > don't want fetched files stored on the box, off too in that case.
>
> > > The opt-outs also cover libdwfl's own debuginfod client, that reads
> > > DEBUGINFOD_URLS in every query: when debuginfod is off, be it with
> > > --no-debuginfod, core.debuginfod=false or by disabling the build-id
> > > cache, the variable is set to the empty string, that the client treats
> > > as an opt-out and fails the query without even looking at its cache,
> > > instead of being exported from /etc/debuginfod.  Tools that manage
> > > DEBUGINFOD_URLS themselves, such as 'perf record --debuginfod', keep
> > > doing so.
>
> > Libdwfl is part of elfutils and so is debuginfod. Is it possible to
> > share clients?
>
> We need to stop using ~/.debug/ and move to have the cache where elfutils
> libraries have it.

+1

> Transitioning should just use ~/.debug if available but saving copies of
> local DSOs like perf does in the elfutils cache directory.
>
> <SNIP>
>
> > > +static bool build_id__equal(const struct build_id *a, const struct build_id *b)
> > > +{
> > > +       return a->size == b->size && memcmp(a->data, b->data, a->size) == 0;
> > > +}
> >
> > This is probably worth moving to the build-id.[ch] file. I see similar
> > logic in places like __dso_id__cmp,  dso__missing_buildid_cache in
> > builitin-buildid-cache.c and sort__dcacheline_cmp. dso__build_id_equal
> > has some special backward compatibility checks.
>
> Agreed, will do.
>
> - Arnaldo