Re: [PATCH] gen_compile_commands: honor the build directory for modules

[Van Vo]

Hi Nicolas,

> So, if I get youre reply correctly, the issue this was not discovered
> "in the wild" but can only be reproduced by manual use of
> gen_compile_commands.py.

Yes, that is correct. I found this through code inspection and reproduced
it with manual invocations, not from a failure in an existing workflow.
My later test with Kbuild-generated artifacts was to validate the path
handling; it was not a failure through Kbuild's compile_commands.json
target.

While looking for earlier reports after your reply, I found Omer Zalman's
RFC dated 28 September 2026 [1]. It describes the same failure with -d,
an absolute modules.order path, and .mod paths resolved from the caller's
directory. The author also used a fixture and explicitly had not tested
that invocation against a real kernel build. I only found this RFC during
this follow-up search, after submitting my patch.

There is also a real project wrapper in DRBD that invokes the generator
directly with -d [2]. It scans the directory without passing modules.order,
so it is not evidence of this particular failure.

I do not currently have a report of this failure in a real-world workflow.

[1] https://marc.info/?l=linux-kbuild&m=179063910090741&w=1
[2] https://github.com/LINBIT/drbd/blob/6d3787572ac95313afe7d139df7b24d589a00a6b/misc/gen-compile-commands.sh

Thanks,
Võ Ngọc Văn

On Fri, Oct 16, 2026 at 8:42 PM Nicolas Schier <n.schier@xxxxxxxxx> wrote:
>
> On Fri, Oct 16, 2026 at 01:25:33PM +0700, Van Vo wrote:
> > Hi Nathan,
> >
> > > How was this discovered?
> >
> > This came from code inspection of standalone use of -d with an explicit
> > modules.order argument, followed initially by a constructed reproducer.
> > It was not a failure encountered through Kbuild's compile_commands.json
> > target.
> >
> > I have now reproduced it using real Kbuild-generated artifacts from
> > 0fcb05b72344f40806cbe810de8872f1698bf053 on Linux/aarch64, with Python
> > 3.11.13 and GCC 12.2.0. This was a partial lib/ build with CONFIG_TEST_LKM=m.
> >
> > The following uses a disposable source tree at that base. The guards
> > prevent overwriting an existing output directory or result file:
> >
> >     (
> >         set -eu
> >         test ! -e out
> >         test ! -e outside.json
> >         test ! -e inside.json
> >         make O=out -j1 allnoconfig
> >         bash scripts/config --file out/.config --enable MODULES \
> >             --enable RUNTIME_TESTING_MENU --module TEST_LKM
> >         make O=out -j1 olddefconfig
> >         make O=out -j1 V=1 lib/
> >
> >         outside_exit=0
> >         python3 -B scripts/clang-tools/gen_compile_commands.py -a ar \
> >             -d out -o outside.json out/lib/modules.order ||
> >             outside_exit=$?
> >         printf 'outside_exit=%s\n' "$outside_exit"
> >
> >         inside_exit=0
> >         (cd out &&
> >          python3 -B ../scripts/clang-tools/gen_compile_commands.py -a ar \
> >              -o ../inside.json lib/modules.order) ||
> >             inside_exit=$?
> >         printf 'inside_exit=%s\n' "$inside_exit"
> >         test "$outside_exit" -eq 1
> >         test "$inside_exit" -eq 0
> >         python3 -c \
> >             'import json; assert len(json.load(open("inside.json"))) == 1'
> >     )
> >
> > Kbuild generated lib/test_module.o, lib/test_module.mod,
> > lib/.test_module.o.cmd and lib/modules.order under out/. Both .mod and
> > modules.order contain lib/test_module.o.
> >
> > The old script's invocation from the source directory fails with:
> >
> >     FileNotFoundError: [Errno 2] No such file or directory:
> > 'lib/test_module.mod'
> >     outside_exit=1
> >
> > The same artifacts work when the old script is invoked from out/:
> >
> >     inside_exit=0
> >
> > The successful invocation produces one JSON entry for lib/test_module.c.
> > The patched script succeeds from the source directory and produces an
> > identical entry. The old script also fails with absolute paths for -d
> > and modules.order; the patched script succeeds with those paths too.
> >
> > Only the standalone generator invocation is outside the build directory;
> > compilation itself runs under out/. This is not a full kernel build or
> > a boot/runtime test.
>
> Thanks for the detailed reproducer.
>
> Quoting from Nathan:
> | How was this discovered? This cannot happen within Kbuild because this
> | script is always invoked from within the build folder, so was it a
> | manual invocation or something else?
>
> So, if I get youre reply correctly, the issue this was not discovered
> "in the wild" but can only be reproduced by manual use of
> gen_compile_commands.py.
>
> --
> Nicolas