[PATCH] x86/rust: use `-Zindirect-branch-cs-prefix` for RETPOLINE builds

[Miguel Ojeda]

The C compiler is asked to add a `cs` prefix to the `call` and `jmp`
instructions that go through the indirect thunks for the `r8`-`r15`
registers, which makes them 6 bytes long instead of 5. That gives
`patch_retpoline()` enough room to inline them, e.g. as `lfence; call
*%r11` for `spectre_v2=retpoline,lfence`.

However, Rust code does not get the prefix, e.g. for a
`CONFIG_MITIGATION_RETPOLINE=y` build:

    $ objdump -dr rust/kernel.o
    ...
      e2b: e8 00 00 00 00        call   e30 <...>
                e2c: R_X86_64_PLT32  __x86_indirect_thunk_r11-0x4

Moreover, since commit 4a1e02b15ac1 ("x86,retpoline: Optimize
patch_retpoline()"), `emit_indirect()` assumes there is enough room
for the inlined sequence -- e.g. with `spectre_v2=retpoline,lfence`,
the kernel warns and dies while patching a site inside `core`:

    WARNING: arch/x86/kernel/alternative.c:736 at apply_retpolines+0x2ee/0x400, CPU#0: swapper/0/0
    ...
    Call Trace:
     <TASK>
     ? _RNvMs6_NtNtCshS2CUL9MEDn_4core3fmt8buildersNtB5_9DebugList21finish_non_exhaustive+0x5b/0xd0
     ...
     alternative_instructions+0x1d/0x60
     arch_cpu_finalize_init+0x5a/0xb0
     start_kernel+0x1f4/0x280

Similarly, paranoid FineIBT rewrites each kCFI call site into a fixed
20-byte sequence, i.e. the 14 bytes of the kCFI check plus a 6-byte
call. For a 5-byte call, that overwrites the first byte of the next
instruction, corrupting Rust code:

    SMP alternatives: Using paranoid FineIBT CFI
    ...
    Oops: double fault: 0000 [#1]
    ...
    RIP: 0010:_RNvNtCsdVypMZsLRAh_4core3fmt5write+0x1e9/0x210
    Code: ... b8 68 81 16 9f 41 3b 43 f5 2e 4d 8d 5b f0 75 fd 41 ff e3 <cc> c0 eb 02 b0 01 ...
    ...
    Kernel panic - not syncing: Fatal exception in interrupt

Non-paranoid FineIBT is not affected, since it only rewrites the kCFI
check and leaves the call itself to `apply_retpolines()`.

We added support for `-Zindirect-branch-cs-prefix` [1][2] in Rust 1.91.0
[3]. Thus pass it for `RETPOLINE` builds when possible, which makes all
sites indeed get the prefix:

      e3c: 2e e8 00 00 00 00     cs call e42 <...>
                e3e: R_X86_64_PLT32  __x86_indirect_thunk_r11-0x4

The kernel above boots fine with `spectre_v2=retpoline,lfence`, inlining
the Rust sites as well.

For older Rust compilers, set the LLVM module flag instead, which
has the same code-generation effect. This means that every Rust version
that the kernel supports gets the prefix, that the kernel boots fine with
`spectre_v2=retpoline,lfence`, and that paranoid FineIBT no longer
corrupts.

In addition, add it to our target modifier workaround for `rustdoc`.

Cc: stable@xxxxxxxxxxxxxxx # 6.18.y+
Link: https://github.com/rust-lang/rust/issues/116852 [1]
Link: https://github.com/rust-lang/compiler-team/issues/899 [2]
Link: https://github.com/rust-lang/rust/pull/140740 [3]
Reported-by: Kent Overstreet <kent.overstreet@xxxxxxxxx>
Assisted-by: LLM
Signed-off-by: Miguel Ojeda <ojeda@xxxxxxxxxx>
---
 arch/x86/Makefile | 1 +
 rust/Makefile     | 2 +-
 2 files changed, 2 insertions(+), 1 deletion(-)

diff --git a/arch/x86/Makefile b/arch/x86/Makefile
index 8af6b80cffdd..01c848855f99 100644
--- a/arch/x86/Makefile
+++ b/arch/x86/Makefile
@@ -21,6 +21,7 @@ RETPOLINE_CFLAGS	:= -mretpoline-external-thunk
 RETPOLINE_VDSO_CFLAGS	:= -mretpoline
 endif
 RETPOLINE_CFLAGS	+= $(call cc-option,-mindirect-branch-cs-prefix)
+RETPOLINE_RUSTFLAGS	+= $(if $(call rustc-min-version,109100),-Zindirect-branch-cs-prefix,-Zllvm_module_flag=indirect_branch_cs_prefix:u32:1:override)
 
 ifdef CONFIG_MITIGATION_RETHUNK
 RETHUNK_CFLAGS		:= -mfunction-return=thunk-extern
diff --git a/rust/Makefile b/rust/Makefile
index da1a7409d984..35868a47e18b 100644
--- a/rust/Makefile
+++ b/rust/Makefile
@@ -167,7 +167,7 @@ pin_init-flags := \
 rustdoc_modifiers_workaround := $(if $(call rustc-min-version,108800),-Cunsafe-allow-abi-mismatch=fixed-x18)
 
 # Similarly, for doctests (https://github.com/rust-lang/rust/issues/146465).
-doctests_modifiers_workaround := $(rustdoc_modifiers_workaround)$(if $(call rustc-min-version,109100),$(comma)sanitizer)
+doctests_modifiers_workaround := $(rustdoc_modifiers_workaround)$(if $(call rustc-min-version,109100),$(comma)sanitizer$(comma)indirect-branch-cs-prefix)
 
 quiet_cmd_rustdoc = RUSTDOC $(if $(rustdoc_host),H, ) $<
       cmd_rustdoc = \

base-commit: 24a19b953e2a3d0ea5a8c4951a0ab8f32720c907
-- 
2.56.0