Re: [BUG] crypto: caam - RSA encrypt doesn't always complete new data in out_buf

From: Kepplinger-Novakovic Martin

Date: Tue Mar 17 2026 - 07:46:12 EST

Am Freitag, dem 13.03.2026 um 10:18 +0100 schrieb Lukas Wunner:
> On Tue, Mar 10, 2026 at 08:57:36AM +0000, Kepplinger-Novakovic Martin wrote:
> > Am Donnerstag, dem 26.02.2026 um 14:27 +0100 schrieb Lukas Wunner:
> > > There's an endianness issue here: 30313000 is the zero byte prescribed
> > > by EMSA-PKCS1-v1_5 ("in_buf[ps_end] = 0x00;" in rsassa_pkcs1_sign()),
> > > followed by the first three bytes of hash_prefix_sha256[] in reverse order.
> > >
> > > Then 6009060d are the next four bytes of hash_prefix_sha256[], again
> > > in reverse order. And so on until 20040005, which are the last four
> > > bytes of the prefix in reverse order.
> > >
> > > How are you generating that hexdump? What's the CPU's endianness?
> > > Is the caam RSA accelerator using a different endianness?
> >
> > imx6ul is armv7, little endian byte order and the following returns 1
> > which supports that:
> > echo -n I | od -o | head -n1 | cut -f2 -d" " | cut -c6
>
> Please double-check whether your .config enables CONFIG_CPU_BIG_ENDIAN
> or CONFIG_CPU_LITTLE_ENDIAN, just to cover all bases.

CONFIG_CPU_LITTLE_ENDIAN is enabled, not CPU_BIG_ENDIAN. thanks for the hint.

>
> > I always print the hex dump in the following way (here "out_buf" at line
> > https://elixir.bootlin.com/linux/v6.19.6/source/crypto/rsassa-pkcs1.c#L247
> > )
> > print_hex_dump(KERN_ERR, "out_buf1:", DUMP_PREFIX_OFFSET, 16, 4, out_buf, 32, true);
>
> Please use 1 instead of 4 as 5th parameter of print_hex_dump().
> Using 4 only makes sense if the memory location you want to dump
> contains 32-bit values. That's not the case here as the signature
> is a bytestream.
>
> I guess if you use 4, print_hex_dump() dumps the 32-bit values
> in big endian order for human readability, but that's confusing
> if the memory location actually contains a bytestream.

true, thanks, there would have been other debug examples I should have looked at. Changed this, see below.

>
> > Again, with this revert, the problem seems to be the same, only that
> > the data that rsassa_pkcs1_verify() is starting to check here
> > https://elixir.bootlin.com/linux/v6.19.6/source/crypto/rsassa-pkcs1.c#L266
> > is still "old" but now zeroes, not the input-data, thus failing with
> > -EBADMSG instead of -EINVAL.
>
> Actually the "out_buf2" that you've included in this message...
>
> https://lore.kernel.org/all/1a65ac92579fadb4bfc76b32a3a4f1c6df022801.camel@xxxxxxxxxxxxx/
>
> ...looks like a valid verified (i.e. encrypted) signature,
> the only thing that's weird is the endianness issue and
> that there's a bunch of zero bytes at the beginning of
> the buffer.

yes. As mentioned in this thread's very first email, always only the first 16 bytes of out_buf stay old, thus wrong, after crypto_req_done(). With
commit 8552cb04e083 reverted, it's 16 bytes of zeroes, but otherwise no change, see the dumps below.

>
> Please re-generate the hexdump of "out_buf" after the call
> to crypto_wait_req(), once with a stock kernel and once with
> 8552cb04e083 reverted, and use 1 as 5th argument to
> print_hex_dump().

so, nothing new, but a failure-case dumped as byte-stream running a stock kernel (the "usual" caam config without ahash api...)

[ 2.339406] start rsassa_pkcs1_verify
[ 2.339413] slen: 256
[ 2.339425] child_req address: bc6b492f full size: 64 + 48 + 256 = 368
[ 2.339455] out_buf1:00000000: 87 03 da f2 82 c2 dd af 7c 44 2f 86 d3 5f 4c 93 ........|D/.._L.
[ 2.339476] out_buf1:00000010: 48 b9 fe 07 17 bb 21 f7 25 23 4e aa 22 0c 16 b9 H.....!.%#N."...
[ 2.339500] SRC BUF in out_buf1 CRC: 60791b87
[ 2.339513] start caam_rsa_enc
[ 2.339523] key:00000000: 00 c8 62 cf 40 74 62 cf 00 00 00 00 00 00 00 00 ..b.@tb.........
[ 2.339540] key:00000010: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
[ 2.339567] edesc:00000000: 01 00 00 00 01 00 00 00 00 00 00 00 00 00 00 00 ................
[ 2.339584] edesc:00000010: 00 00 00 00 00 00 00 00 00 00 00 00 ec 7e 49 cf .............~I.
[ 2.722145] req:00000000: 00 00 00 00 00 00 00 00 88 22 2e c0 a4 dc 83 d0 ........."......
[ 2.722178] req:00000010: 40 c7 62 cf 00 02 00 00 b4 dc 83 d0 b4 dc 83 d0 @.b.............
[ 2.722200] CAAM: calling caam_jr_enqueue
[ 2.722210] key:00000000: 00 c8 62 cf 40 74 62 cf 00 00 00 00 00 00 00 00 ..b.@tb.........
[ 2.722227] key:00000010: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
[ 2.727096] CAAM: completion callback
[ 2.742603] OUT BUF in out_buf2 CRC: 12298efd
[ 2.742622] out_buf2:00000000: 87 03 da f2 82 c2 dd af 7c 44 2f 86 d3 5f 4c 93 ........|D/.._L.
[ 2.742643] out_buf2:00000010: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
[ 2.742659] out_buf2:00000020: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
[ 2.742675] out_buf2:00000030: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
[ 2.742691] out_buf2:00000040: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
[ 2.742706] out_buf2:00000050: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
[ 2.742722] out_buf2:00000060: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
[ 2.742737] out_buf2:00000070: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
[ 2.742753] out_buf2:00000080: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
[ 2.742769] out_buf2:00000090: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
[ 2.742785] out_buf2:000000a0: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
[ 2.742801] out_buf2:000000b0: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
[ 2.742818] out_buf2:000000c0: ff ff ff ff ff ff ff ff ff ff ff ff 00 30 31 30 .............010
[ 2.742833] out_buf2:000000d0: 0d 06 09 60 86 48 01 65 03 04 02 01 05 00 04 20 ...`.H.e.......
[ 2.742850] out_buf2:000000e0: 69 f0 98 59 7f 4b b0 81 8d 11 88 be e7 29 a9 3e i..Y.K.......).>
[ 2.742866] out_buf2:000000f0: 25 5b 37 86 1c 03 83 8b 55 31 b7 94 77 71 a3 fa %[7.....U1..wq..
[ 2.742883] digest (in):00000000: 69 f0 98 59 7f 4b b0 81 8d 11 88 be e7 29 a9 3e i..Y.K.......).>
[ 2.742900] digest (in):00000010: 25 5b 37 86 1c 03 83 8b 55 31 b7 94 77 71 a3 fa %[7.....U1..wq..
[ 2.742915] Encrypted value had no leading 0 byte.
[ 2.742927] PKEY: crypto_sig_verify error: -22
[ 2.742947] PKEY: <==public_key_verify_signature() = -22
[ 2.742961] X.509: public_key_verify_signature error: -22
[ 2.742970] X.509: <==x509_check_for_self_signed() = -22
[ 2.742983] X.509: x509_check_for_self_signed failed: -22
[ 2.743000] X.509: x509_cert_parse failed: -22
[ 2.743009] Parser recognised the format (ret -22)
[ 2.743022] <==asymmetric_key_preparse() = -22
[ 2.743035] preparse: -22
[ 2.743643] Problem loading in-kernel X.509 certificate (-22)

a failure-case dumped running 8552cb04e083 reverted:

[ 2.307228] start rsassa_pkcs1_verify
[ 2.307236] slen: 256
[ 2.307249] child_req address: f56af4d7 full size: 64 + 48 + 256 = 368
[ 2.307285] out_buf1:00000000: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
[ 2.307306] out_buf1:00000010: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
[ 2.307330] SRC BUF in out_buf1 CRC: 3e37f250
[ 2.307346] start caam_rsa_enc
[ 2.307355] key:00000000: 00 f8 5f cf 80 a3 5f cf 00 00 00 00 00 00 00 00 .._..._.........
[ 2.307372] key:00000010: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
[ 2.307401] edesc:00000000: 01 00 00 00 01 00 00 00 00 00 00 00 00 00 00 00 ................
[ 2.307419] edesc:00000010: 00 00 00 00 00 00 00 00 00 00 00 00 6c 7e 3d c1 ............l~=.
[ 2.307440] req:00000000: 00 00 00 00 00 00 00 00 88 32 2e c0 b4 dc 83 d0 .........2......
[ 2.307458] req:00000010: 40 f7 5f cf 00 02 00 00 94 dc 83 d0 a4 dc 83 d0 @._.............
[ 2.307476] CAAM: calling caam_jr_enqueue
[ 2.307485] key:00000000: 00 f8 5f cf 80 a3 5f cf 00 00 00 00 00 00 00 00 .._..._.........
[ 2.307502] key:00000010: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
[ 2.308771] CAAM: completion callback
[ 2.308840] OUT BUF in out_buf2 CRC: 86440841
[ 2.308860] out_buf2:00000000: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
[ 2.308881] out_buf2:00000010: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
[ 2.308898] out_buf2:00000020: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
[ 2.308914] out_buf2:00000030: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
[ 2.308930] out_buf2:00000040: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
[ 2.308947] out_buf2:00000050: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
[ 2.308963] out_buf2:00000060: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
[ 2.308980] out_buf2:00000070: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
[ 2.308997] out_buf2:00000080: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
[ 2.309013] out_buf2:00000090: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
[ 2.309031] out_buf2:000000a0: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
[ 2.309048] out_buf2:000000b0: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ................
[ 2.309064] out_buf2:000000c0: ff ff ff ff ff ff ff ff ff ff ff ff 00 30 31 30 .............010
[ 2.309082] out_buf2:000000d0: 0d 06 09 60 86 48 01 65 03 04 02 01 05 00 04 20 ...`.H.e.......
[ 2.309099] out_buf2:000000e0: 69 f0 98 59 7f 4b b0 81 8d 11 88 be e7 29 a9 3e i..Y.K.......).>
[ 2.309116] out_buf2:000000f0: 25 5b 37 86 1c 03 83 8b 55 31 b7 94 77 71 a3 fa %[7.....U1..wq..
[ 2.309133] digest (in):00000000: 69 f0 98 59 7f 4b b0 81 8d 11 88 be e7 29 a9 3e i..Y.K.......).>
[ 2.309150] digest (in):00000010: 25 5b 37 86 1c 03 83 8b 55 31 b7 94 77 71 a3 fa %[7.....U1..wq..
[ 2.309168] PKEY: crypto_sig_verify error: -74
[ 2.309188] PKEY: <==public_key_verify_signature() = -74
[ 2.309201] X.509: public_key_verify_signature error: -74
[ 2.309212] X.509: <==x509_check_for_self_signed() = -74
[ 2.309225] X.509: x509_check_for_self_signed failed: -74
[ 2.309240] X.509: x509_cert_parse failed: -74
[ 2.309252] <==asymmetric_key_preparse() = -74
[ 2.309268] preparse: -74
[ 2.413845] Problem loading in-kernel X.509 certificate (-74)

thank you very much for having a look! I'd be happy to test any thoughts you might have.

martin