Re: [PATCH] ipmi:msghandler: Free queued requests of a destroyed user
From: Corey Minyard
Date: Sat Nov 21 2026 - 20:59:16 EST
On Fri, Nov 20, 2026 at 07:15:49PM -0800, Shakeel Butt wrote:
> In the Meta fleet we found hosts whose BMC had stopped answering IPMI
> requests while a power management daemon kept polling it:
>
> - the daemon runs "timeout 2 ipmitool raw 0x30 0x7e" every two seconds;
> - every run leaves one request and one closed IPMI user behind;
> - one host held over a million closed users after 25 days, using about
> 33 GiB and growing by 1.3 GiB a day.
>
> ipmitool opens /dev/ipmi0, sends a request and waits for the answer.
> When timeout kills it, the file is closed and ipmi_destroy_user() runs.
> A request that is still waiting on intf->xmit_msgs or intf->hp_xmit_msgs
> stays there, and its receive message keeps a reference on the user.
> With a BMC that does not answer, the lower layer is slow to fail each
> request: about 15 seconds of retries for SSIF, and up to a minute for
> KCS on kernels without the hosed BMC handling from
> commit bc3a9d217755 ("ipmi:si: Gracefully handle if the BMC is non-functional").
Oops, that's not good. This is in my next queue for the next release
and I added a stable backport.
Thanks,
-corey
> Requests arrive faster than they leave, so the queue, the users and
> their messages grow until the BMC answers again.
>
> Nobody can receive these responses once the user is destroyed. Free the
> user's requests that were not handed to the lower layer yet. Such a
> request is then never sent to the BMC. This only matters for a program
> that sends a request and closes the device without waiting for the
> answer, and with a working BMC a request waits in these lists only
> while another request is being handled.
>
> The request the lower layer is working on (intf->curr_msg) is left
> alone. IPMB and LAN commands do not point to their receive message, so
> they are not touched here. Their receive messages are freed by the
> sequence table loop just above, as before.
>
> Tested in QEMU with a fake BMC that stops answering and four loops of
> "timeout 2 ipmitool raw 0x30 0x7e", with KASAN and lockdep enabled:
>
> - SSIF on linux-next: 465 closed users were held after 4 minutes,
> growing by about 116 a minute. With this patch, 5 are held (the four
> live callers and the request in flight), and the number stays flat.
> - KCS on v6.16, which lacks the hosed BMC handling, with this change
> adapted: 476 held after 4 minutes without it, 5 with it.
> - KCS on linux-next: at most 5 held instead of up to 28.
> - With a working BMC, 1600 requests from 8 parallel ipmitool loops all
> succeed over KCS and SSIF, and nothing is left held afterwards.
>
> QEMU needed small changes to behave like the failing BMC: ipmi-bmc-extern
> makes up a timeout reply after 4 seconds, its KCS keeps IBF set while
> the BMC has not answered, and smbus-ipmi returns the previous reply
> instead of NAKing the read.
>
> Fixes: 7ea0ed2b5be8 ("ipmi: Make the message handler easier to use for SMI interfaces")
> Signed-off-by: Shakeel Butt <shakeel.butt@xxxxxxxxx>
> ---
> drivers/char/ipmi/ipmi_msghandler.c | 41 +++++++++++++++++++++++++++++
> 1 file changed, 41 insertions(+)
>
> diff --git a/drivers/char/ipmi/ipmi_msghandler.c b/drivers/char/ipmi/ipmi_msghandler.c
> index 152dd6b5bb27..52e95e1949b9 100644
> --- a/drivers/char/ipmi/ipmi_msghandler.c
> +++ b/drivers/char/ipmi/ipmi_msghandler.c
> @@ -1339,6 +1339,45 @@ int ipmi_get_smi_info(int if_num, struct ipmi_smi_info *data)
> }
> EXPORT_SYMBOL(ipmi_get_smi_info);
>
> +static void move_user_xmit_msgs(struct list_head *from, struct list_head *to,
> + struct ipmi_user *user)
> +{
> + struct ipmi_smi_msg *msg, *msg2;
> +
> + list_for_each_entry_safe(msg, msg2, from, link) {
> + if (msg->recv_msg && msg->recv_msg->user == user)
> + list_move_tail(&msg->link, to);
> + }
> +}
> +
> +/*
> + * Free the requests of a user that are still waiting to be handed to the
> + * lower layer. Once the user is destroyed, nobody can receive their
> + * responses. If the BMC is slow or does not answer, they could otherwise
> + * stay queued for a long time, each one keeping the user and its own
> + * messages allocated. A request that was already handed to the lower
> + * layer (intf->curr_msg) is not on these lists and is left alone.
> + */
> +static void remove_user_xmit_msgs(struct ipmi_smi *intf,
> + struct ipmi_user *user)
> +{
> + int run_to_completion = READ_ONCE(intf->run_to_completion);
> + struct ipmi_smi_msg *msg, *msg2;
> + unsigned long flags = 0;
> + LIST_HEAD(msgs);
> +
> + ipmi_lock_xmit_msgs(intf, run_to_completion, &flags);
> + move_user_xmit_msgs(&intf->hp_xmit_msgs, &msgs, user);
> + move_user_xmit_msgs(&intf->xmit_msgs, &msgs, user);
> + ipmi_unlock_xmit_msgs(intf, run_to_completion, &flags);
> +
> + list_for_each_entry_safe(msg, msg2, &msgs, link) {
> + list_del(&msg->link);
> + ipmi_free_recv_msg(msg->recv_msg);
> + ipmi_free_smi_msg(msg);
> + }
> +}
> +
> /* Must be called with intf->users_mutex held. */
> static void _ipmi_destroy_user(struct ipmi_user *user)
> {
> @@ -1375,6 +1414,8 @@ static void _ipmi_destroy_user(struct ipmi_user *user)
> }
> mutex_unlock(&intf->seq_lock);
>
> + remove_user_xmit_msgs(intf, user);
> +
> /*
> * Remove the user from the command receiver's table. First
> * we build a list of everything (not using the standard link,
> --
> 2.53.1-Meta
>