[PATCH RFC 0/2] net/mlx5: Cut workqueue traffic of firmware commands and stats

[Breno Leitao]

While looking for Meta's arm64 fleet wide lockstat, I found that mlx5 is
one of the major culprit for the locks in the fleet, and this is the
motivation for this patchset.

On an arm64 host with a dozen mlx5 functions in switchdev mode, lock
contention profiling showed the pool->lock of the unbound worker pool
that all ordered workqueues share as the hottest workqueue lock.
Monitoring daemons read /proc/net/dev, so every mlx5e netdev keeps
queueing update_stats_work. Each refresh sends one to three firmware
commands, and each command is one more work item, on the ordered
cmd->wq of its function.

Patch 1 submits blocking firmware commands from the caller's context,
as page-queue commands already do, and keeps the workqueue for
callback commands.

Patch 2 brings back the 200 msecs stats refresh interval that the
driver used until 2018, as the minimum time between two on-demand
refreshes.

Tested on a ConnectX-4 Lx host (x86, firmware 14.32) against the base
commit, in legacy mode and in switchdev mode with 8 VFs. With one
/proc/net/dev reader in switchdev mode, the firmware commands of a
13 second run drop from 13.4k to 1.8k, and the mlx5_cmd work items of
32 parallel ethtool -S loops drop from 46.7k to none. Channel
reconfiguration, switchdev enable/disable and the load above ran with
no failed firmware commands and no warnings.

Signed-off-by: Breno Leitao <leitao@xxxxxxxxxx>
---
Breno Leitao (2):
      net/mlx5: Submit blocking commands from the caller's context
      net/mlx5e: Rate limit the HW stats refresh from ndo_get_stats64

 drivers/net/ethernet/mellanox/mlx5/core/cmd.c     | 3 ++-
 drivers/net/ethernet/mellanox/mlx5/core/en.h      | 2 ++
 drivers/net/ethernet/mellanox/mlx5/core/en_main.c | 9 +++++++++
 3 files changed, 13 insertions(+), 1 deletion(-)
---
base-commit: 4dd16e31135ea8437a720222df34e8745f385d6d
change-id: 20261030-mlx5-cmd-wq-a156836104af

Best regards,
--  
Breno Leitao <leitao@xxxxxxxxxx>