[PATCH net 0/3] enic: fence DMA ownership and identify V2 VFs

[Satish Kharat]

An automated Sashiko review of the V2 SR-IOV reset paths prompted a
broader audit of ENIC queue teardown. That audit found three prerequisite
correctness problems in code which is already upstream.

First, a datapath WQ or RQ disable can time out while hardware may still
own descriptors and their DMA mappings. Patch 1 replaces iteration-counted
polling with a bounded 500 ms RUNNING poll for each queue. The RQ keeps its
required two disable-and-poll passes. The datapath graph is released only
after every applicable queue reports RUNNING == 0. Because rings, mapped
buffers, page pools, and shared state form a coupled ownership graph, it is
not partially released when only some queues stop.

The queue-disable helpers can return -ETIMEDOUT without observing
RUNNING == 0. Firmware defines RUNNING == 0 as conclusive evidence that
queue DMA has stopped. A nonzero RUNNING value does not establish that
DMA is active, but it leaves DMA ownership unresolved. The existing error
path can nevertheless release DMA-backed resources without obtaining the
required ownership proof.

This series deliberately retains the complete datapath graph if any
datapath queue does not report RUNNING == 0. This can retain resources
belonging to queues which did stop, but keeps the ownership rule simple
in an exceptional failure path. Reclaiming packet buffers individually for
queues which successfully stop is a possible follow-up optimization; it
is not required to prevent DMA into freed memory.

If a queue does not report RUNNING == 0, the whole datapath graph is
quarantined. A final direct queue recheck runs during device removal,
before terminal containment. The graph is released if every queue reports
RUNNING == 0 by then; otherwise it is retained until reboot. Firmware soft
reset, hang reset, and PCI function reset are not treated as independent
ownership proof for a queue whose stop could not be confirmed.

The terminal path detaches CPU entry points, clears bus mastering, disables
the PCI function, and intentionally retains the netdev, vnic_dev, BAR
mappings and reservations, rings, mapped buffers, page pools, and graph
metadata until reboot. Clearing bus mastering blocks new device-initiated
PCI transactions; it does not cancel DMA already in progress or prevent a
queue from draining to RUNNING == 0. The final remove-time recheck
therefore remains meaningful, but bus-master state alone does not prove
that a queue with unresolved DMA ownership relinquished its old targets.
The BAR reservation may outlive driver unbind and module unload while the
PCI device remains enumerated. The resource subsystem stores the supplied
owner-name pointer rather than copying it, so the reservation uses the PCI
device name instead of module rodata.

A PCI driver's remove callback cannot reject driver unbind. With DMA API
debugging enabled, the driver's deliberate retention therefore produces the
expected "pending DMA allocations while released from device" diagnostic
if the module is unloaded after terminal quarantine. The retained BAR
reservation prevents a normal ENIC rebind; reboot is the supported
recovery.

Ring resizing needs an explicit stop because dev_close() cannot propagate
an ndo_stop() error. Patch 1 returns a queue-disable failure to ethtool and
rejects later reconfiguration after quarantine. It also makes reset, IRQ,
NAPI, notification-buffer, and final-remove ownership transitions
idempotent.

Second, an admin-WQ send can time out while hardware may still own the
descriptor and mapping. Patch 2 applies the same bounded polling,
observed-queue-stop rule, and final remove-time recheck to the admin WQ/RQ
graph. The datapath and admin channel are independent complete graphs: a
clean graph may be released even when the other is quarantined, but an
individual graph is never partially released. Shared objects needed by an
unresolved DMA target remain retained. Putting the generic repair first
makes every committed state independently safe.

Third, the in-tree V2 VF always transmits mailbox source zero. Deployed
async PFs compare that field with sender identity from CQ metadata, so a
nonzero VF cannot register. Patch 3 obtains firmware's logical VF index
with CMD_SRIOV_STATS_GET after the admin channel is live and uses it as the
source of subsequent V2 VF mailbox messages.

Patch 3 also requires DEVCMD2 for SR-IOV V2. A V2 VF without DEVCMD2
fails probe before opening its admin channel. A PF which fell back to
DEVCMD1 keeps ordinary Ethernet operation but does not expose V2 SR-IOV;
V1 and USNIC keep their existing DEVCMD1 fallback. With DEVCMD2, -ENODEV
is detected before a descriptor is posted, so only a timeout can leave
ownership of the persistent statistics-command DMA target uncertain. A
later successful blocking command for the same vNIC proves that target
completed; otherwise it remains retained with the shared vNIC lifetime
envelope until reboot.

The companion V2 VF receive-control v3 series carries explicit prerequisite
patch IDs for all three patches.

AI assistance:

An LLM was used for design review, source review, commit-message drafting,
test automation, and review triage. Sashiko was used as an additional
review tool. All findings and generated changes were manually reviewed, and
the author takes responsibility for the series.

Validation:

- Tests used Cisco VIC PF and V2 VF functions. PF/VF probe, open/close,
  repeated clean module unload/reload, link cycling, ring resizing, jumbo
  MTU, and bidirectional ICMP, TCP, and UDP traffic passed.
- A nonzero V2 VF registered and passed traffic with the deployed async PF.
  Before patch 3, the PF rejected the VF's zero mailbox source because it
  did not match the sender identity obtained from CQ metadata.
- The full no-reboot suite exercised exactly 32 secondary unicast
  addresses, 33-address fallback, address churn, station-address
  multicast-list changes, administrative-MAC policy, 25 module reloads,
  10 ring resize cycles, 10 reopen cycles, and five VF destroy/recreate
  cycles.
- A test-only admin-channel overlay injected one false first-attempt WQ
  disable failure and then one false first-attempt RQ disable failure after
  the corresponding real hardware disable had succeeded. Each failed helper
  alone was retried, each retry succeeded, and the production module
  then passed bidirectional traffic.
- A test-only V2 VF overlay forced exactly one posted mailbox send timeout.
  Serialized reset recovery rebuilt the admin channel, refreshed the
  administrative MAC, restored bidirectional traffic, and completed a later
  exact-filter transaction.
- A disposable-VF-guest test overlay held the selected WQ or RQ's observed
  RUNNING value high without preventing the real hardware disable. A 300 ms
  hold verified that both queue types wait rather than time out early. A
  hold through the 500 ms bound quarantined each datapath and admin queue
  path. Clearing the synthetic condition before removal let the final
  recheck release each complete graph. Leaving it set exercised terminal
  retention. The uninstrumented candidate then bound and passed
  bidirectional ENIC traffic after guest reboot.

Queue-stop failure handling was exercised using instrumentation which
kept the RUNNING value observed by the driver nonzero after the underlying
hardware queue had stopped. This exercised the bounded polling,
quarantine, retained-resource, and final remove-time recheck paths. It
validates the driver's handling when RUNNING == 0 is not observed; it does
not attempt to reproduce or characterize the underlying hardware condition
which produces that result. If ownership remains unresolved at removal,
reboot is the supported recovery.

The DEVCMD2 requirement and pre-admin-channel V2 VF rejection were
verified by source review and exact per-commit builds. The physical test
adapter uses DEVCMD2, so the no-DEVCMD2 rejection path was not exercised
on hardware.

Link: https://patch.msgid.link/20260929-b4-enic-sriov-v2-vf-receive-control-v2-0-5962c10061b2@xxxxxxxxx
Signed-off-by: Satish Kharat <satishkh@xxxxxxxxx>
---
Satish Kharat (3):
      enic: retain datapath DMA resources until queues stop
      enic: retain admin-channel DMA until queues stop
      enic: require DEVCMD2 and set the V2 VF mailbox source

 drivers/net/ethernet/cisco/enic/enic.h         |  36 +-
 drivers/net/ethernet/cisco/enic/enic_admin.c   | 182 ++++++---
 drivers/net/ethernet/cisco/enic/enic_admin.h   |   4 +-
 drivers/net/ethernet/cisco/enic/enic_dev.c     |  15 +
 drivers/net/ethernet/cisco/enic/enic_dev.h     |   2 +
 drivers/net/ethernet/cisco/enic/enic_ethtool.c |  10 +-
 drivers/net/ethernet/cisco/enic/enic_main.c    | 508 +++++++++++++++++++++----
 drivers/net/ethernet/cisco/enic/enic_mbox.c    |  13 +-
 drivers/net/ethernet/cisco/enic/vnic_dev.c     |  42 ++
 drivers/net/ethernet/cisco/enic/vnic_dev.h     |   5 +
 drivers/net/ethernet/cisco/enic/vnic_devcmd.h  |  34 ++
 drivers/net/ethernet/cisco/enic/vnic_rq.c      |  18 +-
 drivers/net/ethernet/cisco/enic/vnic_wq.c      |  18 +-
 13 files changed, 735 insertions(+), 152 deletions(-)
---
base-commit: 6d25ffca055a77787c21a36b66c253f76239411b
change-id: 20261010-b4-enic-dma-vf-index-net-generic-first-v1-20261010-d9abf4510335

Best regards,
--  
Satish Kharat <satishkh@xxxxxxxxx>