[PATCH v2 1/2] PCI/VGA: fix inverted vga_tryget() return check in vga_arb_write()

[Hui Peng]

In vga_arb_write(), the "trylock" command handler calls
vga_tryget(pdev, io_state) and checks:

  if (vga_tryget(pdev, io_state)) {
      /* Update the client's locks lists... */

However, vga_tryget() returns 0 on success and a negative errno (-EBUSY)
when the VGA resources are already locked. Because the check tests for a
non-zero return value, a successful vga_tryget() skips incrementing
priv->cards[i].io_cnt / mem_cnt and skips reporting success, leaking the
acquired VGA lock, whereas a failed vga_tryget() (-EBUSY) erroneously
records the lock in priv->cards[i] so a subsequent "unlock" or file
release decrements the lock count without holding the lock.

Check if (vga_tryget(pdev, io_state) == 0) in vga_arb_write().

Tested in QEMU against Linux 7.3.0-rc3 by opening /dev/vga_arbiter and
writing "trylock io+mem" followed by closing the file descriptor. On the
unfixed kernel, write("trylock io+mem") acquires the VGA lock
(locks=io+mem(1:1)) but takes the error branch and returns -EBUSY (-16)
without recording the lock in priv->cards[], permanently leaking the
lock on close(); with this patch applied, write("trylock io+mem")
succeeds (ret = 14) and close() cleanly releases the lock
(locks=none(0:0)).

Fixes: deb2d2ecd43d ("PCI/GPU: implement VGA arbitration on Linux")
Cc: stable@xxxxxxxxxxxxxxx
Assisted-by: LLM
Signed-off-by: Hui Peng <benquike@xxxxxxxxx>
---
Changes in v2:
- Split the vga_tryget() return check fix in vga_arb_write() from the
  PCI_INVALID_CARD hot-unplug cleanup fix into a 2-patch series, and fix
  the commit message to reference vga_arb_write(), as noted by Sashiko.
- Update Fixes: tag from the file-move commit 1d38fe6ee6a8 to
  deb2d2ecd43d ("PCI/GPU: implement VGA arbitration on Linux").

 drivers/pci/vgaarb.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/drivers/pci/vgaarb.c b/drivers/pci/vgaarb.c
index 3de05aee7859..350ab9624eb9 100644
--- a/drivers/pci/vgaarb.c
+++ b/drivers/pci/vgaarb.c
@@ -1272,7 +1272,7 @@ static ssize_t vga_arb_write(struct file *file, const char __user *buf,
 			goto done;
 		}
 
-		if (vga_tryget(pdev, io_state)) {
+		if (vga_tryget(pdev, io_state) == 0) {
 			/* Update the client's locks lists... */
 			for (i = 0; i < MAX_USER_CARDS; i++) {
 				if (priv->cards[i].pdev == pdev) {
-- 
2.49.0