Patching nVidia GPU driver for hot-unplug on Linux
Tags:Recently, I’ve using an extremely cursed setup where my XPS 13 9360 laptop is connected to a Sonnet EchoExpress 2 box rewired for Thunderbolt 3 that has an nVidia Quadro 600 GPU, and Linux is set up for render offload to the eGPU and then frame transfer back to iGPU to be displayed on the laptop’s integrated display, which (to my sheer surprise) not only works quire reliably, but even gives me higher FPS in Team Fortress 2 than the iGPU.
There’s only really one downside: if the eGPU falls off the bus, either because someone™ pulled out the cable, or because the stars didn’t align quite right this morning and it decided to enumerate seemingly at random (sometimes this is preceeded by whining from PCIe AER, sometimes not, I think it’s some sort of hardware issue like a badly inserted PCIe card, but I’m not entirely sure), the nVidia driver… hangs. Hangs quite deliberately, as the sources to the kernel driver show. This leaves the Xorg instance bound to the eGPU hung forever (which confuses bumblebee, but is otherwise not especially bad), and also prevents any new ones from using the eGPU (which is bad).
Anyway, I was kind of annoyed of rebooting every time it happens, so I decided to reboot a few more dozen times instead while patching the driver. This has indeed worked, and left me with something similar to a functional hot-unplug, mildly crippled by the fact that nvidia-modeset is a completely opaque blob that keeps some internal state and tries to act on it, getting stuck when it tries to do something to the now-missing eGPU.
Turns out, there are only a few issues preventing functional hot-unplug.
-
In
nvidia_remove, the driver actually checks if anyone’s still trying to use it, and if yes, it tries to just hang the removal process. This doesn’t actually work, or rather, it mostly works by accident. It starts an infinite loop callingos_schedule()while having taken theNV_LINUX_DEVICESlock. While in the default configuration this indeed hangs any reentrant requests into the driver by virtue ofNV_CHECK_PCI_CONFIG_SPACEtaking the same lock (inverify_pci_bars, passing theNVreg_CheckPCIConfigSpace=0module option eliminates that accidental safety mechanism, and allows reentrant requests to proceed. They do not crash due to memory being deallocated innvidia_remove(so you don’t get an unhandled kernel page fault), but they still crash due to being unable to access the GPU. -
The NVKMS component (in the
nvidia-modesetmodule) tries to maintain some state, and change it when e.g. the Xorg instance quits and closes the/dev/nvidia-modesetfile. Unfortunately, it does not expect the GPU to go away, and first spews a few messages todmesgsimilar tonvidia-modeset: ERROR: GPU:0: Failed to query display engine channel state: 0x0000857d:0:0:0x0000000f, after which it appears to hang somewhere inside the blob, which has been conveniently stripped of all symbols. This needs to be prevented, but… -
The NVKMS component effectively only exposes a single opaque ioctl, and all the communication, including communication of the GPU bus ID, happens out of band with regards to the open source parts of the
nvidia-modesetmodule. Fortunately, NVKMS calls back into NVRM, and this allows us to associate each/dev/nvidia-modesetfd with the GPU bus ID. -
When unloading NVKMS, it also tries to act on its internal state and change the GPU state, which leads to the same hang.
All in all, this allows a patch to be written that detects when a GPU goes away, ignores all further NVKMS requests related to that specific GPU (and returns -ENOENT in response to ioctls, which Xorg appropriately interprets as a fault condition), correctly releases the resources by requesting NVRM, and improperly unloads NVKMS so it doesn’t try to reset the GPU state. (All actual resources should be released by this point, and NVKMS doesn’t have any resource allocation callbacks other than those we already intercept, so in theory this doesn’t have any bad consequences. But I’m not working for nVidia, so this might be completely wrong.)
After the GPU is plugged back in, NVKMS will try to act on its internal state again; in this case, it doesn’t hang, but it doesn’t initialize the GPU correctly either, so the nvidia-modeset kernel module has to be (manually) reloaded. It’s not easy to do this automatically because in a hypothetical system with more than one nVidia GPU the module would still be in use when one of them dies, and so just hard reloading NVKMS would have unfortunate consequences. (Though, I don’t really know whether NVKMS would try to access the dead GPU in response to the request acting on the other GPU anyway. I decided to do it conservatively.) Once it’s reloaded you’re back in the game though!
Here’s the patch, written against the nvidia-legacy-390xx-390.87 Debian source package:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
diff -ur original/common/inc/nv-linux.h patchedl/common/inc/nv-linux.h
--- original/common/inc/nv-linux.h 2018-09-23 12:20:02.000000000 +0000
+++ patched/common/inc/nv-linux.h 2018-10-28 07:19:21.526566940 +0000
@@ -1465,6 +1465,7 @@
typedef struct nv_linux_state_s {
nv_state_t nv_state;
atomic_t usage_count;
+ atomic_t dead;
struct pci_dev *dev;
diff -ur original/common/inc/nv-modeset-interface.h patched/common/inc/nv-modeset-interface.h
--- original/common/inc/nv-modeset-interface.h 2018-08-22 00:55:23.000000000 +0000
+++ patched/common/inc/nv-modeset-interface.h 2018-10-28 07:22:00.768238371 +0000
@@ -25,6 +25,8 @@
#include "nv-gpu-info.h"
+#include <asm/atomic.h>
+
/*
* nvidia_modeset_rm_ops_t::op gets assigned a function pointer from
* core RM, which uses the calling convention of arguments on the
@@ -115,6 +117,8 @@
int (*set_callbacks)(const nvidia_modeset_callbacks_t *cb);
+ atomic_t * (*gpu_dead)(NvU32 gpu_id);
+
} nvidia_modeset_rm_ops_t;
NV_STATUS nvidia_get_rm_ops(nvidia_modeset_rm_ops_t *rm_ops);
diff -ur original/common/inc/nv-proto.h patched/common/inc/nv-proto.h
--- original/common/inc/nv-proto.h 2018-08-22 00:55:23.000000000 +0000
+++ patched/common/inc/nv-proto.h 2018-10-28 07:20:49.939494812 +0000
@@ -81,6 +81,7 @@
NvBool nvidia_get_gpuid_list (NvU32 *gpu_ids, NvU32 *gpu_count);
int nvidia_dev_get (NvU32, nvidia_stack_t *);
void nvidia_dev_put (NvU32, nvidia_stack_t *);
+atomic_t * nvidia_dev_dead (NvU32);
int nvidia_dev_get_uuid (const NvU8 *, nvidia_stack_t *);
void nvidia_dev_put_uuid (const NvU8 *, nvidia_stack_t *);
int nvidia_dev_get_pci_info (const NvU8 *, struct pci_dev **, NvU64 *, NvU64 *);
diff -ur original/nvidia/nv.c patched/nvidia/nv.c
--- original/nvidia/nv.c 2018-09-23 12:20:02.000000000 +0000
+++ patched/nvidia/nv.c 2018-10-28 07:48:05.895025112 +0000
@@ -1944,6 +1944,12 @@
unsigned int i;
NvBool bRemove = NV_FALSE;
+ if (NV_ATOMIC_READ(nvl->dead))
+ {
+ nv_printf(NV_DBG_ERRORS, "NVRM: nvidia_close called on dead device by pid %d!\n",
+ current->pid);
+ }
+
NV_CHECK_PCI_CONFIG_SPACE(sp, nv, TRUE, TRUE, NV_MAY_SLEEP());
/* for control device, just jump to its open routine */
@@ -2106,6 +2112,12 @@
size_t arg_size;
int arg_cmd;
+ if (NV_ATOMIC_READ(nvl->dead))
+ {
+ nv_printf(NV_DBG_ERRORS, "NVRM: nvidia_ioctl called on dead device by pid %d!\n",
+ current->pid);
+ }
+
nv_printf(NV_DBG_INFO, "NVRM: ioctl(0x%x, 0x%x, 0x%x)\n",
_IOC_NR(cmd), (unsigned int) i_arg, _IOC_SIZE(cmd));
@@ -3217,6 +3229,7 @@
NV_INIT_MUTEX(&nvl->ldata_lock);
NV_ATOMIC_SET(nvl->usage_count, 0);
+ NV_ATOMIC_SET(nvl->dead, 0);
if (!rm_init_event_locks(sp, nv))
return NV_FALSE;
@@ -4018,14 +4031,38 @@
nv_printf(NV_DBG_ERRORS,
"NVRM: Attempting to remove minor device %u with non-zero usage count!\n",
nvl->minor_num);
+ nv_printf(NV_DBG_ERRORS,
+ "NVRM: YOLO, waiting for usage count to drop to zero\n");
WARN_ON(1);
- /* We can't continue without corrupting state, so just hang to give the
- * user some chance to do something about this before reboot */
- while (1)
+ NV_ATOMIC_SET(nvl->dead, 1);
+
+ /* Insanity check: wait until all clients die, then hope for the best. */
+ while (1) {
+ UNLOCK_NV_LINUX_DEVICES();
os_schedule();
- }
+ LOCK_NV_LINUX_DEVICES();
+
+ nvl = pci_get_drvdata(dev);
+ if (!nvl || (nvl->dev != dev))
+ {
+ goto done;
+ }
+
+ if (NV_ATOMIC_READ(nvl->usage_count) == 0)
+ {
+ break;
+ }
+ }
+ nv_printf(NV_DBG_ERRORS,
+ "NVRM: Usage count is now zero, proceeding to remove the GPU\n");
+ nv_printf(NV_DBG_ERRORS,
+ "NVRM: This is not actually supposed to work lol. Hope it does tho 👍\n");
+ nv_printf(NV_DBG_ERRORS,
+ "NVRM: You probably want to reload nvidia-modeset now if you want any "
+ "of this to ever start up again, but like, man, that's your choice entirely\n");
+ }
nv = NV_STATE_PTR(nvl);
if (nvl == nv_linux_devices)
nv_linux_devices = nvl->next;
@@ -4712,6 +4749,22 @@
up(&nvl->ldata_lock);
}
+atomic_t *nvidia_dev_dead(NvU32 gpu_id)
+{
+ nv_linux_state_t *nvl;
+ atomic_t *ret;
+
+ /* Takes nvl->ldata_lock */
+ nvl = find_gpu_id(gpu_id);
+ if (!nvl)
+ return NV_FALSE;
+
+ ret = &nvl->dead;
+ up(&nvl->ldata_lock);
+
+ return ret;
+}
+
/*
* Like nvidia_dev_get but uses UUID instead of gpu_id. Note that this may
* trigger initialization and teardown of unrelated devices to look up their
diff -ur original/nvidia/nv-modeset-interface.c patched/nvidia/nv-modeset-interface.c
--- original/nvidia/nv-modeset-interface.c 2018-08-22 00:55:22.000000000 +0000
+++ patched/nvidia/nv-modeset-interface.c 2018-10-28 07:20:25.959243110 +0000
@@ -114,6 +114,7 @@
.close_gpu = nvidia_dev_put,
.op = rm_kernel_rmapi_op, /* provided by nv-kernel.o */
.set_callbacks = nvidia_modeset_set_callbacks,
+ .gpu_dead = nvidia_dev_dead,
};
if (strcmp(rm_ops->version_string, NV_VERSION_STRING) != 0)
diff -ur original/nvidia/nv-reg.h patched/nvidia/nv-reg.h
diff -ur original/nvidia-modeset/nvidia-modeset-linux.c patched/nvidia-modeset/nvidia-modeset-linux.c
--- original/nvidia-modeset/nvidia-modeset-linux.c 2018-09-23 12:20:02.000000000 +0000
+++ patched/nvidia-modeset/nvidia-modeset-linux.c 2018-10-28 07:47:14.738703417 +0000
@@ -75,6 +75,9 @@
static struct semaphore nvkms_lock;
+static NvU32 clopen_gpu_id;
+static NvBool leak_on_unload;
+
/*************************************************************************
* NVKMS executes queued work items on a single kthread.
*************************************************************************/
@@ -89,6 +92,9 @@
struct nvkms_per_open {
void *data;
+ NvU32 gpu_id;
+ atomic_t *gpu_dead;
+
enum NvKmsClientType type;
union {
@@ -711,6 +717,9 @@
nvidia_modeset_stack_ptr stack = NULL;
NvBool ret;
+ printk(KERN_INFO NVKMS_LOG_PREFIX "nvkms_open_gpu called with %08x, pid %d\n",
+ gpuId, current->pid);
+
if (__rm_ops.alloc_stack(&stack) != 0) {
return NV_FALSE;
}
@@ -719,6 +728,10 @@
__rm_ops.free_stack(stack);
+ if (ret) {
+ clopen_gpu_id = gpuId;
+ }
+
return ret;
}
@@ -726,12 +739,17 @@
{
nvidia_modeset_stack_ptr stack = NULL;
+ printk(KERN_INFO NVKMS_LOG_PREFIX "nvkms_close_gpu called with %08x, pid %d\n",
+ gpuId, current->pid);
+
if (__rm_ops.alloc_stack(&stack) != 0) {
return;
}
__rm_ops.close_gpu(gpuId, stack);
+ clopen_gpu_id = gpuId;
+
__rm_ops.free_stack(stack);
}
@@ -771,8 +789,14 @@
popen->type = type;
+ printk(KERN_INFO NVKMS_LOG_PREFIX "entering nvkms_open_common, pid %d\n",
+ current->pid);
+
*status = down_interruptible(&nvkms_lock);
+ printk(KERN_INFO NVKMS_LOG_PREFIX "taken lock in nvkms_open_common, pid %d\n",
+ current->pid);
+
if (*status != 0) {
goto failed;
}
@@ -781,6 +805,9 @@
up(&nvkms_lock);
+ printk(KERN_INFO NVKMS_LOG_PREFIX "given up lock in nvkms_open_common, pid %d\n",
+ current->pid);
+
if (popen->data == NULL) {
*status = -EPERM;
goto failed;
@@ -799,10 +826,16 @@
*status = 0;
+ printk(KERN_INFO NVKMS_LOG_PREFIX "exiting in nvkms_open_common, pid %d\n",
+ current->pid);
+
return popen;
failed:
+ printk(KERN_INFO NVKMS_LOG_PREFIX "error in nvkms_open_common, pid %d\n",
+ current->pid);
+
nvkms_free(popen, sizeof(*popen));
return NULL;
@@ -816,14 +849,36 @@
* mutex.
*/
+ printk(KERN_INFO NVKMS_LOG_PREFIX "entering nvkms_close_common, pid %d\n",
+ current->pid);
+
down(&nvkms_lock);
- nvKmsClose(popen->data);
+ printk(KERN_INFO NVKMS_LOG_PREFIX "taken lock in nvkms_close_common, pid %d\n",
+ current->pid);
+
+ if (popen->gpu_id != 0 && atomic_read(popen->gpu_dead) != 0) {
+ printk(KERN_ERR NVKMS_LOG_PREFIX "awwww u need cleanup :3 "
+ "in nvkms_close_common, pid %d\n",
+ current->pid);
+
+ nvkms_close_gpu(popen->gpu_id);
+
+ popen->gpu_id = 0;
+ popen->gpu_dead = NULL;
+
+ leak_on_unload = NV_TRUE;
+ } else {
+ nvKmsClose(popen->data);
+ }
popen->data = NULL;
up(&nvkms_lock);
+ printk(KERN_INFO NVKMS_LOG_PREFIX "given up lock in nvkms_close_common, pid %d\n",
+ current->pid);
+
if (popen->type == NVKMS_CLIENT_KERNEL_SPACE) {
/*
* Flush any outstanding nvkms_kapi_event_kthread_q_callback() work
@@ -844,6 +899,9 @@
}
nvkms_free(popen, sizeof(*popen));
+
+ printk(KERN_INFO NVKMS_LOG_PREFIX "exiting nvkms_close_common, pid %d\n",
+ current->pid);
}
int NVKMS_API_CALL nvkms_ioctl_common
@@ -855,20 +913,58 @@
int status;
NvBool ret;
+ printk(KERN_INFO NVKMS_LOG_PREFIX "entering nvkms_ioctl_common, pid %d\n",
+ current->pid);
+
status = down_interruptible(&nvkms_lock);
if (status != 0) {
return status;
}
+ printk(KERN_INFO NVKMS_LOG_PREFIX "taken lock in nvkms_ioctl_common, pid %d\n",
+ current->pid);
+
+ if (popen->gpu_id != 0 && atomic_read(popen->gpu_dead) != 0) {
+ goto dead;
+ }
+
+ clopen_gpu_id = 0;
+
if (popen->data != NULL) {
ret = nvKmsIoctl(popen->data, cmd, address, size);
} else {
ret = NV_FALSE;
}
+ if (clopen_gpu_id != 0) {
+ if (!popen->gpu_id) {
+ printk(KERN_INFO NVKMS_LOG_PREFIX "detected gpu %08x open in nvkms_ioctl_common, "
+ "pid %d\n", clopen_gpu_id, current->pid);
+ popen->gpu_id = clopen_gpu_id;
+ popen->gpu_dead = __rm_ops.gpu_dead(clopen_gpu_id);
+ } else {
+ printk(KERN_INFO NVKMS_LOG_PREFIX "detected gpu %08x close in nvkms_ioctl_common, "
+ "pid %d\n", clopen_gpu_id, current->pid);
+ popen->gpu_id = 0;
+ popen->gpu_dead = NULL;
+ }
+ }
+
up(&nvkms_lock);
+ printk(KERN_INFO NVKMS_LOG_PREFIX "given up lock in nvkms_ioctl_common, pid %d\n",
+ current->pid);
+
return ret ? 0 : -EPERM;
+
+dead:
+ up(&nvkms_lock);
+
+ printk(KERN_ERR NVKMS_LOG_PREFIX "*notices ur gpu is dead* owo whats this "
+ "in nvkms_ioctl_common, pid %d\n",
+ current->pid);
+
+ return -ENOENT;
}
/*************************************************************************
@@ -1239,9 +1335,14 @@
nvkms_proc_exit();
- down(&nvkms_lock);
- nvKmsModuleUnload();
- up(&nvkms_lock);
+ if(leak_on_unload) {
+ printk(KERN_ERR NVKMS_LOG_PREFIX "im just gonna leak all the kms junk ok? "
+ "haha nvm wasnt a question. in nvkms_exit\n");
+ } else {
+ down(&nvkms_lock);
+ nvKmsModuleUnload();
+ up(&nvkms_lock);
+ }
/*
* At this point, any pending tasks should be marked canceled, but
Here’s some handy scripts I was using while debugging it:
1
2
3
4
5
#!/bin/sh -ex
modprobe acpi_ipmi
insmod nvidia.ko NVreg_ResmanDebugLevel=-1 NVreg_CheckPCIConfigSpace=0
insmod nvidia-modeset.ko
dmesg -w
1
2
3
#!/bin/sh
rmmod nvidia-modeset
rmmod nvidia
1
2
#!/bin/sh
exec Xorg :8 -config /etc/bumblebee/xorg.conf.nvidia -configdir /etc/bumblebee/xorg.conf.d -sharevts -nolisten tcp -noreset -verbose 3 -isolateDevice PCI:06:00:0 -modulepath /usr/lib/nvidia/nvidia,/usr/lib/xorg/modules
And finally, here are the relevant kernel and Xorg log messages, showing what happens when a GPU is unplugged:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
[ 219.524218] NVRM: loading NVIDIA UNIX x86_64 Kernel Module 390.87 Tue Aug 21 12:33:05 PDT 2018 (using threaded interrupts)
[ 219.527409] nvidia-modeset: Loading NVIDIA Kernel Mode Setting Driver for UNIX platforms 390.87 Tue Aug 21 16:16:14 PDT 2018
[ 224.780721] nvidia-modeset: nvkms_open_gpu called with 00000600, pid 4560
[ 224.807370] nvidia-modeset: detected gpu 00000600 open in nvkms_ioctl_common, pid 4560
[ 239.061383] NVRM: GPU at PCI:0000:06:00: GPU-9fe1319c-8dd3-44e4-2b74-de93f8b02c6a
[ 239.061387] NVRM: Xid (PCI:0000:06:00): 79, GPU has fallen off the bus.
[ 239.061389] NVRM: GPU at 0000:06:00.0 has fallen off the bus.
[ 239.061398] NVRM: A GPU crash dump has been created. If possible, please run
NVRM: nvidia-bug-report.sh as root to collect this data before
NVRM: the NVIDIA kernel module is unloaded.
[ 240.209498] NVRM: Attempting to remove minor device 0 with non-zero usage count!
[ 240.209501] NVRM: YOLO, waiting for usage count to drop to zero
[ 241.433499] nvidia-modeset: *notices ur gpu is dead* owo whats this in nvkms_ioctl_common, pid 4560
[ 241.433851] nvidia-modeset: awwww u need cleanup :3 in nvkms_close_common, pid 4560
[ 241.433853] nvidia-modeset: nvkms_close_gpu called with 00000600, pid 4560
[ 250.440498] NVRM: Usage count is now zero, proceeding to remove the GPU
[ 250.440513] NVRM: This is not actually supposed to work lol. Hope it does tho 👍
[ 250.440520] NVRM: You probably want to reload nvidia-modeset now if you want any of this to ever start up again, but like, man, that's your choice entirely
[ 250.440870] pci 0000:06:00.1: Dropping the link to 0000:06:00.0
[ 250.440950] pci_bus 0000:06: busn_res: [bus 06] is released
[ 250.440982] pci_bus 0000:07: busn_res: [bus 07-38] is released
[ 250.441012] pci_bus 0000:05: busn_res: [bus 05-38] is released
[ 251.000794] pci_bus 0000:02: Allocating resources
[ 251.001324] pci_bus 0000:02: Allocating resources
[ 253.765953] pcieport 0000:00:1c.0: AER: Corrected error received: 0000:00:1c.0
[ 253.765969] pcieport 0000:00:1c.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, (Receiver ID)
[ 253.765976] pcieport 0000:00:1c.0: device [8086:9d10] error status/mask=00002001/00002000
[ 253.765982] pcieport 0000:00:1c.0: [ 0] Receiver Error (First)
[ 253.841064] pcieport 0000:02:02.0: Refused to change power state, currently in D3
[ 253.843882] pcieport 0000:02:00.0: Refused to change power state, currently in D3
[ 253.846177] pci_bus 0000:03: busn_res: [bus 03] is released
[ 253.846248] pci_bus 0000:04: busn_res: [bus 04-38] is released
[ 253.846300] pci_bus 0000:39: busn_res: [bus 39] is released
[ 253.846348] pci_bus 0000:02: busn_res: [bus 02-39] is released
[ 353.369487] nvidia-modeset: im just gonna leak all the kms junk ok? haha nvm wasnt a question. in nvkms_exit
[ 357.600350] nvidia-modeset: Loading NVIDIA Kernel Mode Setting Driver for UNIX platforms 390.87 Tue Aug 21 16:16:14 PDT 2018
1
[ 244.798] (EE) NVIDIA(GPU-0): WAIT (2, 8, 0x8000, 0x000011f4, 0x00001210)