[PATCH] drm/vkms: sanitize display mode to prevent hrtimer livelock

[w15303746062@163.com]

From: Mingyu Wang <25181214217@stu.xidian.edu.cn>

SyzKaller reported an RCU stall caused by a livelock in
vkms_crtc_handle_vblank_timeout.

The root cause is that vkms derives its vblank timer period directly
from the display mode, but does not validate the mode parameters in
its atomic_check. A fuzzer or malicious user can pass a mode with
either an explicit vrefresh > 1000 Hz, or a combination of huge
crtc_clock and tiny htotal/vtotal that causes the computed vrefresh
to overflow or result in a frame duration close to zero.

When the period approaches zero, the hrtimer fires faster than it can
be handled, leading to an interrupt storm that deadlocks the CPU.

Fix this by rejecting modes in atomic_check where:
 - the explicit vrefresh exceeds 1000 Hz, or
 - the implied frame duration (derived from crtc_clock/htotal/vtotal)
   is less than 1 ms (equivalent to >1000 Hz).

This also protects against integer overflow in drm_mode_vrefresh().

SyzKaller logs snippet of the failure:
[  392.807933][    C2] vkms_vblank_simulate: vblank timer overrun
...
[  592.384301][    C3] rcu: INFO: rcu_preempt detected stalls on CPUs/tasks:
[  592.438915][    C0] RIP: 0010:native_queued_spin_lock_slowpath+0x23e/0x9c0
[  592.440560][    C0] Call Trace:
[  592.440570][    C0]  <IRQ>
[  592.448399][    C0]  drm_handle_vblank+0x132/0xc70
[  592.449106][    C0]  __hrtimer_run_queues+0x1f5/0xb30

Fixes: 02e2681ffe1a ("drm/vkms: Convert to DRM's vblank timer")
Cc: stable@vger.kernel.org
Signed-off-by: Mingyu Wang <25181214217@stu.xidian.edu.cn>
---
 drivers/gpu/drm/vkms/vkms_crtc.c | 22 ++++++++++++++++++++++
 1 file changed, 22 insertions(+)

diff --git a/drivers/gpu/drm/vkms/vkms_crtc.c b/drivers/gpu/drm/vkms/vkms_crtc.c
index 35ddc553a5e6..20b97dd0cc5f 100644
--- a/drivers/gpu/drm/vkms/vkms_crtc.c
+++ b/drivers/gpu/drm/vkms/vkms_crtc.c
@@ -116,9 +116,31 @@ static int vkms_crtc_atomic_check(struct drm_crtc *crtc,
 	struct drm_crtc_state *crtc_state = drm_atomic_get_new_crtc_state(state,
 									  crtc);
 	struct vkms_crtc_state *vkms_state = to_vkms_crtc_state(crtc_state);
+	struct drm_display_mode *mode = &crtc_state->adjusted_mode;
 	struct drm_plane *plane;
 	struct drm_plane_state *plane_state;
 	int i = 0, ret;
+	int vrefresh;
+	u64 frame_ns;
+
+	/*
+	 * Reject modes that would cause an hrtimer storm.
+	 * A virtual display cannot meaningfully refresh at >1000 Hz,
+	 * and an extremely high vrefresh (or a crafted combination of
+	 * crtc_clock/htotal/vtotal that overflows the vrefresh computation)
+	 * would produce a frame duration close to zero, locking the CPU
+	 * in an interrupt loop.
+	 */
+	if (crtc_state->enable) {
+		vrefresh = drm_mode_vrefresh(mode);
+		if (vrefresh > 1000)
+			return -EINVAL;
+		if (!mode->crtc_clock || !mode->htotal || !mode->vtotal)
+			return -EINVAL;
+		frame_ns = div_u64((u64)mode->htotal * mode->vtotal * 1000000ULL, mode->crtc_clock);
+		if (frame_ns < 1000000) /* <1 ms => refresh >1000 Hz */
+			return -EINVAL;
+	}
 
 	if (vkms_state->active_planes)
 		return 0;
-- 
2.34.1