Turns out the simplest way to zoom in on a video beyond "fit to screen" is to use the Mac OS builtin Accessibility Zoom feature. I could not find any application that could do it (Finder Preview, Quicktime, VLC, MPV).
System Preferences > Accessibility > Zoom to see your keyboard/mouse shortcuts. For example, on my system I can use
ctrl + 2 finger trackpad scroll to zoom in/out. Or
opt + cmd + =/- with the keyboard. After you zoom in, you can use the mouse to pan around.