MMM well working backwards I see that you need to call the Mac's sizeWindow which is called from ioSetDisplayMode in sqMacWindow.c which is called by primitiveSetDisplayMode in interp.c which is primitive 92
In Interpreter primitiveSetDisplayMode talkes about looking at DisplayScreen setDisplayDepth:extent:fullscreen:
which is where the trail grows cold.
Not so cold: After examining the code is was fairly clear what to do if you wanted to exercise the primitive. Add the following method to DisplayScreen class:
depth: depthInteger width: widthInteger height: heightInteger flag: aBoolean <primitive: 92> self primitiveFail
And run the doIt:
DisplayScreen depth: 8 width: 100 height: 100 flag: false
This will reduce the window to 100 by 100. Presumably, there was a reason the code executing this was deleted from the general release