How should I allow for non-rack equipment that will sit on a shelf?

Start at the shelf platform, not at the top of the shelf's nominal rack position.

Because the platform sits towards the bottom of the U occupied by the shelf, roughly the first U of equipment height can occupy that same vertical region. If the device is taller, you then need enough clear rack space above it for the remainder, together with any ventilation or access clearance it needs.

Also check depth, rear connectors and cable bends. A modem that’s 40mm high may be delightfully well behaved; a 130mm-high NAS sitting on the same shelf has rather more ambitious plans for the U positions above it.

Round upwards where necessary and allow useful clearance rather than designing a three-dimensional game of Tetris around the device.