Can I leave gaps between different groups of modules to make the panel easier to read?

Yes. There is no requirement to populate every position consecutively.

Leaving a deliberate gap can be a perfectly sensible way to separate a menagerie of network, fibre or AV connections when the numbered ports alone are not enough to make the layout obvious.