The object of this note is the moduli spaces of cubic fourfolds (resp., Gushel-Mukai fourfolds) which contain some special rational surfaces. Under some hypotheses on the families of such surfaces, we develop a general method to show the unirationality of the moduli spaces of the $n$-pointed such fourfolds. We apply this to some codimension 1 loci of cubic fourfolds (resp., Gushel-Mukai fourfolds) appeared in the literature recently.