Under radical environmental change, populations may need to adapt quickly to avoid substantial declines in abundance and threats to their persistence. The outcome of this race between evolution and demography depends on the genetic architecture of adaptation, which determines how fast evolution can proceed. In particular, adaptation may require coordinated evolution at multiple loci (e.g., cooperating ion transporters for ion uptake), with single-locus changes being deleterious. Such selection on coadapted genes leads to a fitness landscape with a valley, which can in turn favor the evolution of structural variants that link beneficial alleles at different loci. Here, we investigate how epistasis and recombination jointly affect population dynamics under such a fitness valley. We assume that adaptation occurs from standing genetic variation and model the eco-evolutionary dynamics deterministically. We show that recombination has strong impacts on population decline and recovery in this context. Higher recombination rates cause evolutionary trajectories to be pulled toward unfit states, leading to prolonged evolutionary plateaus, during which the population can decline precipitously. In highly detrimental cases where coadapted mutations are located on different chromosomes, chromosomal fusions that are preexisting at low frequency can lead to faster population recovery by allowing the genetic system to escape the attraction to unfit intermediate states. Our results provide insights into eco-evolutionary dynamics in systems where chromosome number varies drastically among sibling species, such as the copepod Eurytemora affinis species complex, and offer new perspectives on the impacts of genome architecture on population dynamics in stressful environments.