The proliferation of Sargassum in the Tropical Atlantic has occurred almost every year since 2011, but a strong variability of biomass is observed among years. Essential knowledge about the drivers of Sargassum growth and decay is still lacking to explain this interannual variability. Benefiting from accurate basin scale Sargassum detections provided by remote sensing, and from physical and biogeochemical ocean simulations, we developed a Lagrangian drift-growth model to simulate Sargassum distribution over the period 2016–2020. The resulting trajectories and biomass time series of Sargassum aggregates were analyzed to highlight the main limiting factors of growth and decay. The nitrogen and phosphorous concentrations are found to be weakly restrictive compared to physical limiting factors, especially the temperature. In particular, the warm waters found off northern Brazil appear to be instrumental in triggering the end of seasonal growth of Sargassum. The timing of the seasonal warming of this region strongly impacts the quantities of Sargassum simulated each year. This suggests that this region should be monitored to anticipate the development of Sargassum and resulting strandings.