In this study, membrane fouling caused by paperboard mill treated effluent (PMTE) was investigated based on a dead-end ultrafiltration (UF) pilot-scale study. The membranes employed were commercial hydrophobic UF membranes made of polyethersulfone (PES) with a molecular weight cut-off of 10 kDa, 50 kDa, and 100 kDa. Membrane fouling mechanism during dead-end filtration, chemical analysis, field emission scanning electron microscopy (FESEM), energy-dispersive spectrophotometry (EDS), attenuated total reflection-Fourier transform infrared (ATR-FTIR) spectroscopy and 3D fluorescence excitation–emission matrix (3DEEM) analysis were applied to understand which fraction of the dissolved and colloidal substances (DCS) caused the membrane fouling. The results indicated that the phenomenon controlling fouling mechanism tended to be cake layer formation (R2 ≥ 0.98) for all membranes tested. The 3DEEM results indicate that the majority of the organic foulants with fluorescence characteristics on the membrane were colloidal proteins (protein-like substances I+II) and macromolecular proteins (soluble microbial products, SMP-like substances). In addition, polysaccharide (cellulosic species), fatty and resin acid substances were identified on the fouled membrane by the ATR–FTIR analysis and play an important role in membrane fouling. In addition, the FESEM and EDS analyses indicate that the presence of inorganic foulants on the membrane surfaces, such as metal ions and especially Ca2+, can accelerate membrane fouling, whereas Mg and Si are linked to reversible fouling.