Consequential works in cognitive neuroscience have led to the formulation of an interactive dual-stream model of language processing: the dorsal stream may process the phonological aspects of language, whereas the ventral stream may process the semantic aspects of language. While it is well-accepted that the dorsal route is subserved by the arcuate fasciculus, the structural connectivity of the semantic ventral stream is a matter of dispute. Here we designed a longitudinal study to gain new insights into this central but controversial question. Thirty-one patients harboring a left diffuse low-grade glioma—a rare neurological condition that infiltrates preferentially white matter associative pathways—were assessed with a prototypical task of language (i.e. verbal fluency) before and after surgery. All were operated under local anesthesia with a cortical and subcortical brain mapping—enabling to identify and preserve eloquent structures for language. We performed voxel-based lesion-symptom (VLSM) analyses on pre- and postoperative behavioral data. Preoperatively, we found a significant relationship between semantic fluency scores and the white matter fibers shaping the ventro-lateral connectivity (P < 0.05 corrected). The statistical map was found to substantially overlap with the spatial position of the inferior fronto-occipital fasciculus (IFOF) (37.7%). Furthermore, a negative correlation was observed between semantic fluency scores and the infiltration volumes in this fasciculus (r = -0.4, P = 0.029). Postoperatively, VLSM analyses were inconclusive. Taken as a whole and when combined with the literature data, our findings strengthen the view that the IFOF plays an essential role in semantic processing and may subserve the direct ventral pathway of language.