The underlying neurobiology of major depression (MD) is likely to represent an interaction between genetic susceptibility and environmental factors such as stress. We investigated, in a multimodal high-resolution magnetic resonance imaging (MRI) genetic study, whether reduced hippocampal volumes and other brain alterations are associated with the tri-allelic polymorphism of the serotonin transporter and childhood stress in patients with MD and healthy subjects. Patients with MD and healthy participants were investigated using high-resolution MRI and genotyping for serotonin transporter polymorphism in the promoter region of the serotonin transporter gene (SLC6A4, 5-HTTLPR). Region of interest analysis of the hippocampus, whole-brain voxel-based morphometry (VBM), and assessment of childhood stress were carried out. Patients carrying the risk S-allele developed smaller hippocampal volumes when they had a history of emotional neglect compared with patients who only had one risk factor (environmental or genetic). In patients, childhood stress also predicted further hippocampal white matter alterations independently from the genotype. Moreover, the left prefrontal cortex was smaller in patients, whereby childhood stress resulted in larger prefrontal volumes in those subjects carrying the non-risk L-allele, suggesting preventive effects. The findings indicate that subjects with both environmental and genetic risk factors are susceptible to stress-related hippocampal changes. Structural brain changes due to stress represent part of the mechanism by which the illness risk and outcome might be genetically mediated.