The essential role of mammary gland stroma in the regulation of mammary epithelial development, function, and cancer has long been recognized. Only recently, though, the functions of individual stromal cell populations have begun to become more clarified. Mammary fibroblasts have emerged as master regulators and modulators of epithelial cell behavior through paracrine signaling, extracellular matrix production and remodeling, and through regulation of other stromal cell types. In this review article, we summarize the crucial studies that helped to untangle the roles of fibroblasts in mammary gland development. Furthermore, we discuss the origin, heterogeneity, and plasticity of mammary fibroblasts during mammary development and cancer progression.