Excessive adiposity has long been associated with increased incidence of breast cancer in post-menopausal women, and with increased mortality from breast cancer, regardless of the menopausal status. Although adipose tissue-derived estrogen contributes to obesity-associated risk for estrogen receptor (ER)-positive breast cancer, the estrogen-independent impact of adipose tissue on tumor invasion and progression needs to be elucidated. Here, we show that adipose stromal cells (ASCs) significantly stimulate migration and invasion of ER-negative breast cancer cells in vitro and tumor invasion in a co-transplant xenograft mouse model. Our study also identifies cofilin-1, a known regulator of actin dynamics, as a determinant of the tumor-promoting activity of ASCs. The cofilin-1-dependent pathway affects the production of interleukin 6 (IL-6) in ASCs. Depletion of IL-6 from the ASC-conditioned medium abrogated the stimulatory effect of ASCs on the migration and invasion of breast tumor cells. Thus, our study uncovers a link between a cytoskeleton-based pathway in ASCs and the stromal impact on breast cancer cells.