The sex determing region Y-box 4 (SOX4) gene is a critical developmental transcriptional factor that is overexpressed in prostate cancer (PCa). While we and others have investigated the role of SOX4 overexpression in PCa, the molecular mechanism underlying its aberrant expression remains unclear. Immunohistochemistry were utilized to detect SOX4 expression and the correlation between estrogen receptor β (ERβ), androgen receptor (AR) and SOX4 in a cohort of 94 clinical specimens. Real-time quantitative PCR and Western blotting were used to study the transcript and protein expression levels. Immunofluorescence staining and co-immunoprecipitation were performed to assess the interaction and subcellular location of ERβ and AR. Chromatin immunoprecipitation (ChIP) assays and Luciferase reporter assays were performed to explore the binding and transcriptional activities of ERβ and AR to the SOX4 promoter. Cellular function was evaluated by MTS, invasion and wound healing assays. SOX4 expression is up-regulated in Castration-Resistant Prostate Cancer (CRPC) tumors compared to hormone-dependent PCa (HDPC) cases. Increased expression was also observed in PCa cells after long-term androgen-deprivation treatment (ADT). In vitro data indicated that SOX4 is an AR transcriptional target and down-regulated by dihydrotestosterone (DHT) via AR. 17β-estradiol (E2) up-regulates SOX4 expression in the absence of androgen through the formation of a protein complex between ERβ and AR. Knockdown of AR or ERβ blocks the E2-induced SOX4 expression. ChIP assays confirmed that both ERβ and AR bind to the SOX4 promoter in response to E2. Functionally, silencing SOX4 significantly attenuates the proliferative effect, as well as the capacity of migration and invasion of E2 on PCa cells. Clinically, overexpression of SOX4 is significantly associated with ERβ expression in PCa. In addition, this association is still retained in CRPC patients with poor prognosis. These findings suggest that SOX4 is a novel DHT-repressed AR-target gene. E2 could promote proliferation of PCa cells through the up-regulation of SOX4 under androgen-depleted environment. Our data provides a possible molecular basis for the overexpression of SOX4 in CRPC and may facilitate the detection and prevention of the emergence of CRPC.