Born in 1934 and the youngest of seven siblings, Harold Brown, also known as Hal, grew up in York, Pennsylvania, a small working-class town. After graduating high school, Brown attended Penn State, but left after his first semester in order to play minor league baseball with the St. Louis Browns. After sustaining an injury, Brown moved to San Diego in 1953 to attend San Diego State College on a basketball scholarship. His studies were interrupted by two years of military service, but he returned to SDSC and finished his degree in physical education and speech. Upon graduating in 1961, Brown began a six-year teaching career at a local junior high school. During this time, he also became extremely active in the San Diego Civil Rights Movement. Brown was a member of the El Cajon Valley Open Housing Committee, which sought to integrate neighborhoods in El Cajon and La Mesa. In addition, he helped to found and chair the local chapter of the Congress of Racial Equality (CORE). This organization fought unfair employment practices, and organized civil rights marches, demonstrations and sit-ins across San Diego.

Later, Brown became the Deputy Director for the US Peace Corps in Lesotho. In 1971, he returned to the US and became California State University's Assistant to the Vice President for Academic Affairs and Assistant to the Vice President for Administration, making him the first Black administrator at San Diego State. Later, he served as the Associate Dean of Planning and Director of the Afro-American Studies Program, which he helped to establish. He hired many of the department's faculty and helped create the program's curriculum. In 1973, he was appointed Director of Campus Information Systems. Afterward, he became the Associate Dean for External Relations in the College of Business Administration. As Associate Dean, he developed a certificate program in community economic development and founded the Center for Community Economic Development (CCED). The purpose of the program was to create economic self-reliance in less affluent communities, such as southeast San Diego, by offering classes in accounting, computers, financial planning, and leadership. The CCED also provided resources and consulting for community groups. It was one of only a handful of such programs in the United States. Brown retired in 1997, but continued consulting until 2004. Harold Brown has been the recipient of numerous awards both for his service to the university and to the community.