Stephen Smith grew up as an indentured servant in Pennsylvania. From a young age, he was assigned to work in the lumberyards by Thomas Boude, whose wealth stemmed from his extensive lumber business. After buying his freedom for $50 at the age of 21, he continued to work in the lumberyards until establishing his own lumber business in 1822, as well as dealing coal. By the 1850s, Smith was grossing $100,000 in annual sales. By 1857, Smith was worth $500,000 (approximately $13.5 million today). On top of being a businessman, Smith was a minister and served as chairman of the black abolitionist organization in Columbia, PA.