James Gardner was born in Paisley, Renfew, Scotland in 1829. The family moved to Glasgow and in 1850. His older brother, Alexander Gardner, was interested in the socialist ideas being advocated by Robert Owen. Inspired by the New Harmony community established by Robert Dale Owen and Fanny Wright in Indiana, Gardner helped establish the Clydesdale Joint Stock Agricultural & Commercial Company. The plan was to raise funds and to acquire land in the United States.