James married first 31 Dec 1878 to Nancy Jane (Nannie) Agee, the daughter of Jonathan Bethel Agee and Sarah Elizabeth Burrow. He married second in 1908 to Martha Pamelia Eleanor Goodwin, the daughter of James C. Goodwin and Eliza Jane Farmer. Martha was married first to William Preston (Billy) Agee, the brother of James.