James was the son of William Preston (Billy) Agee & Martha Pamelia Eleanor Goodwin.

Martha was the daughter of John Marion Wilkins Agee & America Elizabeth Mount.