Ed was born in Marlin, Texas, on February 22, 1929 to E. N. Smith Sr. and Bessie Edmundson Smith. He attended Marshall schools, college at Kemper Military School and Baylor University, then Baylor Law School. He met and married the love of his life, Geraldine Norris, on November 23, 1950. His law school was interrupted by a call to the U.S. Air Force, where he served as Procurement Officer at Barksdale Air Force Base. After his service, he returned to Baylor and finished law school.