Gifford was born in Santa Monica, California, the son of Lola Mae (nee Hawkins) and Weldon Gifford, an oil driller. He graduated from Bakersfield High School. Following Gifford's death in 2015, his wife Kathie Lee Gifford said that her late husband grew up in a poverty-stricken home and that he and his family sometimes ate dog food. She said they lived in 29 places even before Gifford attended high school because his father could not find work during the Depression.

Gifford married his college sweetheart, USC's homecoming queen Maxine Avis Ewart, on January 13, 1952, after she became pregnant while they were students at USC. They had three children, Jeff (b. 1952), Kyle and Victoria, and five grandchildren. Victoria married Michael LeMoyne Kennedy, son of Robert F. Kennedy. Gifford was then married to fitness trainer Astrid Lindley from 1978 to 1986. The first two marriages ended in divorce. Gifford married television presenter and singer Kathie Lee Johnson, who was 23 years his junior, on October 18, 1986. The couple settled in Greenwich, Connecticut, with their son, Cody Newton Gifford, and daughter, Cassidy Erin Gifford. Gifford and his third wife Kathie Lee both shared the same birthday, which was August 16. The couple co-hosted ABC's coverage of the 1988 Winter Olympics in Calgary.  Gifford had an older sister and younger brother, Winona and Waine.  In 1997, the tabloid magazine Globe arranged to have Gifford secretly videotaped being seduced by former flight attendant Suzen Johnson in a New York City hotel room. They published photos and stories. ESPN reported that the tabloid paid Johnson $75,000 to lure Gifford to the room, while The Atlantic said it was $125,000. National Enquirer Editor Steve Coz observed, "There's a difference between reporting the news and creating the news ... [w]ithout The Globe, there would be no story here. I'm in the tabloid industry, and this is way over the top. It's downright cruel."  According to the former lawyer of Johnny Carson, Henry Bushkin, Gifford had an affair with Carson's second wife Joanne in 1970.

How long was their marriage?