James was born in Holton, Kansas; his mother, died in 1954 when he was five. His father was a janitor and a handyman. After four years at the University of Kansas residing at Stephenson Scholarship hall, James joined the Army in 1971. He was the last person in Kansas to be sent to fight in the Vietnam War, although he never saw action there.

In his Baseball Book 1990, James heavily criticized the methodology of the Dowd Report, which was an investigation (commissioned by baseball commissioner Bart Giamatti) on the gambling activities of Pete Rose. James reproached commissioner Giamatti and his successor, Fay Vincent, for their acceptance of the Dowd Report as the final word on Rose's gambling. (James' attitude on the matter surprised many fans, especially after the writer had been deeply critical of Rose in the past, especially what James considered to be Rose's selfish pursuit of Ty Cobb's all-time record for base hits.)  James expanded his defense of Rose in his 2001 book The New Historical Baseball Abstract, with a detailed explanation of why he found the case against Rose flimsy. James wrote "I would characterize the evidence that Rose bet on baseball as...well, not quite non-existent. It is extremely weak." This countered the popular opinion that the case against Rose was a slam dunk, and several critics claimed that James misstated some of the evidence in his defense of Rose. Derek Zumsteg of Baseball Prospectus wrote an exhaustive review of the case James made and concluded: "James' defense of Rose is filled with oversights, errors in judgment, failures in research, and is a great disservice to the many people who have looked to him for a balanced and fair take on this complicated and important issue."  In 2004, Rose admitted publicly that he had bet on baseball and confirmed the Dowd Report was correct. James remained steadfast, continuing to insist that the evidence available to Dowd at the time was insufficient to reach the conclusion that it did.

Why did he want to defend Rose?
James wrote "I would characterize the evidence that Rose bet on baseball as...well, not quite non-existent. It is extremely weak.