Taylor spent most of 1973 enjoying his new life as a married man and did not return to the recording studio until January 1974, when sessions for his fifth album began. Walking Man was released in June and featured appearances of Paul and Linda McCartney and guitarist David Spinozza. The album was a critical and commercial disaster and was his first album to miss the Top 5 since his contract with Warner. It received poor reviews and sold only 300,000 copies in the United States. The title track failed to appear on the Top 100.  However, James Taylor's artistic fortunes spiked again in 1975 when the Gold album Gorilla reached #6 and provided one of his biggest hit singles, a cover version of Marvin Gaye's "How Sweet It Is (To Be Loved by You)", featuring wife Carly on backing vocals and reached #5 in America and #1 in Canada. On the Billboard Adult Contemporary chart, the track also reached the top, and the follow-up single, the feelgood "Mexico" also reached the Top 5 of that list. A well-received album, Gorilla showcased Taylor's electric, lighter side that was evident on Walking Man. However, it was arguably a more consistent and fresher-sounding Taylor, with classics such as "Mexico", "Wandering" and "Angry Blues". It also featured a song about his daughter Sally, "Sarah Maria".  Gorilla was followed in 1976 by In the Pocket, Taylor's last studio album to be released under Warner Bros. Records. The album found him with many colleagues and friends, including Art Garfunkel, David Crosby, Bonnie Raitt and Stevie Wonder (who co-wrote a song with Taylor and contributed a harmonica solo). A melodic album, it was highlighted with the single "Shower the People", an enduring classic that hit #1 on the Adult Contemporary chart and almost hit the Top 20 of the Pop Charts. However, the album was not well received, reaching #16 and being criticized, particularly by Rolling Stone. Still, In The Pocket went on to be certified gold.  With the close of Taylor's contract with Warner, in November, the label released Greatest Hits, the album that comprised most of his best work between 1970 and 1976. With time, it became his best-selling album ever. It was certified 11x Platinum in the US, earned a Diamond certification by the RIAA, and eventually sold close to 20 million copies worldwide.

Answer this question "what was some of his downs?" by extracting the answer from the text above.
The album was a critical and commercial disaster and was his first album to miss the Top 5 since his contract with Warner.