Answer by taking a quote from the following article:

Graziano was the son of Ida Scinto and Nicola Barbella. Barbella, nicknamed Fighting Nick Bob, was a boxer with a brief fighting record. Born in Brooklyn, Graziano later moved to an Italian enclave centered on East 10th Street, between First Avenue and Avenue A in Manhattan's East Village. He grew up as a street fighter and learned to look after himself before he could read or write.

Graziano heard from a couple of his friends about a tournament going on with a gold medal for the winner. He entered under the name of Joe Giuliani and was trained by Tobias (Toby) Zaccaria of Kings County (Brooklyn), NY. He fought four matches and ended up winning the New York Metropolitan Amateur Athletic Union Boxing Competition (1939). He sold the gold medal for $15 and decided that boxing was a good way to make cash.  A couple of weeks into amateur fighting, Graziano was picked up for stealing from a school. He went to Coxsackie Correctional Facility, where he spent three weeks, with boyhood friend Jake LaMotta, and then he went on to the New York City Reformatory where he spent five months. After he got out of the reformatory, he headed back to the gym to earn money and while there, met Eddie Cocco who started his professional career. He entered the ring under the name Robert Barber. A couple of weeks later, Graziano was charged with a probation violation and sent back to reform school where he was charged with starting a minor  riot. He was then sent to Rikers Island.  When Graziano got out of jail he enlisted in the military but went AWOL after punching a captain. He escaped from Fort Dix in New Jersey and started his real boxing career under the name of "Rocky Graziano". He won his first couple of bouts. After gaining popularity under the name of Graziano, he was found by the military. After his fourth bout, he was called into manager's office to speak with a couple of military personnel. Expecting to be prosecuted and sent back to the military or jail, he fled. He returned to the military a week later. He turned himself in, but he was pardoned and given the opportunity to fight under the army's aegis.

Did he go back to jail?
he was called into manager's office to speak with a couple of military personnel. Expecting to be prosecuted and sent back to the military or jail, he fled.