The Game of Death is an incomplete 1972 Hong Kong martial arts film directed, written, produced by and starring Bruce Lee, in his final film attempt. Lee died during the making of the film. Over 100 minutes of footage was shot prior to his death, some of which was later misplaced in the Golden Harvest archives. The remaining footage has been released with Lee's original Cantonese and English dialogue, with John Little dubbing Lee's Hai Tien character as part of the documentary entitled Bruce Lee:

The original plot involves Lee playing the role of Hai Tien, a retired champion martial artist who is confronted by Korean underworld gangs. They tell him the story of a pagoda where guns are prohibited, and under heavy guard by highly skilled martial artists who are protecting something (which is not identified at all in any surviving material) held on its top level. The gang boss wants Hai to be a part of a group whose purpose is to retrieve said item. They would be the second group to try to do so as the first attempt with a previous group had failed. When Hai refuses, his younger sister and brother are kidnapped, forcing him to participate. Hai, as well as four other martial artists (two of whom were played by James Tien and Chieh Yuan), then fight their way up a five-level pagoda, encountering a different challenge on each floor. The setting of the pagoda was at Beopjusa temple in Songnisan National Park in South Korea.  The pagoda, called Palsang-jon, is the only remaining wooden pagoda in South Korea. At the base of the pagoda they fight 10 people, all black belts in Karate. While inside the pagoda, they encounter a different opponent on each floor, each more challenging than the last. Although his allies try to help out, they are handily defeated, and Hai must face each of the martial artists in one-on-one combat. He defeats Filipino Eskrima master Dan Inosanto, Korean Hapkido master Ji Han-jae, and finally Kareem Abdul-Jabbar, who fights with a free and fluid style mirroring Lee's Jeet Kune Do. Because Abdul-Jabbar's character has great size and strength in addition to a fighting style as potent as Lee's, he can only be defeated once Hai recognises that an unusually high sensitivity to light is his greatest weakness.  Immediately after defeating the giant guardian, Hai turns around and descends the staircase, heading out of the pagoda. Despite all the talk of something awaiting up top of the (now unguarded) flight of stairs, there is no mention of anyone going up to retrieve it. No surviving material explains how this will affect Hai or his captive siblings.

You noted there were gangs, was there a gang boss?
The gang boss wants Hai to be a part of a group whose purpose is to retrieve said item.