Some key members of the Kobe Yamaguchi-gumi have left to form a splinter group in central Hyogo prefecture, Japanese media report.
One member said they were unhappy at the way their former gang chief was collecting money.
Police said they would be on alert for any possible violence.
The Yamaguchi-gumi gang is the largest of Japan's organised crime syndicates, known as the yakuza. Police say it has about 23,000 members.
Yakuza gangs have been part of Japanese society for a long time; there are believed to be about 60,000 members nationwide.
The gangs themselves are not illegal, although much of their earnings are gained illicitly through gambling, prostitution, drug trafficking and, increasingly, cyber-hacking.
The new group told reporters it would be called Ninkyo Daintai Yamaguchi-gumi and, although it would not have a boss, would be represented by Yoshinori Oda, 50, a former executive of the old gang.
The Kobe Yamaguchi-gumi gang was formed two years ago after powerful factions split away from the main Yamaguchi-gumi group.
Japan's police fear turf battle among yakuza after split in Kobe Yamaguchi-gumi gang
Police in Japan are on high alert after one of the country's largest yakuza crime gangs split into rival factions sparking fears of a turf battle.
Japanese media have reported that members of the Kobe Yamaguchi-gumi have formed a splinter group in Hyogo prefecture, between Osaka and Kobe, on Sunday (30 April).
The former members had once belonged to the Kobe Yamaguchi-gumi after they split from the Yamaguchi-gumi in August 2015 in the largest spilt within the Japanese criminal underworld since 1984.
The BBC reported that one member of the new syndicate said they were unhappy at the way their former gang chief was collecting money.
The new group announced they would be called Ninkyo Daintai Yamaguchi-gumi and, although it would not have a boss, the de facto leader would be Yoshinori Oda, 50, a former high-ranking member of the Kobe Yamaguchi-gumi.
Famed for their elaborate tattoos Yakuza crime gangs contain around 102,000 members with the Yamaguchi-gumi gang the largest with around 23,000 members.
The gangs make money from prostitution, drug trafficking, gambling and even cyber-hacking.
Once a tolerated evil in Japan's society, the gangs are not actually illegal although police in 2016 began cracking down on many of their activities.
Fuji TV reported that former leaders of the Kobe Yamaguchi-gumi were seen after congregating at an office in Amagasaki City, Hyogo Prefecture.
Japan Today reported that a police spokesman said the meeting was most likely held to form a new breakaway group and they would be on alert for any possible violence.
Between March and September 2016, police reportedly arrested 976 gang members linked to yakuza criminal gangs as factions competed for territory, both across the country and internationally, after the 2015 split.
The arrests were made often for minor infractions, incurring huge legal bills and jailing some mobsters for small prison terms in order to frustrate the group's leadership.
The separation of the two factions left at least 25 dead, including a police officer, and 70 more injured in a series of incidents which injured innocent bystanders as well as gangsters.