According to a report in Daily Mail, the brawl started after the two sides exchanged insults before throwing plates as other visitors in the restaurant tried to separate them.
Colombian actor Manolo Cardona and TV presenter Carolina Guerra were seen involved in the fight. One person was injured in the fight.
Earlier, Brazilian fans burst firecrackers to distract Colombian footballers
The restaurant brawl followed the Brazilian fans' setting off fireworks outside the Colombian team hotel at 4 am before the quarterfinal clash to disrupt their focus.
Both teams are under tremendous pressure ahead of the South American derby. While Brazil have drawn one of their four matches played so far, Colombia have won all their matches. In the pre-quarterfinals, Brazil had to sweat it out to beat Chile in the penalty shootouts while Colombia had an easy win over former champions Uruguay.