They believe cultural diversity is good for the country but they're worried that cultural differences will stop everyone from getting along.
An 11-year study by a collaboration of Australian universities has found 85 per cent of Australians acknowledge racial prejudice occurs in the nation, and one in five has been a victim of racist verbal abuse.
The study found that 6.5 per cent of the 16,000 Australians surveyed were against multiculturalism, The Sydney Morning Herald reported.
Professor Kevin Dunn, from the University of Western Sydney's school of social science, said the study revealed that the majority of Australians are pro-multiculturalism but are anxious that the diversity will not be managed well.
"Over 40 per cent of those surveyed feel that cultural differences pose a threat to societal harmony. So if you take that alongside the 87 per cent that are pro-multiculturalism, clearly you've got a third of the nation that tolerate cultural diversity, but are concerned at the impact it will have on society," Professor Dunn said.
"The Cronulla riots and the recent attacks on people of Indian descent are an example of this. The figures show that 85 per cent of Australia acknowledge there is racial prejudice in the country," he added.
Dunn believes previous governments have done nothing to address the issue for the past decade, with Aboriginal and Torres Strait Islander Social Justice Commissioner Tom Calma spending only part of his time dealing with race discrimination.
"For the last decade, the government hasn't appointed a full-time Race Discrimination Commissioner. Take the issue of the Indian students recently. There's no way that the commission is resourced enough to deal with their (the Indians') reports (of violent attacks on them), and to offer support and advice," he said.
The survey also found that at least one in five Australians experience verbal abuse such as offensive slang names for different cultural groups, or swearing and offensive gestures, while 11 per cent feel they don't belong or are inferior.
The study is part of the 'Challenging Racism Project'.