India's grievance with Pakistan was that about 42 terrorist camps were functioning in that country, Antony told.
"Without destroying these terrorist camps, it would be difficult to have permanent cooperation in Indo-Pak relations," he said.
However, government view was to continue with dialogue, he said, adding, Defence Secretaries of India and Pakistan would hold discussions on May 30 and 31.
Emphasising on the need for dialogue, Antony said India wants to have good relations with Pakistan.
"Dialogue is an important factor. Without dialogue nothing can be achieved," he said.
Antony said government's stand was clear that it wants to maintain a cordial relationship with all its neighbours.
"We can change our friends, but we cannot change our neighbours," he said, adding "We know the reality that we have to live along with our neighbours. Even though we have problems with our neighbours, we are trying to re-establish our friendship."
Pointing out that India has boundary dispute with China, he said efforts were being made to solve the problem.
The Indo-Chinese defence cooperation which was stopped recently, would be started again and in June a defence group from India would be visiting China, he said.