The 200-year-old statues of the British ruler were kept in the backyard of the museum for installation in a Queen Victoria park, work on which is underway.
Three men who came to the museum Friday after buying tickets vandalized the statues.
Security personnel nabbed two of them while the third escaped.
A police official told IANS that they were on lookout for others who abetted the crime. The two arrested youth - Rinku and Arvind Tyagi - are from Vallabhgarh and Faridabad, near New Delhi.
The accused told police they were incensed by the "symbols of British colonialism" and had decided to do away with them.