At an event in Bihar on Tuesday, Singh said: "If Rajiv Gandhi had married a Nigerian and if she wasn't white skin, would Congress accept her (Sonia Gandhi) as a leader?"
The remark created a furore with the Congress lashing out at the union minister of state for micro, small and medium enterprises and the ruling Bharatiya Janata party (BJP) distancing itself from it.
Singh, however, said "it was an off the record remark".
"People do talk off the record with each other and it's a different thing how it is presented. If I have hurt anyone's sentiments, including Sonia and Rahul, I regret what I said," he said.
Congress leaders condemned the remark and demanded action against him.
Senior Congress leader Digvijaya Singh said: "Action should be taken against him."
He also said that it was not the first time Singh made derogatory remarks. "It reflects the mindset of BJP," he added.
However, the BJP distanced itself from the controversy.
"The BJP doesn't do politics over colour or caste. We believe in 'Sabka saath sabka vikaas," party leader Shahnawaz Hussain said.