Poll officials said 66.6 percent voting was recorded till 3 p.m.
"Balloting has been largely peaceful and is moving on smoothly. No untoward incident has been reported yet," an official said.
Incumbent MP Lalubhai Patel of the Bharatiya Janata Party faces Ketan Dahyabhai Patel of the Congress, Bhavesh Patel of the Bahujan Samaj Party and Kessur Goan of the Aam Aadmi Party.
Voting was moderate in the first couple of hours, with 14.35 percent turnout after polling began at 7 a.m. However, it picked up during the day. By 1 p.m., the turnout was 52.65 percent.
District Election Officer Ramesh Verma said the turnout was expected to be higher than the 71.32 percent recorded in the 2009 polls.
Polling is taking place Wednesday at 119 polling stations, with over 111,000 people eligible to exercise their franchise.
Daman and Diu comprise two broadly detached districts in the country's western coast. Daman is an enclave on Gujarat's southern coast, situated 160 km north of Mumbai, while Diu covers an island off the southern coast of Gujarat's Kathiawar Peninsula, 64 km southeast of Veraval, as well as a small area on the mainland.
The union territory is a former Portuguese colony.