As Fallon appeared on television to read a prepared statement, repeating his letter to May, the U.S. government said the resignation "should in no way impact our relationship" with Britain.
In his resignation letter to Prime Minister Theresa May, Fallon wrote: "A number of allegations have surfaced about MPs in recent days including some about my previous conduct".
Fallon had apologised this week for putting his hand on the knee of a journalist, Julia Hartley-Brewer, in 2002, but he was not being investigated over the incident.
Ruth Davidson, leader of the Conservatives in Scotland, said it was time to break with a culture in politics when powerful people "use positions of power to demand things from others".
Media captionSir Michael Fallon: "Not right for me to go on as defence secretary".
The ex-cabinet minister has tonight said: "In recent days allegations have been made about MPs' conduct, including my own".
Sir Michael is the first politician to quit following recently revealed claims of serious sexual abuse in Parliament.
Mrs May said she appreciated the "serious manner" in which Sir Michael had considered his Cabinet role and paid tribute to "a long and impressive ministerial career". The Telegraph understands that there will not be a reshuffle but that Sir Michael will be directly replaced.
Some observers questioned the wisdom of moving the chief whip at a time when the government is fragile and every vote it faces in the House of Commons is on a knife-edge.
The prime minister said on Wednesday that action would be taken when there were allegations and evidence of sexual misconduct.
"We have a duty to ensure that everyone coming here to contribute to public life is treated with respect."