The dramatic blaze resulted in the building's partial collapse according to a Sao Paulo Fire Department spokesperson.
Brazilian President Michel Temer visited the scene early Tuesday morning, but rushed away after some victims began throwing rocks and debris at his vehicle.
More than 150 firefighters are battling the blaze which is thought to have been caused by a gas explosion.
The building was owned by the federal government and had previously served as the headquarters of the Federal Police.
There is no official information on the number of missing persons or on the identity of the one confirmed victim.
Television images showed the horrifying moment when a fireman tried to reach a man clinging to the building as it went down more than one hour after the fire started. About 50 families reportedly were living in the abandoned building.
Romulo de Souza, 49, said he was part of a squatter occupation in the neighboring building.
"Happily the majority got out", he said. People were seen jumping from the upper floors in an attempt to escape the fire, according to eyewitnesses.
After starting a fire in the house started evacuation, but it is unknown how many people were in the building at the time of the fire and managed to evacuate.
Firefighters believe clearing the rubble, which was still smouldering hours afterward, could take days. The building, which was abandoned and was being used by homeless people for shelter, collapsed even as firefighters were trying to bring the flames under control.
Sao Paulo is Brazil's financial capital and the most populous city in Latin America but suffers huge economic inequality.
Then-mayor of Sao Paulo, Joao Doria, had launched an aggressive crackdown on these communities.