More explosions were then heard coming from inside the building.
Sohrab Qaderi, a member of the provincial council, told Reuters that at least four militants, armed with rocket-propelled grenades and machine guns, are still fighting the police.
Following the explosion, several gunmen opened fire and engaged in clashes with the security forces near the government facility.
The militant attack is the latest in a series across Afghanistan this year.
Attahullah Khogyani, the spokesman for the governor of Nangarhar province, said a suicide vehicle bomb exploded outside the building, which belongs to the provincial finance directorate.
Jalalabad is the capital of restive Nangarhar, which borders Pakistan.
"There were eight terrorists who carried out a co-ordinated attack in Jalalabad city, killing ten people", Khogyani said.
Security forces and attackers are now fighting each other.
No individual or group including the Taliban militants has so far claimed responsibility behind the attack.
Dozens of people have been killed in voter registration centres in recent weeks, leading to fears that people could stay away from elections that are seen as a major test of the government's credibility.
The Taliban and IS have made clear their intention of disrupting the elections.
Operation Al Khandaq will target USA forces and "their intelligence agents" as well as their "internal supporters", a Taliban statement said in late April.
Nine journalists, including AFP's chief photographer in Kabul Shah Marai, were among 25 people killed in a double suicide blast claimed by IS in the capital on April 30.