On Monday, Lise Grande, the United Nations humanitarian coordinator for Iraq, said more than 30,000 people had already fled the city and surrounding areas.
Iraqi government forces broke through ISIL lines inside Tal Afar on Friday and reached the old city centre and the neighbourhood around the Ottoman-era citadel, the military said.
A security official who spoke on condition of anonymity said that based on records of prisoners who were at Badoush most are believed to have been Shiite Muslims or other minorities.
A massacre at Badoush in June 2014 left 600 male inmates dead.
Earlier Friday, another Iraqi military source, speaking anonymously due to restrictions on talking to media, told Anadolu Agency that 6,000 civilians had been evacuated from Tal Afar within the past 48 hours due to fierce fighting between Iraqi forces and Daesh.
Iraqi military spokeman General Yahya Rasool told AP news agency on Friday after taking control of several neighbourhoods, Iraqi forces are now on the outskirts of the neighbourhood of al-Qalaa.
A predominantly Turkmen district, Tal Afar was overrun by Islamic State of Iraq and the Levant fighters in 2014.
Iraqi forces backed by a USA -led coalition launched fresh military operations on Sunday aimed at decisively wresting Tal Afar from the grip of the terrorist group.
The spiritual leader of Iraq's Shiite majority is calling on doctors from across Iraq to help civilians fleeing clashes in the latest fight against the Islamic State group in Iraq.
The city was cut off during the nine-month Mosul offensive by troops and allied militiamen from the Shia-dominated paramilitary Popular Mobilisation (Hashd al-Shaabi) force.
Rasool says about 10,000 civilians remain inside the city but that authorities have secured safe corridors for them.