Michael Jackson, who drastically changed his face with plastic surgery through the years, talked several times about how his father would mock the size of his once-broad nose, calling him "big nose".
It was gathered that Joe was hospitalised in June with terminal cancer.
"We were not being told where he was and couldn't get the full picture".
In 1979, Michael broke from the group - as well as his father's management - and went to even greater acclaim with the landmark albums "Off the Wall" and "Thriller".
Jackson had also suffered minor strokes before that, including one in 2012.
Joe's family journey was highlighted in the ABC mini-series, The Jacksons: An American Dream, in 1992, and in VH1's Man in the Mirror: The Michael Jackson Story in 2004.
They released I Want You Back in 1969 and the hit is a favourite to this day.
He also began his daughter Janet's career, helping her become a recording artist at just 16 years old. Janet Jackson said her father only beat her once, but that they had a distant relationship.
In an interview with Oprah Winfrey in 1993, Michael Jackson revealed that the drive behind his and his brothers' early success was a harsh regime enforced by Joe, bordering on physical and mental abuse.
The outlet reported Jackson had planned to live out his final days in Vegas, but it had not happened.
He said Joe presided over rehearsals with a belt.
During the 1960s, Jackson took control of his sons' burgeoning musical efforts and the Jackson 5 was born, later signing to Motown records in 1967. "'I'm Joseph to you.'" Receiving the inaugural Janet Jackson award at the 2018 Radio Disney music awards on 22 June, she paid tribute to "my incredible father", who "drove me to be the best that I can". Joe is very weak and getting significantly weaker.
He was married to Katherine Jackson and together they had 11 children.