There's a lot for a factors tbh. The first half of her career she was America's sweetheart, and I think being so prim & proper alienated a lot of Black people, in the same way Whitney got a lot of backlash for being "whitewashed". Then, since Afro and the whole marriage scandal people have realised that a lot of that image was "put on", which alienated her further because people thought she was fake
By this time Beyoncé and Rih were the main girls, meanwhile Brandy was taking 4 years breaks so she inadvertently let them take her spot.
So you can attribute it to a lack of consistency and having a corny image. I do think the last five years the narrative has shifted a lot and I'd even say it's "cool" to like Brandy now - I think the Two Eleven performances and the whole 90s resurgence thing helped a lot - even mainstream publications refer to her as the "vocal bible".
