I don't think he made it up. I think he believe's what he's saying.
I think he assumes it was more than just sex because as he admitted he was infatuated with her and she didn't see it for him like that.
Brandy didn't need to say anything at all and she also need to fess up or explain her sex life either.