If you told me that you weren't hired for a job my first response definitely wouldn't be "it's because of how you look", it would have been "Well maybe there are other candidates who meet their qualifications a lot more."
I'd assume that you dressed properly for an interview.