In our society, it's unacceptable to treat people this way who belong to a different religion or ethnic group. Maybe it's because people think fat people could and should change their lifestyle. Our culture has worshipped thin for so long that it perceives having bigger bodies as something to be avoided.
We tend to ignore what we're afraid of becoming. You see this when someone is grieving a lost loved one or who is terminally ill. This applies to encountering homeless people as well. The culture has taught us to see them as "problems", not people.