I find that people of the English speaking world have also invested quite a lot of energy in the belief that their countries are no longer ‘colonial’ (like they assume mine was). They know better. They would never do a thing like colonialism, not them personally. It’s not that they would feel comfortable speaking to an actual black person — they wouldn’t — they just think that I should learn better manners when making reference to them.
There are specific forms of jargon that one absolutely needs to be able to refer to “the other” in a way that separates one from seeming to have any mean, colonial intent. I have to learn this from these good people, otherwise I’m still among the great unwashed.