After a lot of thought, I think I have synthesised our stance on this issue to:
Don't be a dick, unless you want to piss people off.
In the next section I am going to use the word hate a lot more than usual, I do this because where people react to use of culture they don't like, they respond viscerally, as do people getting called out. It is due to the cognitive dissonance that we feel when encountering opposing world views and culture is the foundation of identity in the broadest sense of the word (resists getting sidetracked by making points about the reductionism of identity politics).
In the cultural sphere, everything has to be taken in context. People love aspects of their culture and dislike how it is used by some. Many geeks hate Big Bang Theory, with reason. It is black-face but with nerds. @Hypershock hates the Me Hallelujah ad because it is an example of crass use of a Christian Hymn, it annoys me too despite not generally worrying about offending Christians. Indigenous peoples hate the use of sacred imagery by the dominant culture in their societies. It is worth keeping in mind that crass use of someone's culture may piss them off, it may piss off anyone who respects that culture, and you do so at your own risk.
Sometimes people want to offend another's culture, and if you are happy to wear the flack, there should be no prohibition against it. There are a lot of situations where cultural practice needs to be ridiculed for the sake of progress. One of the best examples I can think of is the Sisters of Perpetual Indulgence campaign against homophobia in the Northern Territory, where a group of gay men dressed as Nuns and performed an exorcism on the NT Parliament. The Catholic Church has long campaigned against gay rights and deserved the ridicule.
In general, people hate having their personal choices policed and hate seeing others get called out for something that is, to them, inoffensive.
When someone calls-out cultural appropriation, there is a good chance that they are attacking someone else's identity. A festival goer in a native American headdress probably wears it to signify that they reject the dominant culture and identify with the 'natives'. This is a long tradition, even used by the participants in the Boston Tea Party, that helped kick off the American Revolution. The headdress wearer is being culturally insensitive, but criticism would be better received and importantly, more effective, if it was explained patiently why it is uncool rather than subjecting them to hostile ridicule, especially because they don't mean offence.
For the most part, people like to share their culture. Calling out people for adopting cultural practices that don't offend the culture's practitioners is a massive dick move. Calling out someone for wearing a Kimono for example, whilst Japanese Kimono makers are trying to market them globally is actually an attack on Kimonos as a cultural item.
Culture is not static, everything is a remix. If people don't participate in cultures, they die. If locals aren't so interested in traditional cultural items due to identifying with other (usually modern) cultures, sharing more broadly is a good option to keep it alive.
Some other things to think about:
Are there gatekeepers to a culture? Who gets to decide if a cultural practice if offensive within a culture? Sometimes a few get offended by a specific cultural practice, whilst most don't care, other times it is cut and dried and everyone within a culture thinks other's use of it is a dick move.
Despite what some people say, all cultures can be used in a way that causes offence to its participants and no-one other than the participants themselves can decide who gets offended. The idea that nerd culture can't be appropriated whilst cultures based location can be appropriated is just bullshit. It goes back to the point don't be a dick unless you want to piss someone off. Claiming that one culture is valuable and another culture is not, pisses off the participants in the culture you devalue. Do they perhaps need offending? That is up to you.