I hate to say it, but it sounds as if this person is critiquing YOU rather than your work. Maybe gently ask where in the work itself he thinks you've gone wrong? (If you haven't already done that.) Maybe take him a step further. What he's saying, in essence, is that no writer can write outwith their own experience. Which, of course, is nonsense. Any writer who writes a story with multiple characters is writing outwith their own experience. Is he saying that nobody can write a historical novel? Or that nobody can write a futuristic novel? Nobody can write a story set in another country, or with a character who comes from another country? Tell him that's what research is for. Also, writing from your own perspective doesn't allow for the kind of insight that outsiders can give. To paraphrase Robert Burns: Oh, if some power would give us the gift to see ourselves as others see us!
Actually the person is a pretty insightful critic on content (and a great writer), I just used her as an example of things you tune out vs. things you accept (An old rabbi of mine used to say "eat the fish, spit out the bones.") But yeah that's a whole different debate, and yes she does generally think it's inherently problematic to write from the perspective of someone having cultural issues outside of one's own experience - especially in the case of white writers entering the thoughts of characters whose peoples were colonized by white people - and in fairness she does sincerely believe that and apply it to her own writing. It's just something that she and I aren't ever going to agree on, which is okay and it's just a matter of learning how to co-exist with people who have radically different moral beliefs. Acknowledge that they're never going to change any more than they want you do, and stay friends (that, and she really likes one of my stories where I'm writing from a the point of view of a non-white woman with a history of sex-slavery and human trafficking - so that goes to show that we're all willing to blur our "lines" if the story is good - I know I do.)
I don't understand why people who make those arguments only stick to criticizing cultural appropriation on the basis of "white male privilege." Cos if you believe that, wouldn't you also believe that it's inherently problematic for men to write female characters? Why does racial or cultural privilege make it problematic to write other races or cultures, but gender privilege doesn't make it problematic to write from other genders? Someone explain the inconsistency to me, because I don't get it.
In general the debate is moving that direction anyway. But honestly this thread isn't supposed to be about that so sorry for diverting it.