The problem with this whole debate is it mixes up different meanings of the word "British".
First there's the ethnic sense. The ethic group, the British people. If you're brown you're not ethnically British, simple as.
Second there's the legal sense. If someone is a UK citizen, they can be said to be a British citizen, and therefore legally British.
Third there's the cultural sense. This one's a little more vague. But generally a good rule of thumb is: could you tell that someone is foreign from a 5 minute phone call? If not then they're probably culturally British.
Often there is overlap, but in principle it's possible to be any one type of British without the other two.
Steve Laws, Rupert Lowe, most people in the UK: ethnically, legally and culturally British.
Rishi Sunak, Krishnan Guru-Murthy, other 2nd or 3rd generation immigrants: legally British, arguably cultural British. Not ethnically British (they're brown).
People living in the British Isles after the Anglo-Saxon invasions: ethnically British. But neither legally or cultural British (the UK didn't even exist yet).
And today there is a worryingly large group of people who are not ethnically British, can barely speak the language, and are only British in the legal sense because they have a British passport.
I think there's no way to deny this reasoning. The British people exist as an ethnic group. Brown people cannot be ethnically British. It's not a hateful or exclusionary statement anymore than saying a man who is 4ft tall isn't tall.
And also it clears up the various ways brown people claim to be British. You have a UK passport? Yes you're legally British, but not ethnically.
You were born and raised in Britain, love football and tea ad the rest of it? That's British in the cultural sense, not the ethnic sense.
If you're clear about these different meanings then there's no confusion.