Non-Anglo white person here. I have definitely been chastised in public, in the US, for speaking my native language. There's a big intersection between xenophobia and racism, but you are absolutely right: They aren't the same.