Hey, thanks for reading! We are talking about a word that is a racist hate crime, that the black community started using as a way to take away its demeaning power. So no, absolutely not racist to say only black people have a right to use the word.
Probably it time to stop using it all together.
P.S.: There is no “reverse racism".