lol @ the first part.
For the second part, when did you put your belt on? Did your instructor put it on then ask you to put it off? How did you decide that you are black belt?
You can't drop in rank if you were never increased in the first place.
Basically, how was it communicated to you that you're a black belt?
Well that is the problem.
Even if he apologises and put the brown belt, the ibjjf could just turn around and give the same story they did to that barvo guy.