Can you reject influence on the label shift that happens if someone gains influence on you while already having it?

Can you reject influence on the label shift that happens if someone gains influence on you while already having it?

Can you reject influence on the label shift that happens if someone gains influence on you while already having it? We’re having a bit of a disagreement in my group about this and were hoping for an official clarification.