The following section, from Erving Goffman’s essay On Face Work, understood in political terms, is a lot more helpful than explanations provided by those who generally appeal to facts as being a trump card (‘appealing to the referee’, or ‘speaking to the manager’).
Facts are of the schoolboy’s world—they can be altered by diligent effort but they cannot be avoided. But what the person protects and defends and invests his feelings in is an idea about himself, and ideas are vulnerable not to facts and things but to communications. Communications belong to a less punitive scheme than do facts, for communications can be by-passed, withdrawn from, disbelieved, conveniently misunderstood, and tactfully conveyed. And even should the person misbehave and break the truce he has made with society, punishment need not be the consequence. If the offense is one that the offended persons can let go by without losing too much face, then they are likely to act forbearantly, telling themselves that they will get even with the offender in another way at another time, even though such an occasion may never arise and might not be exploited if it did. If the offense is great, the offended persons may withdraw from the encounter, or from future similar ones, allowing their with-drawal to be reinforced by the awe they may feel toward someone who breaks the ritual code. Or they may have the offender withdrawn, so that no further communication can occur. But since the offender can salvage a good deal of face from such operations, withdrawal is often not so much an informal punishment for an offense as it is merely a means of terminating it. Perhaps the main principle of the ritual order is not justice but face, and what any offender receives is not what he deserves but what will sustain for the moment the line to which he has committed himself, and through this the line to which he has committed the interaction.
Goffman, Erving. Interaction Ritual (pp. 43–44). Taylor and Francis. Kindle Edition.