Personally, I think someone should only get an oathbreaker flag for being ejected from a religion, a non-removable mark of the character's betrayal to the Gods.
Leaving a Cabal / Tribunal on good terms should carry no oathbreaker flag, and being ejected from a cabal/tribunal could come with a potentially removable flag but not as serious as being ejected from a religion.
And just for fun, if one is ejected from a religion and marked oathbreaker, they can type 'thuban' into the command line to automatically join the Thuban religion