Leaving a faction is actually a bit trickier than joining one. To leave a faction one must "offend" the faction leader in some way.

This is accomplished by the following:
Attacking a member of the same factions (**Warning: striking another may lead to you being labeled an outlaw**);
Attacking an NPC soldier of the same faction;
Initiating a loyalty quest and not completing it;
Allowing your loyalty to time out (just ignore the messages, they'll go away sooner or later.)

I'm not quite sure what would happen if you tried deliver a token to an opposing faction leader but I suspect you'd end up leaving the faction you were in...

Info by DogBreath