These people are becoming more unlikable every episode.
Rachel: "I know I haven't told you anything about why I left you, what I did while I was gone, why I am making goo goo eyes at your uncle and why I haven't had anything to do with you for the last fifteen years, but. . . (SLAP!) I'm your mother dammit!"
I guess their way of 'improving' Charlie is by making someone else annoying and unlikable.
Miles: "Hey buddy, I know the plan was to assassinate Monroe, but I couldn't kill my old boyfriend so I bailed. Sorry you took the fall. How about we try that again? I promise I'll stick around this time. Oh, by the way, the militia tracked me here so your wife is in danger and they are about to destroy this peaceful town you have managed to help create. Sorry about that also."
Add that to his atrocities from his militia days and you can see why 'his men' don't want any part of him or his plan.