You know..I do agree that he deserves to be happy. We all do. What I want to know is how you can leave your wife and kids, regardless if he has someone else..and be happy. How can you be happy with what you did...how you left things with your woman and children? I dont think you can. Somehow I think that deep inside, many of these guys do have some measure of a conscience (they can't all have absolutely NO remorse for the damage they've caused), and that when they are alone and look in the mirror, they can see what they really are.
As far as the town people, they have no idea. Let them believe what they want. Dont waste your precious time and energy worrying about people that dont matter anyway.