If the kids are spending more time at your place, then your ex should be paying some child support. Even if he is unemployed, the courts will attribute some income to him and figure out child support payment. I guess you never went to court over support payments?
He won't be paying any child support, if the children are with him all the time...and then you would be paying him child support.
You know..I do agree that he deserves to be happy. We all do. What I want to know is how you can leave your wife and kids, regardless if he has someone else..and be happy. How can you be happy with what you did...how you left things with your woman and children? I dont think you can. Somehow I think that deep inside, many of these guys do have some measure of a conscience (they can't all have absolutely NO remorse for the damage they've caused), and that when they are alone and look in the mirror, they can see what they really are.
As far as the town people, they have no idea. Let them believe what they want. Dont waste your precious time and energy worrying about people that dont matter anyway.