Did you switch it round for your son so he could see that if he lived with his dad he would have the same problem only with you? Im pretty certain your children wouldnt be able to cope being seperated from you.
Children quickly adjust to new set ups. dad is trying to undermine everything and cause waves, dangling carrots.
My son wanted to live with dad at one point, because they had so much fun. at one point I got so fed up I said 'ok'. I called both of their bluffs. My ex who had been going on and on and on about how his son should live with him almost had heart attack when I rang and said. Quite effective really, its one issue i dont have to deal with anymore. And of course the more time my son spent with dad the less attractive it all became.
What do you think your ex would do if you called him on it and said maybe equal parenting might be a good idea? If hes simply trying to freak you out the actual intention might not exist. Do you think he would really go for it or does he expect you to fight him tooth and nail but uses it as a stick to beat you with via your children.....

Considering he has never been alone with our children for more than an hour...I think he would tell me that he would take them, go through all the motions, bail, then turn it into something that is my fault
He is playing the good guy with my oldest, my youngest and him have NEVER had a relationship. He doesn't like him because he isn't into sports. Crazy but that is how he is. When he talks about the boys its always "John this, John that and never anything about Joe" I think he really thought that our oldest would just turn on me.
But as my Mommy told me yesterday when I just wanted to give up the fight. " Your all that little boys has that is stable, don't ever give up just because 'ex' is being a jerk. he will need you and you have to be there stronger than ever" I love my Mommy she always know what to say to me.