You are making choices that have consequences. Unfortunately, it sounds like the choices YOU made were in YOUR self interests and the negative consequences will be felt by others. If you want to move, ethically, you SHOULD plan to give up your 50:50. That said, US family courts are generally strongly biased in favor of women and against men, so legally, you stand a good chance of using that bias to your advantage and taking your son away from his dad. If that's the kind of person you are and what you want, go for it - the courts will likely help you take him away. Statistically speaking, your odds are about 85% you'll prevail and get to take your son.