I was told that a divorce shouldn't stop me from working. Just get a reliable babysitter (family member, if possible) for when you are working nights. Are you divorced yet? Because in your divorce agreement it should state that you are fully responsible for the kids when they are with you, that would include you paying for a babysitter if you have to be out periodically. I wouldn't say he has a case against you at all on this but I am not a lawyer.
You say he is skinny flint broke? So would he hire a lawyer to fight you on this issue? If he's that broke, I doubt it. Just do what's best for the kids, which is likely to have one parent working and providing a safe home. You can't go wrong if you think and act for the children first before anything else.
Angie, you don't need your ex bothering you about your work schedule. Put him in his place. File a Restraining Order. Just go to your police station. They will advise you. Just say the truth. How you think he may hurt you. How he looks at you in a menacing manner. You feel threatened by him. Keep it general and nothing that can be disproven, i.e., he punched me in the face. The restraining order is almost automatic and will ruin his life. He will be consumed by it and will leave you alone.