I think you should follow the other posters advice and see what kind of help you can get. At the same time, I agree with the first reply, you should ask your kids. They are not so young they don't understand. Just tell them you are concerned and to let you know. I ask mine if they feel comfortable going with stbx, I always tell them if there is a situation they don't feel comfortable getting in a car with someone (a friend or their dad) to call me, I will always rather drop what I am doing and get them than have them get in a car with an unsafe person!! Do your kids know ex has drug history? Mine are 10 and 16 and they know. I hid it for a long time then I realized I am protecting HIM not THEM and they need to know. I decided they needed to know so they understand his behavior is not because of them, it's not that they are unloveable (which they know anyway because I love them to death) but because of him and his issues. Your kids safety is #1 so finding out your rights and keeping your kids informed should help. I'll pray for all of you to be protected. I've said it once and I'll say it again, something is WRONG with their brains. So be CAREFUL, when someone cannot empathize or feel remorse...you definitely want to take precautions.
the moment I read that you are afraid he may harm your children to get back at you had my alarms ringing because in most cases when children are hurt, the mother had usually voiced her concerns yet no one had taken them seriously. As his ex partner you know him better than most and if you seriously believe he could harm your children as an act of vengeance (the usual reason in most cases) then you MUST start making lots of noise, lots and lots until you are heard.
If it is in dispute are you able to have the courts test him?