YES! You have every right to fight for you child's safety I fought and my kid's are not allowed around ex's gf alone for more than a few hours. I may not seem like much but it is enough that I have the control there. You have the right take him to court and make it so that he take a drug test on demand. they have home kits. If my ex even remotely smells like beer I have the right to stop visits. But you have to do this the right way, go through the courts. If you take matter into your own hands you will lose.