Absolutely not. He has no right to enter your home without your permission. If he doesn't think it's fit for your child then he can go to the CSEA and tell them and if they find it necessary, they will do a child safety check. If they find everything in order then it'll be dropped and there's no way he'll get primary residency because you won't let him in your home.
In addition to what Shelly said, if you in any way feel threatened or scared by his demands to enter YOUR home then I strongly suggest looking into a restraining order (or whatever they're called these days). See what the courts say when he tries for primary custody with one of those attached to his name!
However, this isn't something I would tell him about. Right now you could consult your attorney and ask how the process works, what the paperwork asks for, etc. I think it's a tool you may need in your toolbox for dealing with him and his threats.