In addition to what Shelly said, if you in any way feel threatened or scared by his demands to enter YOUR home then I strongly suggest looking into a restraining order (or whatever they're called these days). See what the courts say when he tries for primary custody with one of those attached to his name!
However, this isn't something I would tell him about. Right now you could consult your attorney and ask how the process works, what the paperwork asks for, etc. I think it's a tool you may need in your toolbox for dealing with him and his threats.