Why would you think that you have the right to allow or deny him access at all? Would you be ok with someone treating you that way?

You can't and shouldn't gatekeeper the kids away from their own father. The relationship he shares with them is his and their right...just like the relationship they share with you.

Of course the children should see their father as much as they're seeing you. Especially if you care about their emotional well-being.

Based on your post, its not clear that there's anything at all wrong with your ex. Maybe he's upset that you're controlling his ability to see his own children.