States have different options for custody sharing. It sounds as if you have a good case for primary custody, which in some states requires that the other parent provide child support.
Whether or not he comes through with that financial assistance is a separate issue. But he shouldn't be barred from seeing his children.