Hi Cammy.
It sounds like he is trying to tell you what the custody arrangements will be and he doesn't have the right to do that. If there was a set visitation that both of you agreed upon that schedule should stay in place and he is responsible for childcare while he is at work.