Sounds like you are playing power games.
If the kids are safe in his care, then for Heaven’s sake let them enjoy a whole Father’s Day with their dad. Your breakfast can happen another day. How would you feel if it was Mother’s Day and he scuppered your plans to do something fun with your kids fir them day?

Grow up...

you give him the whole father's day with his kids. you need to be flexible with him and then he'll be flexible with you when something is important to you.

thats the problem i have been nothing but flexible every time he asks but when i ask he says no. This is every time. im at my whits end.

I think it depends. If he is flexible with you more often than he is not then I would do it. If he has plans for the kids that they will enjoy then I would probably do it but if he is one of those asshats who just wants to show off that he has his kid on Fathers' Day and expect them to celebrate him then I would say no.