I think it depends. If he is flexible with you more often than he is not then I would do it. If he has plans for the kids that they will enjoy then I would probably do it but if he is one of those asshats who just wants to show off that he has his kid on Fathers' Day and expect them to celebrate him then I would say no.