He is playing games with you and controlling you. You will have to make a decision on if you want to live this way. Sadly, he doesn't love you. People who love you don't treat you with such disrespect,
Make an appointment with an attorney (first visit usually free) and see where you stand.
You will live in limbo, if you continue with him. He wants his cake and to eat it too. He wants to be free to be with another woman (women) and keep you on the side.