If he purchased the house during the marriage, you were legally entitled to roughly half of his ownership when the divorce was finalized. If the house's property division was not specified in the divorce agreement, it might be possible to reopen it but it will probably cost money and time. Any verbal agreement is very likely going to evaporate in the process.
It is never a good idea to rely on verbal agreements when it comes to finances in a divorce. Exchanging money through a third party provides legal evidence of payment and benefits both of you.

Glenda: Alimony needs to be in writing. Verbal agreements are not enforceable when it comes to the law. I received alimony for several years. I needed an Attorney to get it and it went before a Judge for approval.