Its very normal and healthy for children to identify and pine after the same sex parent. Its also normal for kids to want both parents in their life. Nothing unusual about it and, in doing it, he's not lessening your importance to his life at all...and you shouldn't take it that way.
Its really about what he needs...not about what you need or your ego.
My advice...find a good male role model for him if his father refuses to participate in his life. If you have someone in your family that would spend regular time with him, that would be great. A brother, a dad, an uncle....someone you trust that can show him what being a boy is all about.
If you don't have a family member, they have some really great mentoring programs in most places that can help single moms and their kids.
Your son is expressing a very normal need...instead of being threatened by it, be happy that your kid is a normal, healthy kid who's clearly telling you what he needs. I'd be a lot more worried about a kid that wasn't asking where the other parent was...that's abnormal.
singlemommy: Sorry this happened. I would just tell him he wasn't ready to be a father and it's best you and he (your son and you) just make a life for yourselves.