Depends a bit - does he complain when you take him? Does the cost put a strain on you finances? Does he have friends on the team? If the answer is no to any of those, then I'd say try another sport and save yourself the time/trouble/money.
If he is having fun and enjoys it, then don't worry about it. Just make sure he is on the appropriate level team and don't chase titles or leagues. My oldest kid was super soft at age 10-11 (good skater but no hands/shot or aggressiveness), then decided one day he didn't like being average, worked his ass off, grew a bunch, and is now an absolute beast in HS. Other kids dominate peewee only to average out and not even play high school. Just enjoy the ride and time you get to spend together.
How low level are you at right now? If he is already in town hockey then who cares how soft he is if he is having fun. If you have to fight with him to go to every practice or game then pull him. You never know when the light switch turns on and he wants to play at a higher level.