I think it will be fine - in moderation and no or very little jumping.
In a pasture set up, young horses will jump anything they perceive as jumpable. But when a horse is worked, he is not given a choice and/or could already be tired. That's where it can be damaging.
I was going to say you could do different heights and/or jump exercises, but I recommend that you wait on that. Start your young horse on ground driving instead... Or just go for long walks that incorporate natural obstacles - hills - low/mild and steep, ditches, rocks, limbs/logs, puddles, streams, ponds, rivers etc...