Just remove the fruit if it sets too much. My first year trees almost never hold fruit much past bloom. I almost always leave an appropriate size crop on 2nd year and older trees. A modest crop doesn't stunt the trees especially in long growing season areas. If the tree is stunted it's usually because of lack of water, weed competition, or another lees than optimal growth factor.