Many soil nutrients we add to our gardens can be very soluble and those will move easily out of your soil and into the ground water and become pollutants. You can have a too rich soil when excess quantities are added. Another problem occurs when an excess of one nutrient is present and it interfers with your plants ability to utilize others which then makes the plant more susceptible to insect pests and plant diseases.
This is one reason why having periodic good, reliable soil tests done is a good idea, to know what your nutrient balance is.

It is possible to overdo any thing

Listen to your plants not oldtimers are you having trouble or not getting the results you want? 20 yards delivered will compost down to 10-15 yards finished and that doesn't seem to excessive for a garden of your size

You can have too much nitrogen. But you'd have to work at that - like adding too much urea in one form or another.

The best vegetable and flower gardens I've ever grown were in a foot thick layer of pure, decomposing horse manure.

Have you had a good soil test?
I ask because after getting one I found the P levels were very high.
I was recommended not to add any more manure as it would keep pushing the P higher.

Balance, as long as you have balance, there should not be any problem.
The first organic garden was in a pig holding pen. I did not add anything to the rotten manure, I grew big red tomatoes.
The second was in a cow/horse feeding stall, again I added nothing to the rotten manure & straw.
I had beans,tomatoes, & peppers & no problem, the last tomatoes were picked the first week of November. The day before the first hard frost.
So balance is important, soil test is best place to start.

Once upon a time, like the 1960's, organic gardeners were taught to add large amounts of manure, rock dusts, and other organic matter to build up nutrient reserves in the soil. Since then research has shown that does not happen, that excess soluble nutrients simply flow with excess water into the ground water as pollutants. Once we were taught that very large amounts of organic matter, like 25 to 30 percent, would help hold these nutrients in the soil, but research has found that too much organic matter will also pollute the ground water.
Balance is what is needed, and periodic soil tests can help you get that balance. What you want is a good, healthy soil with balanced nutrients that will grow strong and healthy plants.