Also, the process of improving soil by adding organic matter will usually lead to a more fluffy texture, so the soil will increase in volume over time.

I'm not sure exactly what you mean about it being 'ground level', but if you can handle an inch or two of 'fluff', you should be OK. I don't think you'd have to remove a foot.

You would need to acquire about 300 cubic yards of organic matter to get a 12" layer to till into your soil. That's lots of dumptruck loads of it. Removing soil to avoid raising the soil level would require an equal amount of truckloads.

Raised planting beds are the easiest and most effective way to deal with difficult situations like this. You get a higher yield for less money and less work.

Make shallow planting boxes and dig about 6 inches of organic material (compost, bark fines, whatever you can get) into the top 6-12 inches of dirt.

Optimum bed width is 4-5 feet, so you can reach the middle from the pathway. Leave enough space between beds to maneuver a wheelbarrow. On the bright side, your paths are already graveled and ready to use.

This will confine the organic matter where you need it the most, where you will be planting.

Do a couple of beds a year, adding more organic material from your compost heap as you get it.

Plant deep-rooted plants, such as okra and squash, and when they die back, cut them off at ground level, leaving the root mass in the ground. It puts organic matter several feet into the ground as the roots decay.

===========
Use straw bales - straw bale gardening - and keep digging the decomposed bales into the dirt under the bed areas, along with any compost you have made.

=======
Or do a combination ... start with the straw bales and at the end of the season, use the bales in a raised bed as mulch.

My garden is in the ground, but I hoe up raised beds that I don't walk on, and they end up higher than the pathways. They stay a couple inches higher than the pathways, because I layer manure and hay on them every year, and also because I walk on the pathways which lowers them by compacting them. All of it works fine. (There are no built raised beds -- just some of the dirt that I mounded up a bit into beds about 3 - 4 feet wide). Would this work for you? Or just skip the mounding and plant.

Unless you are growing those plants that specialized to thrive in gravel beds, you need to make the soil more hospitable to them.

We used class 5, 2 and sand.

What size is yours?

We also used to unroll sod rolls on a granite drive-way to preserve them. They would root themselves in a matter of days.

If you are going to do an area that large, the entire area, get some one in their with a plow and plough it up, then put a few truck loads of dirt on it and plough it again.
I guarantee you, while it may not be the most pleasant soil to work with it will grow some items well.

You could also get some one in their with a skid-steer pile it up and sell to landscapers.
It is a thousand percent better for paving base than pit gravel.

Although I haven't worked much with decomposed granite myself, I think that it will support many types of plants as it is. But those plants that need a different soil can be nurtured with imported clay soils.

In Alaska last year, I helped make some garden beds built up on a rock and gravel fill. The location was a beach that had been built up with a boulder retaining wall and backfilled. The 'soil' was very 'young', full of stuff like decomposed granite.

After making the raised beds (we used a chainsaw mill to make the boards from yellow cedar), we filled them with whatever soil we could scratch up. We screened the soil to remove the stones and used the stones between the beds for walking paths. Hard work.

Some vegetables did ok in that raw soil, and cold wet climate; lettuce and radish for example. We used kelp, sawdust, and humus from the forest as mulch and soil amendments. Very different than the clay back here in Boise.

I wonder what climate the OP lives in.