Since it is there, and it's a small area, I'd leave it and see what happens. Perhaps do not mix it in, and try a large-seeded crop like potato, sun choke, runner bean etc. if it works out poorly you could remove the sand later.

I've been adding sand to heavy clay soil for over 40 years, and it always improves plant growth. I grow some of the rarest and most difficult plants on Earth in clay soil amended with sand. It is best if the sand makes up at least 25% of the soil by volume. Don't listen to the hysteria; just try it yourself. The main problem is that you need to wait until the clay is workable: not too wet, not too dry.

But remember that there are many other variables that you need to worry about, such as adjusting soil acidity and providing fertilizer and water at the right times. Hopefully, there is plenty of sun in that area, or you aren't going to have much luck with veggies.

If you want to understand how sand interacts with clay and silt to make soil, Google "soil triangle". Loam and sandy loam soils are the best garden soils for general-purpose gardening. If you have clay loam, you can make loam or sandy loam by adding sand. If you have pure clay, you probably need to replace it with purchased soil. My rule of thumb: any soil which produces a good crop of weeds or grass can be amended to make good garden soil.

One of the best things about using sand is that the improvement in your soil is permanent, because sand does not decompose like compost. So you won't have to spend the rest of your life lugging compost and digging it in. On the other hand, it's always a good idea to add a little compost every year. Mulching with compost is the easiest way to do this, and helps to keep the soil cool and moist.