If the drain does what it's supposed to, this area will not be mucky after you're done. If you're growing grass after you're all done, you want a good topsoil that can support it. Normally I would say mix in some compost with the soil you have and use it. However, the clay 'fill' you mentioned, if it's subsoil, may be not only poor in organic matter and nutrients but may not drain well depending on what kind of clay it is and whether it has much silt in it or is pure heavy clay. You may want to get a topsoil that drains better, just make sure when you pick it out that it's not heavy clay too. Depending on the total thickness of soil, you might be able to use some of the clay and buy less topsoil, but you may still have the permeability problem so that may not work.

Definitely put down a fabric layer on top of the stone to keep the soil from filling in all the spaces between the stones later on.

Thanks for your response. The clay (dirt) pile I have was originally dug out of the ditch several years ago prior to me buying the property. The top layer of this pile was covered by vegetation for at least 5 years and some of the soil was probably muck. The soil does break up somewhat easy and I think I will see if it mixes with topsoil. I'm not sure if my compost really qualifies as good compost, but I would like to use it up anyways. If it doesn't work well enough, I guess I can always tap in several surface drains to help remove the water along this pipe. Yes, I will use fabric prior to covering the stone. I also wrapped the pipe and stone around it with fabric prior to putting the final stone cover on. Thanks for your input. It has helped me come up with an acceptable plan.

Great, hope it works out for you! Water problems can be a hassle.