Plants will not mind the rocks if you are able to work with them. There is a possibliity herbisides were used before and/or after the area was converted. Make certain there is no weed-block or plastic underneath.

You could depending on what is underneath the gravel or pebbles. If there is any kind of weed barrier under them that will need to be removed.
Whether "topsoil" is really what you want will depend on what that "topsoil" is. The usual definition of "topsoil" is the top 4 to 6 inches of soil that usually contains some organic matter. The soil you want for any garden would be something with 92 to 95 percent mineral and 5 to 8 percent organic matter.