Three years ago I made a path using about 3" of crushed limestone. It was great, much better than the dirt I had before.

Each year I seem to get more and more self seeding plants in it. The crushed limestone is quite fine, and I think it just gives some seed a great bed for germinating.

Would a larger gravel not have this problem? How large should it be?

Limestone may hold moisture better than some less porous stone. That might not happen so much with a granite or quartz stone. I'm not sure that it would make a big difference, though.

What is done by the higher end landscape contractor's in my area is to make the base for the path out of a much more solidly compactable material such as a road base or reprocessed asphaltic gravel (ground up asphalt mixed with gravel, not a hotmix) that is then compacted. Then a minimal thickness of a surface stone is put over that for the aesthetic surface. The compaction inhibits growth, the thin surface inhibits the development of soil on top (and makes it easy to walk on).

It is not the best solution for all circumstances. A couple of negatives are that it is not a truly pervious walk and will have runoff and it is more difficult to clean up if you change your plans.