What type of caterpillars are these? They are eating my broccoli's leaves. Aside from the obvious holes in the leaves are they causing harm to the broccoli itself?
Cross-striped cabbage worm. The leaves supply nutrients to the plant, so if they are decimated, it stands to reason that you will get less production out of it.
I would hand pick and kill them.