I would be interested in what other have to say about this. Like you the rest of the garden flourishes but I always get small heads on my broccoli. I have tried different areas of the garden, etc but always the same results. I always have enough to put up in the freezer, but can't get really big heads out of the plant.

Well, I have gotten some huge and very tender heads.
I live in East Central Indiana. I grow both spring broccoli and fall broccoli and fall cauliflower. One recently picked head of cauliflower may have been 10 inches across.

Timing of seed sowing is critical to good success. I sow spring broccoli about the 17th of March inside and take them outside as much as possible. Then I set them out in the garden after about 26 days. If it gets extra cold a night or two before the roots are established in the soil, cover them overnight. They only rarely need that..
I sow cauliflower seeds in plastic pots out on the picnic table about June 16th...and perhaps 3 or 4 broccoli. On about July 5th I sow more broccoli and perhaps a few more about the 12th. I set these out in the garden after about 26 days while they are smallish [important] and water well if needed for as long as needed [very important].

I sow Emerald Crown and Imperial broccoli [Stokes Seeds] and Symphony cauliflower [Fedco] and perhaps a couple early Snow Crowns.

Broccoli and cauliflower are sweetish soil lovers. I don't have worm trouble in the spring, but have to keep them off in the fall.

Dan, it would help to know what zone you are in and what the weather's like. That really has a big influence on your crop.

Im in wanganui new zealand, sorry i dont know how that works out with zones, very temperate climate, i have a full garden all year round, as i said i cant find any pattern, it doesnt matter which time of year it is or how hot/cold its brocolli roulette if ill get a crop or not.

Broccoli needs to mature in cool and moist weather for nice big tender heads. Here October is best.

Here you go -- Growing broccoli in New Zealand

Here is a link that might be useful: Growing broccoli in New Zealand