cabbage aphids. Witha small number of plants you might try washing them of with a hose or washing with soap and water.
Here is how commercial growers in the west do it. (UCAL Davis)

Here is a link that might be useful: Aphids on broccoli

I tried the spraying off with a hose and they were back the next day. I only have four plants and they are starting to head.
How do you apply soap and water to kill them?
Also I am assuming these little buggers are frost tolerant since we have been in the 27 degree range the past few nights and it hasn't fazed them.

I have the same bugs on my Brussels sprout and napa cabbage leaves.

They are very nasty!

You wrote: "they were back the next day."

Nope, not back.

Instead, you missed a few and they produced more! Squish or apply another harsh spray. repeat as needed.

Instead of using hose sprayer, I use the garden pump sprayer. Then can also add your favorite recipe for them aphids. This way I can do it surgically with less amount of water and more effectively. This past season I had major aphid problem on just my eggplants. I used Sevin dust. NO GOOD. Spray was better.

With only 4 plants you could easily take a bucket of soapy water to them and wash the leaves by slooshing them in the bucket and cleaning the leaves with your hands or a sponge. There's really no need for sprayers or chemicals on 4 plants.