I found 1 black worm in my upper pond a week ago clinging to a rock. It was about an inch long.

I did some searching and came to the conclusion it was some kind of fly larvae. I probably have 100's more but I didnt search for them. They are harmless to the pond and your fish would probably like to dine on them if you can swish them to where your fish can get to them.

They are fly larvae and are as Pond Dragon pointed out, are harmless to your fish.
The larvae can be black or red in colour depending on the species of fly they are going to be i.e. Black Fly or Midge, etc,.
The worms are said to be caviar to the fish.
"Horton"

Yep...just pulled pennywort out of my falls and gobs of the black fly larvae went into the pond. The fish are going nuts!