Feeder goldfish are not kept to any decent standard so it is not uncommon to loose some. One batch of 16 I got once I lost 1 the first night and 1-2 over the next month. I still have the rest of them.

The fish in there are probably just pretty scared as you think. It is a new place for them and they don't know what the threats are.

I give mine some hiding places under some big slate slabs I have. Once the water lettuce gets into gear and covers most of the surface the fish are happiest. They tend to stay under cover but will come out for food as "safety" is just a swish of the tail away.

Do keep an eye on the water quality (ammonia, nitrite) as you still need to grow the bacteria that handles the fish waste. Water changes are the best way to handle that, as are squeezings from the filters of healthy established ponds or aquariums. That gets seed bacteria into your pond.

Fish cluster together and hide at the bottom when they don't feel secure.

They need plants or something to hide under or you'll be serving them up to herons or other fish eating birds.

You say you used filter floss and I wanted to make sure you weren't using the short trimmings. You may have to trim the batting sheets or fold into layers but the fibers don't get wrapped around the pump impeller.