Yes, the fruit falls off if the female blossom wasn't pollinated. When have you been hand pollinating? And how?

Rodney

I've been pollinating in the morning. I peel off the petals from the male & rub the stamen in the female. I don't tie the flower shut I just want it to set fruit. I have to wait some days to get both male & female flowers that open that day.

soundslike you're doing the right thing - early morning is best, the pollen is most alive then