Do you have a male plant to pollinate the female, or is your plant bisexual? Proper pollination has always been the challenge for me with papaya, as most are not self-fertile, and you can't tell the sex of a plant until it flowers. But if you're getting fruit of any size at all, this may not be the problem anyway.

Good luck...