Best fit I thought is Elaeagnus oldhamii (which incidentally has a nickname "taiwan sweet olive") - there may be better pics for you to google to compare with but there are a lot of matches from your description - see the link below and also this one(you may need to cut and paste)

http://www.gumtree.com.au/s-ad/park-ridge/plants-pots/jac-rare-fruit-taiwan-sweet-olive-plant-edible-fragrant/1051769109

Here is a link that might be useful: picture

Agree that it is an Elaeagnus, but not sure of the species. E. multiflora or -- does the shrub have thorns? -- possibly E. pungens.

None of these are native and can be very invasive... the related autumn olive & Russian olive are both very problematic in the Midwest.

Here is a link that might be useful: Elaeagnus

Autumn olive is also problematic in this part of the Southeast.

Thanks Ladies, (if missingtheobvious is also female).

I now am pretty sure it must be some kind of Elaeagnus, but still am not clear which one. It is none of the ones suggested so far. E. oldhami blooms in winter and fruits in spring, so that's out. It doesn't have yellow flowers or thorns, so E. pungens is out. Closest is E. multiflora, but leaves and fruit of mine are longer than the descriptions I found. I can't imagine mine being considered invasive, as in 18 years have never found one single seedling or sucker, and believe me thousands if not millions of seed have dropped on the ground in that time. I also have never seen another tree like this anywhere, none in my town, (which you would think I would if it was so invasive?)

I am thrilled to read that it is 7 to 17 times higher in lycopene than tomatoes, and that there is current research on its high anti-oxidant properties and possibly cancer preventing properties.

I try to be a responsible gardener and not grow invasive species. Have to say I do recommend this if you can find it in your neck of the woods. Unbelievable fragrance in spring, and I'm gonna try and make some jelly this year. Have never seen any pest or problem, have never watered it, and only job required is cutting back branches that extend over the walkway. Makes an excellent screen from my obnoxious neighbours (yeah...bonus...LOL)