It may be Elaeagnus.

Autumn Olive. Your right, it is invasive
Check this article out.
I must say though, that there are very few plants that smell better than this one. The fruit is edible and wildlife loves it

http://www.nature.org/ourinitiatives/regions/northamerica/unitedstates/indiana/journeywithnature/autumn-olive.xml

Here is a link that might be useful: Autumn Olive

Thank you! Just looked it up, that looks like it. I have evergreen Elaeagnus (E. Ebbingei) and didn't even think of it being an Elaeagnus, didn't know there were any deciduous ones. Just read it came into the US in 1830 so I guess it's here to stay. It was just a little scary that I haven't noticed it before this year. I see that it's classified as a highly invasive species on Virginia's invasive species list. I read that it doesn't take full shade, so at least the deep woods understory is safe.

Yes I did find the flowers to have a pretty fragrance.

Thanks again!