Stink horn! AKA Devil's Dipstick. And yes, they look remarkably phallic.

It's growing on decaying organic material (which is a good thing) and considered harmless, although eating them is not recommended. While they tend to grow in damp conditions, I've seen them manage on far less water than I would have imagined they could.

If they bother you, kick them over. The mushroom will desiccate in a few hours of sunlight and disappear quickly.

Here is a link that might be useful: Orange stinkhorn mushroom

I have thoar in one of my flower beds and I hate them. Even if I don't see them, I walk by and can smell it. I dig them out with a shovel because I will not touch it with my hand. They always come back after a damp and/or overcast couple of days.