They're rather non-toxic...even to most animals.

Lillies (which many are toxic) are not in the same family as cannas.

Also, toddlers are not usually allowed to forage at will. Plus, we are taught to never eat anything in the yard without parent's permission. I grew up surrounded by Yew shrubs with those toxic 'berries ' yet was trained to be afraid to even touch them.

You can raise children to be safe in the garden. You'd be horrified if you knew how many toxic plants grow in the typical landscape.