You're right that many kinds of lilies can be toxic... If enough is ingested. Unless you're planning to eat that soil there is absolutely nothing to be concerned about. In fact the flowers of Hemerocalis sp. (a.k.a. daylilies) are edible and with a mild unobtrusive flavor. You could actually plant vegetables immediately adjacent to Agapanthus or Hemerocalis without any fear whatsoever it's not something they exude which may then be taken up by other plants.

Besides that part I'm going to suggest a thorough amending of the spot before planting your veggies. Agapanthus and daylilies are famed for their ability to grow in horrible soil which is generally not so for vegetables.