According to Ye Olde Wikipedia Rosemary may have a noticeable effect on the brain, improving memory but slowing recall. Epileptics should however avoid it.
If you are talking about contraindications for rosemary, add pregnancy to the list. Included below is a link for a whole list of herbs contraindicated during pregnancy.