I'd stay with them longer. They have stringy foliage at first, but yours do sound a bit lackluster. I started my original patch from seed and now they winter sow themselves, so to speak. They spread at a pretty decent clip and make a good-sized plant their first year.

Maybe sprinkle a little Osmacote into the pots? Does their soil have any nutrients in it or is it just a potting mix?

I think mine bloomed their first or second year

Patience is not a word used to describe me...

I fertilized earlier in the spring, but not recently. They are in potting mix. I'll go out there later and check the roots.

If the roots look decent, I would go ahead and plant them. I've had plants from seed that seemed to be unhappy in pots and they would finally get going once I got them on the ground.

IME, these are not a long lived plant. Three or four years seems about normal. They definitely prefer a limy soil. One of those plants that if you have the conditions they like, they are quite easy, but if you don't, forget about it.

Update 2018: I sowed some blue "Fama" last year for this house, and they did ok - much better than my first attempt in 2013. The flower is spectacular -- it is one of the most beautiful shades of blue I've ever seen. They were doing fine most of the summer until the deer got hold of them later in the year, but I noticed the other day they're starting to poke up through the soil, so they survived.

This year, I sowed more "Fama", both white and blue, and they came up nicely on the light cart in the house (cold stratified in the fridge for about a month first). I potted up today into 2" pots. The plants seem to be a little stronger than what I grew previously, but the roots are kind of stringy. So we'll see. Will set the pots outside in a few weeks when the weather is consistently warm. Looking forward to these blooming, they are glorious flowers and put the short "Butterfly Blue" cultivar to shame in both flower form and color as well as overall plant habit.

I only grow the small scabious s.columbaria...which does take a couple of years to really get going but will seed about. For a really penetrating late summer blue, I heartily recommend devil's bit scabious - succisa pratensis...which, along with jasione perennis and various knapweeds, form the backbone of my late summer meadow areas at the wood. All do take 2 years though...although you may get the odd stray bloom during the first year, they are mainly growing roots.

I found the s caucasica to have less overall vigour and reliability in succeeding years - really only a 3 year perennial which was also reluctant to seed about so when they vanished, I didn't replace them.