I don't think it's ever a bad idea to add worms.

It can take some time for evidence of soil life to appear in a raised bed, especially when soil is imported from outside. When purchasing soil from someone I think the buyer needs to ask for a 1 cup sample of that soil and test it with this simple test, before paying for the soil.
1) Structure. From that soil sample put enough of the rest to make a 4 inch level in a clear 1 quart jar, with a tight fitting lid. Fill that jar with water and replace the lid, tightly. Shake the jar vigorously and then let it stand for 24 hours. Your soil will settle out according to soil particle size and weight. A good loam will have about 1-3/4 inch (about 45%) of sand on the bottom. about 1 inch (about 25%) of silt next, about 1 inch (25%) of clay above that, and about 1/4 inch (about 5%) of organic matter on the top.
That will help you know what you are buying better then what the salesperson tells you. The term "Caveat Emptor" (Let the Buyer Beware) applies here especially.

That soil is NOT sterile. It just doesn't have any critters you can see.

Start planting stuff and they will come.

You don't need to buy anything. Worms will come after you start your composting program. I used bagged browns sheared wood and Starbucks coffee to speed up my compost. I can harvest it every two weeks. If you want to buy some bagged soil conditioners you can add that straight to the soil. I mean like those large bags of things like Kellogg's Amend. Read the labels before buying. You want mostly organic matter in those bags. You can dig those in right now to jump start the prep of your bed. All of those fancy soil things and compost starters are a waste of money. Yes, they won't hurt your garden probably, but they are not needed. There is no magic thing you can buy like humus that takes the place of organic matter dug into your soil. It is hard work, so people like to feel that they are making progress when they buy these things. It's like anti aging creams. They make women feel good so they buy them, but the don't stop or remove facial aging. But, there are people on this forum trying to sell these things and posting ads for websites.

Planting "stuff" is not what will attract life to your soil, it is how much organic matter is in that soil when you plant that will atract them.

There is more soil life around live plant roots than where there are no live roots. Live plant roots are part of the soil life system.

The mushroom compost you added is full of microbes. Actually soil has a frightening number of them, even apparently dead soil.

Worms will come if the conditions are right - moisture and organic matter for them to eat. If conditions are not right, any worms you put there will leave and go somewhere else.

If it seems like you need more compost, add a few more bags. It really can't hurt anything. I like to use different kinds for variety but it's not critical.

Use some organic mulch after you plant, (untreated) grass clippings, wood chips, etc. to feed the soil slowly and draw the worms.

Hmm, I wonder if compost can be used as a facial anti-aging treatment. I might try it out on my dear wife if I can catch her in the right mood. Such as 'asleep.' Bwahahahaha!

What kimmsr calls structure above is known correctly in soil science as texture

The texture is of the soil in the garden; structure is what the results of the test are.
If the structure is incomplete the soil is lacking a building block that helps explain why the texture of the garden is as it is.

Well no

For those who may be confused.....a link to a previous GW thread discussing soil structure and texture. Seems like this is a topic with specific definitions which require revisiting with routine regularity :-)

Here is a link that might be useful: structure versus texture

Thanks so much for all the help!