Let's consider two issues in turn: what kind of soil, and then how much.

The quality of your produce is directly related to the quality of the dirt they grow out of. This makes sense, because plants consist of energy from the sun, oxygen and carbon from the atmosphere, and matter from the soil in which they live. Therefore, it is worth your while to make sure that your beds are full of good dirt. You want to pay attention to a few factors in garden soil, mainly nutrient content and drainage. Once you know what you want, you can buy cheap ingredients and mix them to get what you desire.

Base: good ol' dirt
A good base for a garden soil mix is the cheap topsoil that you can get at your garden center. Lots of stores sell topsoil for cheap in the spring, like maybe $2 or less for a 40-pound bag. That seems like a lot of dirt, but it's really a negligible amount once you start filling in your bed.

Nutrients: compost & peat moss
You also want to have a lot of nutrients, so it is a good idea to add in some compost. It's great if you're making it, if not they sell it at garden stores. It's incredibly rich, so it's okay to mix in only a bag or two per raised bed. Peat moss is also a good idea, as it is full of organic matter. It comes in big bags, and is really concentrated so you don't need all that much to have an effect, maybe half a bag for a medium-sized bed.

Drainage: sand/vermiculite & peat moss
You also need drainage, which refers to the ability of water to seep out of the topsoil. (This is what distinguishes a swamp or marsh from "good" land where you can grow things like tomatoes or corn.) This is aided by little holes in the soil. Since water seeks the lowest level, raised beds inherently have good drainage, but you can make the situation even better. It's a good idea to break up your substrate (the area under your bed) pretty well with a fork or spade before building your bed. You also want to make sure to mix in your added soil with some of your substrate, because an abrupt border can lead to problems with compaction: sometimes roots will never penetrate the substrate if it's not mixed, and drainage will be reduced. It's a good idea to mix some sand into your bed's soil to increase drainage; vermiculite (the white chunks in potting soil) also works, but then it looks too much like potting soil and not enough like a garden for my tastes. The light texture of peat moss can also aid quite a bit in creating better drainage, in addition to the nutrient benefits.

That said, it will take a LOT of dirt to fill your raised beds. I built some beds that are 5' wide, 10' long, and 1' high. Each one required a bag of peat moss and 1,600 pounds of topsoil! That's 40 bags of that cheap topsoil, which even at a low price came out to hundreds of dollars for a few beds. I highly recommend against using potting soil or your bill will easily stretch into the thousands of dollars.

The way to do it on a budget is to:

1) buy cheap materials
2) mix it yourself
3) use topsoil that is on sale, or maybe even free from a local landfill
4) you can even fill the bottom with straw: a straw bale is pretty cheap and gets you a lot of organic matter. It will biodegrade in a year or two if it's buried at the bottom of your bed, it will increase drainage, it will not hurt your plants, and it's a lot easier to move than dirt. In Scandinavia, they traditionally grow potatoes in straw without even any dirt, and they do fine (plus then they come out totally clean!)
5) don't use potting soil! unless it's old potting soil that you have tossed in your compost pile or something

I hope all that helps -- good luck building!

Also, it might be a good idea to line the bottom with chicken wire or hardware cloth if you have problems with varmints in your area (so they can't dig up into your bed), or to line the bottom with landscape fabric (to prevent grass etc. from growing up into your bed)