What I'd like to know is what layers of material should I put down?

Should I use pea gravel for the base layer? Or is some cheap, crushed gravel fine?

Do I need sand on top of that gravel?

Just curious what the best combination of layers would be. For what it's worth, I'm in the Denver area.

1)excavate down paver thickness plus +- 7" (for 6" gravel base)
2)apply geo textile fabric
3)add 6" of 0-3/4" 'cheap crushed' gravel base (4" is acceptable for a patio, 6" is better especially in Denver climate) seek recycled crushed gravel if available.
4)tamp gravel base with a vibrating plate over two layers (at 3" and 6")
5)top up low spots to make level grade (sloped slightly for drainage in appropriate direction) tamp.
6)apply 1" layer of sand or limestone screening. sand is preferable for flagstone as it shapes to irregular underside of flagstones and drains slightly better.
7)level off perfect grade playing with a 4' carpenters level or leveled pipes and 2 by 4. slope 1" over 4 ft in appropriate direction for drainage
8)apply pavers (tightly) or flagstone
9)sweep in polymer sand to fill in joints
10)tamp, reapply polymer to top up joints, moisten

Alternative bases for flagstone-
1)geo textile fabric
2)4" crushed gravel (0-3/4") base
3)4"- 6" Concrete pad with rebar
4)apply thin set mortar with 1/8" notched trowel
5)place flagstone
6)apply mortar to joints, have sponge handy

Least thorough option for thick flagstones (2" +)
1)excavate slightly more than 3"
2)apply 1" sand for levelling
3)drop flagstones in place, hammer down with deadblow mallet
4)fill joints with polymer sand, moisten

Hope that helps.

What's the benefit of the 6" of gravel? Everything I've read has suggested 2".

More importantly however is the drainage issue. In Denver obviously you are susceptible to freeze and thaw cycles. A gravel base is permeable allowing water to percolate down and thus reducing the heaving damage by taking it further from the surface where it meets with clay and/or silt. The deeper the better as you want to keep water collection as far down from the surface as possible. It also acts as an insulator and reduces the freeze susceptibility to a deeper level.

Lastly, a gravel base is flexible. To a degree, it can absorb some of the forces nature exerts on your sub grade. The net result is less damage to your surface level.

6" is definitely on the safe side of things, but thats the route I offer for my clients in Zone 4.

Most important however on top of everything is to slope it off so that there is less standing water in the first place. 1" slope over 4' should do ya fine.

enjoy!