I'm guessing it's probably perlite.

Perlite or Vermiculite, minerals that are heated so they expand and then can absorb some moisture but also help keep your potting medium open for drainage.

Unless you have softened water it is most likely hard (dissolved minerals) which become salts and buildups in potted plants. They will appear as whitish yellow and mushy to the touch.
Perlite is white but will become this same yellowish color but not mushy, actually a bit hard.
House plants should be leached regularly, an easy method is to use dish soap and spray on the leaves, top and bottom, then sit in the shower and wash off and through the soil thoroughly. Allow to drain and dry then return to the window.

Or if the soil is kept too moist it could also be mold. I noticed that if I over water my houseplants...without allowing them to dry out between waterings...I tend to see white "chunks" on the surface of the soil. We have naturally soft water so I know it isn't salt deposits, and besides the white chunks disappear when I let the soil dry out. Just another thought...

"Grainy white rocks" is perlite. You may be seeing more because they float to the surface of soil. If you squeeze one between your fingers it should crumble.

Vermiculite is brown and rectangular, more spongy than grainy.