Looks to me like you did leave some collar, especially on the bottom-half of the cut. Might heal slower than it should, and maybe not completely, but I don't think that's going to be any big deal.

The tree will be fine...but is that poison ivy under the tree looks like 3 leaves and a red stem...?

That looks like wild strawberry to me.

Strawberries, random weeds and some regular ivy. No poison ivy. I know I need to weed the area and put some mulch down but I've got about 1000 projects going on with this new house.

What's the worst that's going to happen with this tree? I feel like the picture makes it look like there is more collar than there is. Is it not as big of a deal because the tree is only a few inches across?

All it affects is the time it takes to completely grow over the cut area. It's not serious IMO. We've all made that mistake including me I'm sure. I've never lost a tree to a cut like that.