I would take advantage of the situation and keep a few sprouts that are spaced well enough apart and graft on to them. There is no need to have an apple tree for just ornamental value.

There is a lot of information available on grafting and there are several resources for scion wood. The mature root structure will give you a faster producing tree than if you replaced them with good apple trees, and you would have the benefit of good apples as well.

I must say that I personally have not had a lot of luck grafting, but it is a learning experience. It won't hurt anything for you to try. My most obvious mistake was trying to graft on to rootstocks that barely had any roots. This would not be the case in your situation. My best luck was in situations just like yours.

I can't comment on the removal of the stumps. A picture might help. I'm really just a newbie myself so lets see what others come up with.