My local horticulturist says these are "Finger Galls". The larvae of a mite that has laid eggs into the leaves. As the larvae grow it causes the leaves to grow plant material around them. Apparently too late to do anything about it but at least it should not harm the tree too much.

I would at least wipe them off before they start to eat your leaves.

Why is it that all the bugs like plum trees? Of all the trees in my yard the plum tree is the one with the most damage/pests. I guess bugs have good taste.