I think if you took a razor blade window scraper and just broke the seal of the silicon caulk you could peel it off of the vinyl fence with just a little prying. I have used that technique to break the seal and take apart other pieces that I had glued together.

The caulk would be hard to remove I would bet, but you could try. I wonder if those 3M hooks with the removable sticky stuff would work? Might be worth a try.