Are you using the pavers as a walkway? It just feels odd to constrict yourself against a fence. Using a similar idea:

{{gwi:7758}}

I would add some green and height around the entire border, then have your pavers there (blue), and either add more green or a table to the center.

Hi there thanks for the feedback. The pavers along the border wouldn't be for a walk way - on the left side I plan on putting more of those planter boxes (like the two you see on the left side) to expand my veggie garden. I am going to put some on the right side also but it gets heavier shade so there's only so many shade-loving veggies I can plant there. I hope to build a chicken coop and put it under the shady portion of the right-side also.

Whatever I plant in the back corners I hope to give me some height and visual impact. One of the things I worry about is if you look behind the fence in the back there is a 4-5' drop off to the neighbors yard and it's supported by a concrete retaining wall. Will any trees I plant back there weaken this retaining wall? I worry about the root system pushing it out and costing a lot of money in repairs.