Yellow leaves often means too much water. Without a photo, it's hard to say for sure, but that is my first guess.

Peppers have their share of pests...even the Tomato Hornworm will feast on it's fruit.

Holes can be a number of different beetle or caterpillar types.

In my experience - unless the plants are be severely compromised - I leave them alone. Except if I see hornworm evidence (frass on the leaves...[poop]). Then I must do seek and destroy - actually seek and give to the chickens - a real treat for them.