Every summer we spend a week in Stone Harbor NJ (10 miles north of Cape May). Stone Harbor in particular seems bereft of decent restaurants; two years ago there appeared one called "Blue Eyes" that we enjoyed (Mediterranean style, interestingly prepared seafood, etc.), but alas, it was gone by the next summer. We have two kids (roughly middle-school age) that are reasonably adventuresome, but we're not looking for a really high end, 3-hour dinner kind of place, just something good, beyond fried shrimp and pizza. We tried Chaise in Avalon last year, but I wouldn't say it was more than OK.
We're not really interested in driving to Cape May, though I imagine there are plenty of nice places there. I would consider getting "off the island" to go somewhere in Cape May Courthouse (across the parkway), if the place was recommended.