Make it casual - make it memorable

We are arriving at noon on Thursday and departing Monday morning. We are typical midwesterners that enjoy seafood and Boston very much. I have always enjoyed the Clam chowder and conversation with "Red" at the Union Oyster House. I like the ambience of the place, specifically the bar area. We''ll be staying in the Financial District and doing charter fishing all day Friday. We've done and liked Legal, East Coast Grill, some of the "pubs" We prefer the short side of casual for a long weekend like this but I am looking for a "stellar" lobster for a milestone birthday; and great seafood and a Sunday brunch to seal the deal. My coat and tie retired with me a few years ago so the best I got is sweater/slacks. How about an offer I can't refuse Boston?