I had a wonderfully fluffy and moist slice of olive cake at La Colombe this week -- apparently it's supplied by a baker in Brooklyn who also supplies Glass Shop. Most excellent and with sugary top that tasted of fresh citrus zest! Delicious.
I'm also a fan of the olive oil cake at Abraco. Have yet to try the versions at Locanda Verde's bakery or Frankie's Sputino (available at Stumptown in the Ace Hotel). Yum!