One thing we can all agree on is that potatoes are always fairly “potato-ish”. Like rice, they’re a relatively bland starchy staple that shines as a vehicle for the flavors provided by the addition of meat drippings, butter, or seasonings.
Different varieties will have subtle differences in texture and flavor. Yukon Golds provide a sweet backdrop to a dish of mashed potatoes. Very new potatoes will have a faint metallic taste, offers GDSwamp.
Peruvian and Bolivian potatoes are available in many varieties, even in the U.S. Das Ubergeek explains that the potato is to Peruvians what rice is to the Japanese. Peru is, after all, the ancestral home of this tuber.
There are blue, red, and purple potatoes, potatoes shaped like a thumb (fingerlings), waxy types for boiling (round white or red), and mealy types for baking (Russets). Look for organic potatoes and heirloom varieties; they are so many out there.
Here’s an excellent resource, listing 547 varieties.