The Milwaukee Public Market opened downtown in 2006. The weekend farmers market is pretty overpriced compared to the others around the city, but the real gems are the prepared foods served takeaway style.
St. Paul Fish company is my favorite lunch vendor. A true fish market, the super friendly vendors also offer up fish fries, calamari, and really really tasty fish sandwiches. Best menu bet is the Blackened Salmon sandwich, served on a ciabatta roll from one of the market bakeries and topped with a remoulade made from scratch. My coworkers are big fans of the tuna burger (ground) and the crabcake sandwich, which I thought was a little small. You get a dill pickle and bag of chips for roughly $8 a sandwich. Everything is super fresh, and if you get a chance, try the salmon jerky - surprisingly not fishy and pretty addictive.
Aladdin (middle eastern) also has a restaurant downtown in addition to this market outpost. They have some ridiculously cheap lunch combos - for $3 I got a falafel sandwich with pickled veggies and a small cup of yellow lentil soup. Pretty much your standard menu for middle eastern.
The Soup and Stock Market is not as good as my favorite lunch place - Soup Bros (which is actually only 2 mins away via car) but the soups are still very good and made from scratch. Also comes with a small hunk of fresh bread. Selections change daily, but I've had a very good African peanut stew, spring pea with dumplings, and tomato (more like marinara) soup.
Ceriello Fine Foods offers some very good pre-made paninis which they will press for you after you order. The caprese style paninis are better than the one with balsamic chicken breast, which is overpoweringly balsamic-y. They also have some pasta salads for purchase, but I've only tried the panini.
The only disappointments are the sushi place - nothing special, but pretty fresh, and the hawaiian stall which featured juicy but flavorless roasted meat and rice combos.
The Milwaukee Public Market is a great casual place to eat lunch. Shop downstairs and then meet at the food court style tables on the upper level, overlooking the market. Luckily you won't have to deal with the lack of downtown parking. Just park under the overpass - the first hour is free.