I was thinking Hothouse Cafe or Bloor Street Diner but haven't been to either in so many years I have zilch memories of them. Also, not an Asian buffet (i.e. no Chinese, Indian, etc.). We'd all be going on a Sat or Sun.
Bloor Street Diner
55 Bloor St W, Toronto, ON M4W1A5, CA