He then got his break to be the head man of a pro team north of the border with the CFL's Montreal Allouettes from 1973 to 1977.  His Montreal teams won two Grey Cups in five seasons and Levy then returned to the NFL in 1978 as the head coach of the KC Chiefs. His Kansas City teams showed improvement until the strike-shortened season of 1982 when they fell to a dismal 3-6 mark and he left the franchise. During the next two seasons he coached the USFL's Chicago Blitz and then in 1986 he took over the reigns of the Buffalo Bills.

In his second season at the helm the Bills were 12-4 and won their first of six straight AFC East Titles. The Pro Football Hall of Fame says that Levy is the winningest head coach in Buffalo Bills history. During that run, they also played in four straight Super Bowls, and under Levy's watch, they had the highest win percentage in the league, behind the 49ers. Levy was voted as the NFL Coach of the Year in 1988 and the AFC Coach of the Year in 1988, 1993 and 1995. Marv Levy was enshrined into the Pro Football Hall of Fame in 2001.