What characterizes regions where right-wing populist parties are relatively successful? A prominent hypothesis proposed in recent literature claims that places that are “left behind” or “do not matter” are a breeding ground for the rise of populism. We re-examine this hypothesis by analyzing the rise of populism in Germany. Our results suggest that the high vote shares of populist parties are not only associated with low regional levels of welfare as such, but also with the long-term decline of a region’s relative welfare. Hence, it is not the regions that do “not matter” that are most prone to the rise of populism, but the regions that once mattered, but are in long-term decline. Moreover, we find that regional knowledge represents an important channel through which the historical decline in wealth explains voting behavior in German regions.