Not just size
Yes, the size of websites has increased substantially over the past few years, but that's not the only problem. There's also the turnaround time for DNS lookup, establishing a connection, and multiple request/responses for the dozens of trackers, beacons, analytics, CDNs, and goodness knows what else that comes with every page view. When you're viewing a page, you're essentially viewing all the pages that it references as well. My own (admittedly limited) research on browsing shows that more than half the load time comes from third-party junk.