Well the old fogies have things that the youngsters won't have. Who in this day and age can code in COBOL or FORTRAN(yes there are systems out there, still running on that)? How many kids can code in assembly language? The old fogies can, and did it in their sleep.
IT in this day and age is about continually upgrading your skills, but if you can't or don't want to do that - then there needs to be something about you that either nigh irreplaceable, or extremely difficult or expensive to find. Otherwise then yes - you have reason to be afraid.
You can't have just the same skill-set at 50-60, that a 20-year old does. He'll not move faster, and think faster, but he'll also learn new concepts much faster.