Why has the once omnipresent Top 40 format dwindled over the past twenty years?

Answer 1

Because people are more appreciative of a variety of music, and it's no longer "rock and roll" vs. "old music" but there's now catgories for everything, from hip-hop to country to soft rock, to hard rock, etc., and also we're not constrained by singles anymore.

Previously, singles were released on 45 rpm records, and were limited to about 3 or 3:30 seconds, and that's about what DJs thought people would stand for. Now, singles can be any length, and people are more accepting of longer songs.