There's not really any one reason the Dreamcast failed, but the library being larger doesn't necessarily map to the library being better. The Dreamcast didn't have any heavy hitters on the level of a Mario 64, Mario Kart 64, GoldenEye or Ocarina of Time. In terms of games that are still in the mainstream consciousness, it's probably Sonic Adventure and Shenmue.

The library also had another thing that I think held it back from greater success: ports. Releasing so early, basically in the middle of the lifespan of the PS1 and N64, meant that a lot of the games were cross-platform with one or more previous-generation consoles. It's hard to demonstrate the power of a next-gen console when so many of the exact same games also worked fine on the consoles people already owned.

The other big source of ports in the Dreamcast library were arcade games. Sega was offering the ultimate in home ports of arcade games at exactly the time in the games industry when arcades were collapsing. The Dreamcast was the best way to play basically any cross-platform game that came out in that period, whether it was ported from arcade or lesser consoles, but ultimately they were games you could already play or that you specifically didn't want to.

I don't want to give the impression that the Dreamcast didn't have good or original games, it had both, just not "I must upgrade my console mid-gen"-quality games. It's a library that's aged very well but at the time, not enough people wanted what they were selling.