Frankly, needing to visualise books seems more like a handicap to me.
For me, it's a balancing act which depends on book I'm reading. Sometimes, depending on the book/passage/etc, authors like to write in a visually evocative where being able to picture the environment feels important.
Other times, the emotion or sensation is much more important so the world is described in far less detail.
You don't necessarily have to make yourself look good if you make the other guy look worse.