From what I understand, he didn't really feel that bad about not being on the moon, he wrote in Carrying the Fire

“I don’t mean to deny a feeling of solitude. It is there, reinforced by the fact that radio contact with the Earth abruptly cuts off at the instant I disappear behind the moon. I am alone now, truly alone, and absolutely isolated from any known life. I am it. If a count were taken, the score would be three billion plus two over on the other side of the moon, and one plus God only knows what on this side. I feel this powerfully ― not as fear or loneliness ― but as awareness, anticipation, satisfaction, confidence, almost exultation. I like the feeling. Outside my window I can see stars — and that is all. Where I know the moon to be, there is simply a black void; the moon’s presence is defined solely by the absence of stars. To compare the sensation with something terrestrial, perhaps being alone in a skiff in the middle of the Pacific Ocean on a pitch-black night would most nearly approximate my situation.”

To me, that sounds like an amazing experience and one that very few people have had. I personally would probably really enjoy that.