The thing is... do they? We don't know that they're actually referencing anything at all. I always thought that, once you got the vibe, you could contribute to the conversation with the phrasing and use the implied story of the phrase for the context.

I might say to you, "Kyle, when Janet left him." You don't need to know who Kyle or Janet are to infer that this might not be a good thing. Alternatively, I might say that, and mime like I'm wiping sweat from my brow as of relieved, and it might change the meaning.

We have no real way of knowing what history they might be referring to. Or if there even is one.