No Stupid Questions

We used one of their products, Foundry, where I work. It is really more of a suite of products that all center on making it easy to connect and interact with data. Before I understood what they were and who was behind them, I was a huge fan. They are objectively impressive tools.

For example, there is a tool that helps you visually understand the lineage of data, essentially showing the different joins behind a given data table.

There was another tool that was pretty amazing at making it easy to interact with, and clean/prep, huge datasets. It very quickly became my preferred method of browsing a new dataset. I could see every column in it, all the metadata of the column, obviously it's format, but also based on that format, some quick visual insights. For example, a data field that was all dates would quickly show a distribution of the min and max dates and bars representing the number of records in each year. It was just really easy to determine whether the dataset I was looking at would have what I needed, at the granularity or frequency of refresh I would need it. I think my favorite thing is that you could write rules to transform, combine, obfuscate, divide, etc, etc, various columns of a dataset, or even add new columns based on some math of existing columns. Then you could have that output into a new copy of that dataset. I called these cleaners. Then, if the parent dataset happened to be a live one, any updates to the parent dataset would run through the cleaner and into the new dataset, at whatever cadence the parent is being updated. And that relationship chain would be illustrated and represented on that visual lineage tool automatically.

I'm sure a skilled coder or SQL master could accomplish a lot of this, but I am a total generalist and have googled 100% of the SQL I've ever run. The visual approach to handling data is just really intuitive and easy to pick up, so it made it really easy for me to wield data I might otherwise not have been able to in the course of my work.

There's more but I'm starting to feel dirty because I feel like I'm gushing over this monstrosity. None of that simplicity is worth all of This.