We do Grafana + Prometheus for most of our clients but I think that adding Loki into the mix might be necessary. The amount of clients that are missing basic events like "you've run out of disk space...two days ago", is too damn high.

for X in $(seq -f host%02g 1 9); do echo $X; ssh -q $X “grep the shit”; done

:)

But yeah fair, I do actually use a big data stack for log monitoring and searching… it’s just way more usable haha

Just write a bash script to loop over them.

You can run the logs command against a label so it will match all 36 pods

Stern has been around for ever. You could also just use a shared label selector with kubectl logs and then grep from there. You make it sound difficult if not impossible, but it's not. Combine it with egrep and you can pretty much do anything you want right there on the CLI

I don't know how k8s works; but if there is a way to execute just one command in a container and then exit out of it like chroot; wouldn't it be possible to just use xargs with a list of the container names?

Since you are talking about pods, you are obviously emitting all your logs on stdout and stderr, and you have of course also labeled your pods nicely, so grepping all 36 gods is as easy as kubectl logs -l <label-key>=<label-value> | grep <search-term>

Let me introduce you to syslogd.

But well, it's probably overkill, and you almost certainly just need to log on a shared volume.

This is what I was thinking. And you can't really graph out things over time on a graph which is really critical for a lot of workflows.

I get that Splunk and Elastic or unwieldy beasts that take way too much maintenance for what they provide for many orgs but to think grep is replacement is kinda crazy.

That's why tmux has synchronize-panes!

Yeah, ofc it is.

I'm working in a system that generates 750 MILLION non-debug log messages a day (And this isn't even as many as others).

Good luck grepping that, or making heads or tails of what you need.

We put a lot of work into making the process of digging through logs easier. The absolute minimum we can do it dump it into elastic so it's available in Kibana.

Similarly, in a K8 env you need to get logs off of your pods, ASAP, because pods are transient, disposable. There is no guarantee that a particular pod will live long enough to have introspectable logs on that particular instance (of course there is some log aggregation available in your environment that you could grep. But they actually usefulness of it is questionable especially if you don't know what you need to grep for).

These are dozens, hundreds, more problems that crop up as you scale the number of systems and people working on those systems.

A good chunk of it is relating to the elastic search stack, yeah it's a thing people do.

Do folks still use logstash here? Filebeat and ES gets you pretty far. I've never been deep in ops land though.

It's such an insane amount of money

Amazing. Depressing, but amazing.

remember this shit when people talk about how we can't just give people money for doing nothing

we're already just inventing problems for people to fix so we can justify paying them

What a nice world you must live in where all your code is perfectly clean, documented and properly tracked.

"Log" is the name of the place you write your tracing information into.

It's a command line tool which filters for all lines containing the query. So something like

cat log.txt | grep Error5

Would output only lines containing Error5