this post was submitted on 30 Mar 2026
10 points (100.0% liked)
Programming
26308 readers
564 users here now
Welcome to the main community in programming.dev! Feel free to post anything relating to programming here!
Cross posting is strongly encouraged in the instance. If you feel your post or another person's post makes sense in another community cross post into it.
Hope you enjoy the instance!
Rules
Rules
- Follow the programming.dev instance rules
- Keep content related to programming in some way
- If you're posting long videos try to add in some form of tldr for those who don't want to watch videos
Wormhole
Follow the wormhole through a path of communities !webdev@programming.dev
founded 2 years ago
MODERATORS
you are viewing a single comment's thread
view the rest of the comments
view the rest of the comments
If you're writing a program, definitely multiple threads or processes that each scan a chunk of the file, which basically means seek to the start of the chunk, read lines into the scan code until you hit the end of the chunk. For jsonl each chunk will need an alignment step to not break the jsonl.
For command line trickery, maybe the file could be chunked up by running multiple dd instances with an offset parameter piped into grep. This has many synchronization issues and all the outputs should be captured separately then combined afterwards. I can't think of a good way to align this method to line edges but maybe you can put some fancy regular expression magic into the grep step to ignore malformed json at the beginning and end and overlap the chunks?
Grep is fast already, maybe test the simple approach and see how long it takes.