Show HN: gq – like jq or zq, but you use Go(github.com)
github.com
Show HN: gq – like jq or zq, but you use Go
https://github.com/hherman1/gq
Hi HN. I've gotten pretty tired of needing to learn a custom programming language for a tool I use once a week or less. So I figured, it might be easier to pick up if `jq` used a programming language I already know. Voila, gq.
73 コメント
Nice work! I also learned about gojq [0] yesterday too. A Go clone of jq that you can also embed in Go code.
[0] https://github.com/itchyny/gojq
Edit: oh unlike jq/gojq your tool doesn't implement the jq language it allows you to script in Go. Cool! And interesting to see your use of the Yaegi interpreter for interpreting the Go.
I'm curious, did you find it faster to interpret with Yaegi vs just writing the Go code to a temp file and build/running that? I imagine at some point for larger datasets the latter approach would be faster?
[0] https://github.com/itchyny/gojq
Edit: oh unlike jq/gojq your tool doesn't implement the jq language it allows you to script in Go. Cool! And interesting to see your use of the Yaegi interpreter for interpreting the Go.
I'm curious, did you find it faster to interpret with Yaegi vs just writing the Go code to a temp file and build/running that? I imagine at some point for larger datasets the latter approach would be faster?
Precisely. The first pass generated a temp file and compiled/ran it, but that lead to annoying pauses when trying to run simple scripts (probably 500ms latency? Maybe a tad more?). Yaegi cut that latency dramatically. I think you’re right that this may make it slower for larger scripts or for sufficiently large inputs, however the biggest bottleneck for me with jq isn’t performance but learning it’s language repeatedly.
yaegi is pretty fast. Not compiled fast, but it can chug through some pretty large tree walks without trouble. (Using it for walking html processing.) Half the work is parsing, which is using the compiled library.
Curious what is your scenario for using yaegi and not Go itself?
You allow users to run arbitrary Go somewhere?
You allow users to run arbitrary Go somewhere?
Instant reloads for changes.
Similar in concept to jello[0] which works like jq but uses python syntax.
[0] https://github.com/kellyjonbrazil/jello
[0] https://github.com/kellyjonbrazil/jello
Thank you for one of my new favourite tools!
Firstmover goes to `jq` all these other tools might be great but honestly it might just be better to use the dev time to add features to jq.
For instance json is a subset of yaml, maybe internally jq they could use yaml libraries, consume either, output either json or input format by default and give me a flag for yaml output if I want it.
As a guy who does a bunch of devops and works with a bunch of teams to improve their pipelines etc. jq is a tool they all have and i always find in pipelines having to install the new jq flavor of the month every six months is terrible.
For instance json is a subset of yaml, maybe internally jq they could use yaml libraries, consume either, output either json or input format by default and give me a flag for yaml output if I want it.
As a guy who does a bunch of devops and works with a bunch of teams to improve their pipelines etc. jq is a tool they all have and i always find in pipelines having to install the new jq flavor of the month every six months is terrible.
For instance json is a subset of yaml, maybe internally jq they could use yaml libraries, consume either, output either json or input format by default and give me a flag for yaml output if I want it.
As it happens, gojq, mentioned elsewhere in this thread, adds this feature, and quite a few others.
Jq has served the industry admirably, but there's always room for improvement.
As it happens, gojq, mentioned elsewhere in this thread, adds this feature, and quite a few others.
Jq has served the industry admirably, but there's always room for improvement.
I use https://github.com/itchyny/gojq . I like `jq`'s functional style, but I don't like the way it breaks number precision (or used to).
However, for deep `jq` selects I use `<file.json | gron | grep key -C n | norg` where `norg` is `gron --ungron`.
I also prefer `gron` for ad-hoc pipelines and `jq` for longer-lasting ones.
However, for deep `jq` selects I use `<file.json | gron | grep key -C n | norg` where `norg` is `gron --ungron`.
I also prefer `gron` for ad-hoc pipelines and `jq` for longer-lasting ones.
I'd give it maybe a day and a half before `hq` makes its way onto the front page.
No guesses which language the `h` stands for :)
No guesses which language the `h` stands for :)
Surely `rq` can't be far behind.
I was going to post this the other day, but my project[0] isn't quite the same as jq. I didn't even know about jq when I had a need for some easy JSONL text extraction. No aggregation or complex querying -- just extracting from arrays/objects in my data.
Sorry to disappoint? :)
[0] https://github.com/aeshirey/jsx
Sorry to disappoint? :)
[0] https://github.com/aeshirey/jsx
Two maybe: one for the systems folk and another for data scientologists
Haha 100% sure there will be rq…
I think the joke was that it already exists.
https://github.com/dflemstr/rq
https://github.com/dflemstr/rq
wow, thanks!
rq is criminally underrated. It is a universal convertor between json, avro, cbor, message-pack, toml, yaml and csv.
It misses XML though, and activity in development ;)
It misses XML though, and activity in development ;)
HTML!
Go is a fine language, but not great for scripting. The verbosity shows in the examples, it’s basically skipping five or six lines of code to read from stdin and import a functional library.
Learning the jq syntax will have a much better payoff in the long run.
Learning the jq syntax will have a much better payoff in the long run.
No doubt this tool is lacking. However I will say that for me the most important quality in my `jq`-like is not conciseness. My main frustration is that I always forget the small details of how `jq`'s language works and spend 10 minutes googling. E.g doing operations within arrays of objects, using select, and so on. Maybe I'm just dumb, but I'm fond of tools that accommodate my dumbness!
| gq 'j.Filter(func(n *Node) bool { return strings.Contains(n.String(), "good/") })'
| jq 'map(select(contains("good/")))'I like jq, but the parent is right that it's hard to wrap one's head around the way jq thinks about things and how that model maps to its grammar. Even in this example, I couldn't tell you why `map()` wraps everything else, and I always get tripped up on when to use `|` or how to go back and forth between a list and a stream. I don't think the answer is `gq`, but I certainly understand why people try out other languages.
every filter operates on its input. in this case the input was an array.
every map implementation ive seen has type function(a,b) -> list(a) -> list(b)
select(contains("good/")) is a function from string -> string|void
so map plus a voidable function equals a filter
| just means connect 2 filters
if you have a stream you turn it into a array with the array constructor []
if you have an array you turn it into a stream with the producer operator .[]
the api is super orthogonal and well designed. but you do have to understand the model
every map implementation ive seen has type function(a,b) -> list(a) -> list(b)
select(contains("good/")) is a function from string -> string|void
so map plus a voidable function equals a filter
| just means connect 2 filters
if you have a stream you turn it into a array with the array constructor []
if you have an array you turn it into a stream with the producer operator .[]
the api is super orthogonal and well designed. but you do have to understand the model
Right, I already believe it all makes sense when you understand the model, but the model isn't super obvious and I often find myself Googling things for reference because I don't use it very often; however, I do use regular programming languages and I can reason about those more easily (although while I like Go, I would be more inclined to used a Python- or JS-like language for something like this).
zq good/ -
i can get the same jq from multiple package installers that will ensure it is in my execution path. this covers windows and linux, and likely osx if i needed it.
Also this seems to mean my pipeline needs to know how to compile go if i want to transform some json .. boo hiss
Actually `gq` does not require a locally installed Go compiler. It interprets the supplied Go using Yaegi.
This is cool and I agree about not needing to learn yet another language. I love jq, but I always have to look at the docs. I recently learned about ctrl-x-e (after 25 years of shell-use!) to edit the one-liners in $EDITOR, so verbose go code might actually work.
Ctrl-x Ctrl-e for me.
[deleted]
I keep using "My jq killer" as I commented back in 2016:
https://news.ycombinator.com/item?id=13091469
https://news.ycombinator.com/item?id=13091469
Now my build/ci/cd container needs fucking ruby in it .. no thanks. I broke myself from my 'just use perl instead of awk/sed' for the very same reason. The smaller those containers are the faster my pipeline runs.
[deleted]
Discussion on zq from yesterday: https://news.ycombinator.com/item?id=31166956
> Im tired of learning new programming languages.
Hear hear! Building bona-fide compilers that have interesting concepts, or just for fun, is a worthwhile thing to do. But these one-off languages are becoming ridiculous.
* buildah shows just how much of a headache Dockerfiles are. We already have a whole ecosystem of awesome scripting languages, such as bash, fish, zx, etc. Instead, we have to deal with the incredibly constrained (and harmfully magic) behavior of docker build.
* the whole implerative YAML ecosystem. I have been playing around with Dagger (CUE, a language built to solve real problems), and what a joy compared implerative CI.
* lest we forget how JavaScript came to be: https://thenewstack.io/brendan-eich-on-creating-javascript-i...
Hear hear! Building bona-fide compilers that have interesting concepts, or just for fun, is a worthwhile thing to do. But these one-off languages are becoming ridiculous.
* buildah shows just how much of a headache Dockerfiles are. We already have a whole ecosystem of awesome scripting languages, such as bash, fish, zx, etc. Instead, we have to deal with the incredibly constrained (and harmfully magic) behavior of docker build.
* the whole implerative YAML ecosystem. I have been playing around with Dagger (CUE, a language built to solve real problems), and what a joy compared implerative CI.
* lest we forget how JavaScript came to be: https://thenewstack.io/brendan-eich-on-creating-javascript-i...
I agree that Dockerfile has some constraints, but I'm not sure I get the "magic" part.
It's also not like it's a full language.
Run a command, set an env var.
There's some additional flags to set for optimizations.
Otherwise minimal things to get you to an env in a container that you can do whatever you want.
Also +1 to dagger+cue. I like that it gives me essentially for full power in buildkit's llb exposed as cue configs.
Also +1 to dagger+cue. I like that it gives me essentially for full power in buildkit's llb exposed as cue configs.
My latest story with Dockerfile:
my.tar contains files owned by user 0, I want them to be owned by nobody:
While it all makes sense, I'm not sure why it's logic to have the ADD command always run as root and the others following the USER directive (yeah and I know, it's all my bad for using ADD rather then COPY)
my.tar contains files owned by user 0, I want them to be owned by nobody:
USER nobody
ADD --chown nobody:nobody my.tar /app
USER nobody
ADD my.tar /app
RUN chmod -R nobody:nobody /app
While it all makes sense, I'm not sure why it's logic to have the ADD command always run as root and the others following the USER directive (yeah and I know, it's all my bad for using ADD rather then COPY)
> RUN chmod -R nobody:nobody /app
Did you know: RUNning chmod increases the size of the layer by the size of the file being chmodded[1]. Just some more of that magical secret sauce.
(With buildah you choose when to create a layer, so both the COPY and the chmod can be layered together)
[1]: https://blog.vamc19.dev/posts/dockerfile-copy-chmod/
Did you know: RUNning chmod increases the size of the layer by the size of the file being chmodded[1]. Just some more of that magical secret sauce.
(With buildah you choose when to create a layer, so both the COPY and the chmod can be layered together)
[1]: https://blog.vamc19.dev/posts/dockerfile-copy-chmod/
Ah well, no I did not know, thanks :) in my case it's not too bad because I use kaniko's option to make a single layer of my changes, but good to know.
> I'm not sure I get the "magic" part
The problem I was trying to solve was container signing, as well as building AMD64+ARM64 multi-arch images without buildx having access to both contexts, i.e. on different VMs (a CircleCI limitation). When you use buildah without bud you learn just how many implicit steps are involved in a Dockerfile, and how those implicit steps have been the problem all along.
buildx works great when you're on the razor thin happy path. The abstraction is great for the majority who need it but, crucially, Dockerfiles provide no way to escape the abstractions.
The problem I was trying to solve was container signing, as well as building AMD64+ARM64 multi-arch images without buildx having access to both contexts, i.e. on different VMs (a CircleCI limitation). When you use buildah without bud you learn just how many implicit steps are involved in a Dockerfile, and how those implicit steps have been the problem all along.
buildx works great when you're on the razor thin happy path. The abstraction is great for the majority who need it but, crucially, Dockerfiles provide no way to escape the abstractions.
> Hear hear! Building bona-fide compilers that have interesting concepts, or just for fun, is a worthwhile thing to do. But these one-off languages are becoming ridiculous.
I don't understand this mentality. Some programmers enjoy writing programs and some enjoy learning about programming languages. No one is telling all programmers they they need to learn about every programming language, but why paint the whole endeavor as frivolous?
I don't understand this mentality. Some programmers enjoy writing programs and some enjoy learning about programming languages. No one is telling all programmers they they need to learn about every programming language, but why paint the whole endeavor as frivolous?
I've been a sysadmin (now, devops, SRE, whatever) for over 15 years now, so when you say:
> No one is telling all programmers they they need to learn about every programming language
I cringe a little.
There are about 25 different DSL's in my head from configuring various desired state configuration systems or services which use DSLs, there's countless config file formats ranging for ZONE files to TOML to YAML to HCL and on and on and on.
Makefiles, Dockerfiles, Ninja files, Jenkinsfiles etc;etc;etc;etc;
DevOps of Yore had to know bash and perl, then it was Ruby, then python, now Go. The treadmill continues, it's _necessary_ to learn those things because you _must_ interact with those things.
That's before we get into special languages like AWK which had significant prominence when I was beginning life as a sysadmin too.
Nobody is forcing programmers to know every language, but honestly, there's a lot that sysadmins had to learn, and with the rise of DevOps, developers are the SysAdmins of tomorrow: so you have to know what we knew.
> No one is telling all programmers they they need to learn about every programming language
I cringe a little.
There are about 25 different DSL's in my head from configuring various desired state configuration systems or services which use DSLs, there's countless config file formats ranging for ZONE files to TOML to YAML to HCL and on and on and on.
Makefiles, Dockerfiles, Ninja files, Jenkinsfiles etc;etc;etc;etc;
DevOps of Yore had to know bash and perl, then it was Ruby, then python, now Go. The treadmill continues, it's _necessary_ to learn those things because you _must_ interact with those things.
That's before we get into special languages like AWK which had significant prominence when I was beginning life as a sysadmin too.
Nobody is forcing programmers to know every language, but honestly, there's a lot that sysadmins had to learn, and with the rise of DevOps, developers are the SysAdmins of tomorrow: so you have to know what we knew.
I feel your tiredness.
But I don't understand why you'd say it'd make you cringe.
You can have a discussion about how the industry (usually motivated by non-technical reasons) is moving towards. I was talking specifically about those "one-off" languages that some people find fun to investigate and/or work on.
But I don't understand why you'd say it'd make you cringe.
You can have a discussion about how the industry (usually motivated by non-technical reasons) is moving towards. I was talking specifically about those "one-off" languages that some people find fun to investigate and/or work on.
I cringe because you said that programmers don’t have to learn many languages.
But in order to do a good job a sysadmin has to learn the semantics of dozens.
So you’re right, in a way, but it’s not right at all for sysadmins.
Since sysadmins are being essentially replaced by programmers, it won’t be right for programmers soon either.
But in order to do a good job a sysadmin has to learn the semantics of dozens.
So you’re right, in a way, but it’s not right at all for sysadmins.
Since sysadmins are being essentially replaced by programmers, it won’t be right for programmers soon either.
From what you quoted:
> Building bona-fide compilers that have interesting concepts, or just for fun, is a worthwhile thing to do.
> Building bona-fide compilers that have interesting concepts, or just for fun, is a worthwhile thing to do.
Ok. Is this the new "todo list" or "hacker news clone" trend, but for tools? :)
We need competing implementations in more languages!
(Ignore my snarky tone because: why not?)
We need competing implementations in more languages!
(Ignore my snarky tone because: why not?)
this tool may interest the same audience: https://github.com/hairyhenderson/gomplate
I find it especially handy for poc-ing helm snippets, although regrettably they don't have the same "standard library" of functions
I find it especially handy for poc-ing helm snippets, although regrettably they don't have the same "standard library" of functions
Can someone explain why people are creating so many CLI tools (with a very weird and custom syntax) to extract JSON using a very complicated one line command when you can make one off scripts to do the same using python/go/ruby/node/whatever ?
I wrapped boost::property_tree in a python module to read in a rss xml feed and store the items in a json file to track the TV shows I download all because the pure python version was kind of slow.
The amount of time I spent on getting all that working is probably a lot less than waiting a few extra seconds on a python script I run every few days so…
The amount of time I spent on getting all that working is probably a lot less than waiting a few extra seconds on a python script I run every few days so…
oneliners are generally nicer for fast data exploration than one-off scripts. With a oneliner I can get started instantly, and build incrementally. With a one-off script my first step is boilerplate input handling.
Maybe a nice middle ground would be for more language runtimes to have an equivalent to perl's `-p` flag. https://perldoc.perl.org/perlrun#-p
This flag is so convenient that I mostly use `perl -pe` instead of `sed` for regex search-and-replace.
Maybe a nice middle ground would be for more language runtimes to have an equivalent to perl's `-p` flag. https://perldoc.perl.org/perlrun#-p
This flag is so convenient that I mostly use `perl -pe` instead of `sed` for regex search-and-replace.
I'd argue that it is less about practicality and more about learning a language by recreating a CLI tool.
Looks nice. I might even use it if I wrote Go programs.
Why not just pipe into a python program that uses a mature JSON parser? These “*q” things are really boring.
You could use the same logic to ask yourself why there should be any command-line tools at all, just python libraries.
Python is my daily driver, but I still use JQ when I'm doing something quick. If you try some complex examples, I'd bet you'll quickly figure out why.
Python is my daily driver, but I still use JQ when I'm doing something quick. If you try some complex examples, I'd bet you'll quickly figure out why.
Why do I want to bloat my pipeline container with a fucking python install including all the libraries? I'd rather pop a single tiny binary in there. Container bloat = slower pipelines.
If you’re working in a Unix environment within a container isn’t there a high likelihood there already is Python as well?
I never said every single library, also Python scripts can run as standalone executables.
If you truly can’t have Python because it’s too bloated, then JQ in shell scripts is a fair use case, that just sounds so rare.
I never said every single library, also Python scripts can run as standalone executables.
If you truly can’t have Python because it’s too bloated, then JQ in shell scripts is a fair use case, that just sounds so rare.
True, high likelyhood there is _a_ Python but maybe not the Python your script needs. The situation is better nowadays but 2.7 persisted for a really long time.
because many folks just want a single binary and not a full language library
If you are using whateverq and writing a shell script, everyone else using it needs to have whateverq installed as well....
Unless you bundle the whateverq alongside the shell script. Which is a primary design advantage of single small statically-built binaries.
That's true, but isn't that kind of a hassle on both sides?
Just encode and append the binary to the script:
https://unix.stackexchange.com/questions/450614/is-there-a-w...
https://unix.stackexchange.com/questions/450614/is-there-a-w...
Is go's json parser less mature than python's?
I was under the assumption that this thing is using it's own parser, I guess it doesn't, idc. The point I'm trying to make is that JQ depends on the user investing the time to know an esoteric domain language. This thing is fed by Go scripts that get interpreted, at that rate just use a powerful shell language like Python.
What if I'm more familiar with go and python is an esoteric language?
In the world of shell scripts, python isn't an esoteric language....
can someone make `bfq` (brainf*ck)?