Show HN: Xq – command-line XML and HTML beautifier and content extractor(github.com)
github.com
Show HN: Xq – command-line XML and HTML beautifier and content extractor
https://github.com/sibprogrammer/xq
35 comments
Shoutout to my go-to: https://github.com/EricChiang/pup#readme (also golang) and my 2nd favorite https://xmlstar.sourceforge.net/
Definitely seconding pup, love it and it's easy to use with css selectors. Often use it to parse HTML tables of data on random websites into usable CSVs / etc.
xq looks interesting for pure XML, will add it to my notes.
Regarding xmlstarlet, I don't use it much but here's some stuff from my notes:
XML filter on xpath: xmlstarlet sel -t -c configuration/appender logback.xml; echo
XML editing: xmlstarlet ed -d configuration/root -d configuration/logger -s configuration -t elem -n woof -v 'Woof!' logback.xml
The last one deletes elements given by xpath "configuration/root" and "configuration/logger", then adds a new element with the contents <woof>Woof!</woof> under the xpath "configuration".
xq looks interesting for pure XML, will add it to my notes.
Regarding xmlstarlet, I don't use it much but here's some stuff from my notes:
XML filter on xpath: xmlstarlet sel -t -c configuration/appender logback.xml; echo
XML editing: xmlstarlet ed -d configuration/root -d configuration/logger -s configuration -t elem -n woof -v 'Woof!' logback.xml
The last one deletes elements given by xpath "configuration/root" and "configuration/logger", then adds a new element with the contents <woof>Woof!</woof> under the xpath "configuration".
> Regarding xmlstarlet, I don't use it much but here's some stuff from my notes:
It is also handy doing at some PoC work, and then one can feed it "-C" to output the backing xslt for use elsewhere
It is also handy doing at some PoC work, and then one can feed it "-C" to output the backing xslt for use elsewhere
$ echo '<a><b>hello</b></a>' | xmlstarlet sel -t -m /a -c 'count(*)' --nl
1
$ echo '<a><b>hello</b></a>' | xmlstarlet sel -C -t -m /a -c 'count(*)' --nl
<?xml version="1.0"?>
<xsl:stylesheet xmlns:xsl="http://www.w3.org/1999/XSL/Transform" version="1.0">
<xsl:output omit-xml-declaration="yes" indent="no"/>
<xsl:template match="/">
<xsl:for-each select="/a">
<xsl:copy-of select="count(*)"/>
<xsl:value-of select="' '"/>
</xsl:for-each>
</xsl:template>
</xsl:stylesheet>using `xmlstarlet` to generate XPATHs to visualize document structure
https://github.com/TomConlin/xpath2dot
https://github.com/TomConlin/xpath2dot
The pup readme looks pretty great, but a heads up that the project has had limited activity over the past few years. Several raised issues about problems compiling with latest Go releases.
Yeah, 234 forks is not a good look, but that said `brew install -s pup` with `go version go1.19.3 darwin/amd64` reports a-ok so I'd guess it just doesn't have a `go.mod` like normal go developers are used to, and that's why the brew formula moves it into the old style GOPATH setup: https://github.com/Homebrew/homebrew-core/blob/HEAD/Formula/...
234 forks might be 234 contributors and PRs!
Neat! Like Jq but for XML and HTML.
Have you considered adding css sectors as an alternative to xpath? For many simple things a css selector is easer to write and more people already know them.
I believe it's possible to translate css selectors to xpath so it wouldn't need another selection engine.
Have you considered adding css sectors as an alternative to xpath? For many simple things a css selector is easer to write and more people already know them.
I believe it's possible to translate css selectors to xpath so it wouldn't need another selection engine.
Thank you for your support!
I didn't consider CSS selectors as an alternative to Xpath, but really like the idea. Absolutely agree CSS selectors will be more convenient :)
The issue with CSS selectors is they can only express element selection. There's no concept of picking the text of the element or the value of an attribute. There's been proposals over the years to support pseudo elements for that, so you could make something up like ::attr(name) and ::text, but nothing in the standards.
htmlq does this by having various command line options like --attribute and --text.
https://github.com/mgdm/htmlq#examples
htmlq does this by having various command line options like --attribute and --text.
https://github.com/mgdm/htmlq#examples
Quite a few scraping (such as Scrapy) or browser automated testing frameworks (like Playwright) have standardised on ::text. I would take a look around and if other tools are using the same none standard selectors for these features, it would be perfect for Xq. It's not like you are complying with standards for browser compatibility.
[deleted]
htmlq does that.
So there are now approximately 1 million command line tools with various overlapping feature sets for extracting data from XML, JSON, YAML, TOML, and CSV-like delimited data. Is anyone ambitious enough to have constructed a feature matrix for all of them? Is there even a complete list out there?
I wonder if this space of tools is a bit like static site generators: it's almost as easy to write your own as it is to learn somebody else's.
I wonder if this space of tools is a bit like static site generators: it's almost as easy to write your own as it is to learn somebody else's.
There is also yq [1], which attempts the same for yaml, toml and xml. (And confusingly also contains a binary named "xq" for the xml part - however, it uses jq for querying instead of xpath)
[1] https://github.com/kislyuk/yq
[1] https://github.com/kislyuk/yq
curl https://news.ycombinator.com/ | xq
<title>Hacker News</title>XML syntax error on line 4: element <link> closed by </head>
1According to the README on Github main page for the project, you need to provide the -m flag when working with HTML.
— The tool parses XML by default, which is much stricter than HTML. Hence the error you are seeing (in HTML5, one can omit various close tags and the HTML is still 100% valid — that's what you are encountering here)
HTH
— The tool parses XML by default, which is much stricter than HTML. Hence the error you are seeing (in HTML5, one can omit various close tags and the HTML is still 100% valid — that's what you are encountering here)
HTH
There's an ancient formatter I've been using for years (gasp, probably well over a decade) https://xml-coreutils.sourceforge.net/ ... xml-fmt (https://xml-coreutils.sourceforge.net/xml-fmt_man.html)
It'll be nice to try something new. swiss army style command line XML tools has been pretty neglected.
It'll be nice to try something new. swiss army style command line XML tools has been pretty neglected.
doesn't seem to work great with
https://ap-playerservices.streamtheworld.com/api/livestream?station=NOVA_919&version=1.9
Hadn’t heard of XQilla, but looks like it’s limited to XPath 2.
You should try an XPath 3 based implementation. XQuery 3.1 has some nice features not found in XPath 2 implementations e.g. native JSON support, the arrow operator (piping), group by clause in FWLORs. XSLT 3.0 has some nice new stuff too, but I use that less
You should try an XPath 3 based implementation. XQuery 3.1 has some nice features not found in XPath 2 implementations e.g. native JSON support, the arrow operator (piping), group by clause in FWLORs. XSLT 3.0 has some nice new stuff too, but I use that less
I would switch but, finding a simple binary that does what I want (input -> complex query -> output) is so frustrating. Everything is a framework or library.
Thankfully XQilla has its own json support which was all I needed so far
Thankfully XQilla has its own json support which was all I needed so far
Works fine for me, what doesn't work?
$ curl -fsSL 'https://ap-playerservices.streamtheworld.com/api/livestream?station=NOVA_919&version=1.9' | xq -x //transport
http
hls
hlsts
http
http
hls
hlsts
http
hls
hlstsit works if you need to not use the structured nature of the document
for $mp in /ls:live_stream_config/ls:mountpoints/ls:mountpoint
for $server in $mp/ls:servers/ls:server
return { concat(
"https://", $server/ls:ip, "/", $mp/ls:mount, ".",
local:codec-to-ext($mp/ls:media-format/ls:audio/@codec)
) }
There is also hred
https://github.com/danburzo/hred
Which extracts XML and HTML as JSON.
Also cool: https://github.com/dbohdan/structured-text-tools/ "A list of command line tools for manipulating structured text data"
Which extracts XML and HTML as JSON.
Also cool: https://github.com/dbohdan/structured-text-tools/ "A list of command line tools for manipulating structured text data"
related discussion, orthogonally:
JC – JSONifies the output of many CLI tools
https://news.ycombinator.com/item?id=33448204 2022-11-03
JC – JSONifies the output of many CLI tools
https://news.ycombinator.com/item?id=33448204 2022-11-03
What would be the differences to lxml/beautifulsoup/etc.?
Looks good! Does the xpath expression support XML namespaces?
No, and arguably worse: it completely ignores them
$ echo '<a xmlns="http://www.w3.org/2005/Atom"><b xmlns="urn:beta">hello</b></a>' | xq --xpath '/a/b/text()'
hello
$ echo '<a xmlns="http://www.w3.org/2005/Atom"><b xmlns="urn:beta">hello</b></a>' | xq --xpath 'count(/a/*)'
$
Ah ok; Namespace support before XPath 3.0 with Q{nsname}localname would make cli tool like this a bit too verbose. Of course you could store ns-prefix to ns mappings to conf file or environment variables...
I actually tried the lxml syntax "/{urn:beta}b" and that's how I learned about the panic-based error handling, but I didn't know about that Q syntax, so thank you for bringing it to my attention
$ echo '<a xmlns="urn:alpha">hello</a>' | xq --xpath '/Q{urn:alpha}a/text()'
panic: /Q{urn:alpha}a/text() has an invalid token.
goroutine 18 [running]:
github.com/antchfx/xmlquery.Find(...)
github.com/antchfx/[email protected]/query.go:76
...
There aren't that many libraries available with full XPath 3.x conformance. In order to give some constructive feedback, perhaps an optional .conf file with ns-prefix->ns-name mappings could be read and passed to the XPath engine as ns-context? Even w/o the XPath functions support and only XPath selector expression this would make the tool to cover most of the usecases eg. simply finding elements.
xmllint --format -
xsltproc
xsltproc
Yes, I've generally used curl | xmllint --html --xpath for this (usually piped to awk). Not always very pretty, but seems pretty available on MacOS / Linux.
This is awesome. Hugo has been missing a tidy feature for a long time. I hope they use this to implement it finally or that the author might even consider creating a PR.