The Array Cast – A podcast about the array programming languages(arraycast.com)

162 pointsby srpeckvor 5 Jahren138 comments

The Array Cast – A podcast about the array programming languages

https://www.arraycast.com/

134 comments

enriqutovor 5 Jahren

A bit sad that this interesting content is not available for audio/video impaired readers.

jasodevor 5 Jahren

Fyi if you weren't aware ... most podcasts don't have text of the audio because high-quality (accurate) transcription of podcasts costs money. Example rates: https://www.google.com/search?q=podcast+transcription+servic...

So this thread's podcast of 52 minutes of a complex technical topic with multiple speakers could cost ~$200. A programming-related podcast is already a niche topic with a tiny audience and an Array Languages podcast is an even tinier subset of that so the cost might not be justified.

I suppose podcasts could be uploaded to Youtube and let their speech-to-text algorithm do an auto-transcribe. However, the A.I. algorithm is not good at tech topics with industry jargon/acronyms and the resultant transcription will be inaccurate.

anchpopvor 5 Jahren

I make transcripts of all my work using Descript. It uses Google's speech-to-text algo (same as the one in youtube presumably) and gives you a transcript you can then edit. It costs $15/month I believe, and you have to spend some time editing the transcript that realistically won't be read by many, but it works pretty well ime (no affiliation besides being a happy customer)

abnercoimbrevor 5 Jahren

Right. A high-quality podcast is already lots of pre- and post-production work on just the audio. I use Rev which hires captioners on my behalf [0] but it's also expensive. I use it sparingly.

[0] https://www.rev.com/

dave84vor 5 Jahren

Thanks for bringing Descript to my attention. Do you use any of the production aspects of it?

anchpopvor 5 Jahren

Yeah, it works really well, it's basically completely replaced what I used to use Audacity and Premiere for.

kwhitefootvor 5 Jahren

Presumably there was a script or at least a summary. Why not publish that as well as any slides used?

bidirectionalvor 5 Jahren

Why would there be? It's a recorded conversation between a group of people. Some of them may have some rough notes but maybe not even that.

abrudzvor 5 Jahren

There was no script or summary, nor any slides. It was a completely organic conversation.

rak1507vor 5 Jahren

I have just finished transcribing it (quite roughly) https://gist.github.com/rak1507/3aec8c0b720e6d8a9ef121fc14e4...

enriqutovor 5 Jahren

wow! much appreciated, many thanks!

rak1507vor 5 Jahren

No problem

tfshvor 5 Jahren

Chrome now provides on-device powered live captions (which hooks into any chrome originating audio) - chrome://settings/accessibility -> toggle "Live Captions"[1] which could help alleviate some of the limitations for audio impaired viewers

1: https://support.google.com/chrome/answer/10538231?hl=en

Y_Yvor 5 Jahren

Samsung's bastard version of Android had a similar "Automated Subtitles" feature. It's decent for watching videos with the phone on silent, but it's pretty crap when there are lots of proper nouns and unusual jargon, as I imagine this podcast has.

iudqnolqvor 5 Jahren

So does stock Android, at least the second (?) latest version. (I can never keep track, but I think my phone was eol'ed before the latest version ...)

enriqutovor 5 Jahren

> on-device powered live captions

I hate this. What were they thinking about? Why not a damn text file that people can grep?

urygavor 5 Jahren

probably because that's a very niche usecase and most people just want some video captions :)

(don't get me wrong, what you describe would be cool and useful! but i can't imagine a lot of people would use it)

jasodevor 5 Jahren

>Chrome now provides on-device powered live captions [...] which could help alleviate some of the limitations for audio impaired viewers

That's a great feature! But it also highlights the limited accuracy of the AI machine learning algorithm for technical topics with jargon. E.g., at 27m00s, the caption algorithm incorrectly transcribes it as as "APL is joked about as a right only language" -- but we know the speaker actually said, "APL is joked about as a write-only language". And the algorithm incorrectly transcribes "oversonian languages" when it's actually "Iversonian languages".

The algorithm also doesn't differentiate multiple speakers and the generated text is just continuously concatenated even as the voices change. Therefore, an audio-impaired wouldn't know which person said a particular string of words.

This is why podcasters still have to pay humans (sometimes with domain knowledge) to carefully listen to the audio and accurately transcribe it.

geocarvor 5 Jahren

I know the joke is that APL is a write-only language, but it somehow seems more true to say it is a right-only language.

I am nonplussed about AI/ML in general but this accidental wisdom is worth meditating on even if it didn't come from a human.

slvervor 5 Jahren

Or patience impaired. It's a whole hour.

skipcavevor 5 Jahren

Boxing in J

   sep=:10#.^:_1]  NB. Separate digits   
   ea=:&.> NB Perform on each

   sep ea 23 7534 322 7 24756

│2 3│7 5 3 4│3 2 2│7│2 4 7 5 6│

   #ea n NB. How many?

│2│4│3│1│5│

   +/ea n NB. Sum

│5│19│7│7│24│

   */ea n  NB. Product

│6│420│12│7│1680│

   /:~ ea n NB. Sort

│2 3│3 4 5 7│2 2 3│7│2 4 5 6 7│

[deleted]vor 5 Jahren

not_knuthvor 5 Jahren

This looks interesting. Can't wait to give it a listen! Has anyone already done so? Reviews?

kieckerjanvor 5 Jahren

Nice podcast. Definitely made me want to check out APL. Again. (Disclaimer: I go through this phase every couple of years.)

RojerGSvor 5 Jahren

I listened to the first episode and it was really interesting to listen to really experienced programmers share the things they like the most about array programming and the array-oriented languages they are most familiar with.

BlanketLogicvor 5 Jahren

An interesting episode. Recommended.

One of the presenters is a novice with c++ background and the rest all are experts in their respective array programming languages.

rschovor 5 Jahren

It's good. They've got representatives of all the major array language communities among the hosts (APL, J, K/Q) plus a C++ guy.

__vim__vor 5 Jahren

No fortran guy?

rschovor 5 Jahren

I think the goal in bringing in a C++ programmer was to provide an outsider view, but no, no Fortran guy. I don't think Fortran is an array language in that arrays are not the only datastructures in Fortran, though?

Banana699vor 5 Jahren

Fortran isn't an array language at all, really. Maybe some features of the array languages style are super-imposed atop it with parallizing extensions, dialects, or whatnot; but it's style has always been and still is to write explicit loops and mutate state left and right. John Backus in his famous 1977 turing award speech explicitly named it a representative of the "fat weak" languages he talked of and said it was a thin veneer over assembly, he based his fictional FP language on APL instead.

Maybe grandparent meant it has the same _application_ as array languages, in that its only surviving kingdom is scientific computing where devouring gargantuan arrays of numerics is the only thing that matters, unlike C or C++ that are much more widely used. Maybe that's why it has a long history of being parallized with various tools and runtimes _despite_ its inherent imperativeness. (I don't remember where I heard this, but somebody wrote an analyzer to analyze some Fortran programmes in the 90s and found that over 80%/90% of Fortran programmes are spent doing variations of map, filter and reduce. So it's an extremely imperative language against its intended use. Maybe I will post a link in an edit when I find the source of the claim)

Bostonianvor 5 Jahren

Fortran has had built-in array operations since the Fortran 90 standard. If X and Y are scalars or arrays of the same shape, you can X+Y, X*Y, exp(X), sin(X) etc. You can define your own elemental functions that act on both scalars and arrays of any rank. I still write loops when programming in Fortran 95 but less often than in Fortran 77. So I think modern Fortran is an array language.

slvervor 5 Jahren

Is an array language a language where you only have arrays? That honestly sounds a bit odd.

klibertpvor 5 Jahren

No, an array language is one in which all the built-in operations are applicable to arrays. Take addition as an example:

        4 + 4
    8

In array languages the same operator can be used for arrays; or equivalently you can say that the example above sums two arrays of length 1. In J, you can do this and expect it to work:

       4 4 4 + 2 2 2
    6 6 6

This is true for all the built-ins, and many user defined operations (as long as you don't fiddle with so called rank of the verb you're defining).

Numpy is close to that, but there's still a distinction between arrays and scalars, while in array langauges that distinction is often blurred:

       4 + 1 2 3
    5 6 7

Edit: in J, you can have atoms, or scalars, but you need to box them:

       (3;4;5)
    ┌─┬─┬─┐
    │3│4│5│
    └─┴─┴─┘

but then you can't do anything with them until you unbox them again:

       (3;4;5) + 3
    |domain error
    |   (3;4;5)    +3

       (+&4) each (3;4;5)
    ┌─┬─┬─┐
    │7│8│9│
    └─┴─┴─┘

(Examples straight from J REPL)

slvervor 5 Jahren

It seems almost as if it'd be more useful to not explicitly expose operators as applicable to arrays, but implement SIMD optimization for operator expressions in .map(function) and something like .binaryMap(right, function) ex.

    [1, 2, 3].binaryMap([10, 10, 10], (a, b) => a * b) // Produces [10, 20, 30]

This would be easier to optimize when compiled because the expressions can be simplified and mapped to the right SIMD instructions.

klibertpvor 5 Jahren

Unfortunately, I have no idea about the implementation and what optimizations are done in J or APL for which operations :( I know that there are hardcoded "fast path" expressions (particular combinations of operations) which have much better performance than more general expressions doing the same thing, so it might be that the optimization happens at that level.

OTOH, your example is very verbose when compared to J's version:

       1 2 3 * 10
    10 20 30

Plus, in J it generalizes to higher dimensional arrays:

       i. 3 3
    0 1 2
    3 4 5
    6 7 8
       (1 + i. 3 3) * 10
    10 20 30
    40 50 60
    70 80 90

slvervor 5 Jahren

My example is verbose in order to clearly communicate the principle. I can trivially shorten it to

    [1, 2, 3].map(a => a * 10)

if I wanted to literally carry out that task alone.

    [[0, 1, 2],
     [3, 4, 5],
     [6, 7, 8]].flatMap(a => a * 10)

klibertpvor 5 Jahren

I think this is pseudo-code? `flatMap` and `=>` for lambdas look like Scala, but there are no array literals using `[]` there. Assuming you mean Scala-like semantics, your second example wouldn't work at all:

    scala> Array(Array(1,2),
                 Array(4,5)).flatMap(a => a * 10)
                                                       ^
           error: value * is not a member of Array[Int]

You would need to write it like this:

    scala> Array(Array(1,2),
                 Array(4,5)).flatMap(a => a.map(x => x * 10))

But then you'd get a flattened array of ints, not array of arrays of ints:

    res6: Array[Int] = Array(10, 20, 40, 50)

So to get the same result as

    (i. 2 2) * 10

You'd need to write:

    scala> Array(Array(1,2),
                 Array(4,5)).map(a => a.map(x => x * 10))
    res7: Array[Array[Int]] = Array(Array(10, 20), Array(40, 50))

...which is more verbose, no? :) EDIT: obviously, I mean "map in map" part, not the Array initialization!

The problem with list comprehensions and `map`, `filter` and friends is that they work very well for flat lists, or lists of lists in some specific circumstances (ie. when you can use `flatMap`). 2D arrays in the general case, and arrays with higher dimensions are really hard to work with using these primitives, unless you have some clever overloads, like what Clojure does. I think Haskell also has a solution for this, but I don't know it enough to comment further, unfortunately :)

slvervor 5 Jahren

What I can't understand is what would J do if I tell it to sum these two arrays:

    1 2 3 4 5
    6 7 8

I.e. 5 elements and 3 elements

rklmnovor 5 Jahren

Like was mentioned, the simplest behavior is treat this as an error case.

Other possibilities include:

(*) extending the length of the short array to match the length of the long array -- padding with zeros on the right -- before adding.

(*) finding all possible sums between pairs with one number from one array and the other number from the other array.

(*) cyclically extending the shorter array (probably not useful in this example, but since the example didn't come with a use case that's difficult to know for sure).

(*) Treating each array as a decimal number (or a number in some other base)

(*) combining the two array as a single list of numbers and summing everything to a single value

and... so on...

One point being that programming languages support arbitrarily complex operations.

Another point being that "sum" in this context carries some understandable ambiguity -- the sort of thing which we usually try to constrain with use cases or examples or similar elaboration.

klibertpvor 5 Jahren

       1 2 3 + 4 5 6 7
    |length error
    |   1 2 3    +4 5 6 7

In general - depends on the operation. Sometimes the verb checks that both operands have the same rank and dimensions, sometimes the shorter/smaller side gets applied column-wise (or cell-wise/page-wise):

       (2 2 $ 1 2 3 4)  NB. $ means "reshape"
    1 2
    3 4
       (2 2 $ 1 2 3 4) + 1 2
    2 3
    5 6

Sometimes the shorter side is repeated as much as needed to get the correct length, and sometimes the shorter side gets extended with 0s, and sometimes the longer side gets truncated:

       (2 2 $ 1 2 3 4 5 6)
    1 2
    3 4
       (2 4 $ 1 2 3 4 5 6)
    1 2 3 4
    5 6 1 2

To be perfectly honest: it's very unintuitive to me and I have to check the docs pretty often to see how the given verb behaves outside of the simplest case (ie. when the rank and dimensions match). But, I learned J as a hobby and never invested enough time into it to really learn all the built-ins, nor did I try using J outside of toy examples, so maybe this behavior becomes convenient once you internalized it?

EDIT: forgot to mention, you can also control the effective rank of the verb, like this:

       <"0 (2 2 $ 1 2 3 4)
    ┌─┬─┐
    │1│2│
    ├─┼─┤
    │3│4│
    └─┴─┘
       <"1 (2 2 $ 1 2 3 4)
    ┌───┬───┐
    │1 2│3 4│
    └───┴───┘
       <"2 (2 2 $ 1 2 3 4)
    ┌───┐
    │1 2│
    │3 4│
    └───┘

the `<` verb means "box", without the `"n` part you'd get the last value, which is a 1-dimensional array of boxes of length 1 (with 2x2 array inside the box). By selecting rank of the verb, you can decide on which level the verb should be applied: with `"0` it's applied to the smallest possible chunks of the array (cells - you get 2x2 array of boxes), with `"1` you apply the verb to bigger chunks (rows), and so on, for input arrays of any dimensions (so if you have a 3x2x2 array, `"2` will apply the verb to the 3 2x2 arrays).

rschovor 5 Jahren

even less verbose:

(>: i. 3 3) * 10

rschovor 5 Jahren

...or

10 * >: i. 3 3

FWIW

dzaimavor 5 Jahren

That just removes a tiny tiny bit of dynamic dispatch overhead. Which is still needed anyways, as array languages can often dynamically switch between 1-bit, 8-bit, 16-bit, 32-bit integer (and 64-bit float) arrays, depending on the elements, completely transparently to the user.

slvervor 5 Jahren

Most compilers can inline statically defined closures in these contexts. And tracing JITs do this even when the closure is not define statically (but is stable).

It's more about allowing the SIMD goodness without the ambiguity and restrictions of "scalar operators work on arrays" implemented naively.

klibertpvor 5 Jahren

> without the ambiguity

There's no ambiguity! In J, all (with some exceptions) operations work on arrays, period. `1 + 1` is not an addition of two scalars, but instead of two arrays of length 1. As I mentioned, you can have scalars, but a) they still live in arrays and b) you can't do anything with them without unboxing. So there's no ambiguity, as far as I can tell.

Also, APL and J are implemented as intepreters, but that doesn't preclude using SIMD instructions when executing expressions. I'm 100% sure the operations are not "implemented naively" :)

rschovor 5 Jahren

> `1 + 1` is not an addition of two scalars, but instead of two arrays of length 1

I think that's true of (Dyalog) APL, but not of J [1]. APL follows the nested array model (that you describe), while J follows the flat array model that does have true scalars.

[1] https://aplwiki.com/wiki/Array_model#Flat_array_theory

dzaimavor 5 Jahren

AFAIK, J considers `1` to be an array - `L. 1` and `L. 1 2` both give 0 (`L. (1;2)` gives 1).

APL's simple scalar numbers are much more like regular numbers in non-array languages.

rschovor 5 Jahren

Traditional array languages (APL,J) are not amenable to static compilation due to other dynamic issues, though. I think Dyalog APL is experimenting with a bytecode interpreter, but no JITs in sight either (that I know of). The very dynamic aspect of those languages makes that difficult. I'd love a compileable similar language, though. To replace R/Python for statistics that are sooooo annoying when exploring data due to verbosity.

rklmnovor 5 Jahren

That's not really the issue.

It's certainly true that, depending on how the code is written, you may not be able to optimize out the language implementation from the compiled program. But that just turns into a link against a library with support for the language for your hypothetical compiled program.

That said, the compiler being hypothetical is a very real obstacle. But even there the problem is not that the language can't be compiled -- it's that no one has bothered to implement a compiler for it.

Anyways, if you throw in some ML type inference, and some tools for characterizing the resulting code and its interfaces, you can generate code from an array language which is quite similar to the code you would get from a variety of other languages.

dzaimavor 5 Jahren

A compiler could inline SIMD with or without an operation having an explicit map, the dynamic type checks needed are gonna be the same. (and, since this is an array language, the tiny extra overhead of completely dynamic dispatch is gonna be small compared to the executed SIMD afterwards anyways)

You lose a lot of simplicity & brevity by requiring explicit mapping, and gain close to nothing.

mlochbaumvor 5 Jahren

That's the case in APL and J. K uses nested lists to represent arrays, and has non-lists (atoms). But the convention is that an n-times nested list is considered an n-dimensional array so even an atom is an array, with 0 dimensions.

There's a page on the various approaches to arrays in the APL family at https://aplwiki.com/wiki/Array_model .

rklmnovor 5 Jahren

Mostly yes.

Numbers (and character) are implemented as arrays with 0 dimensions. Text would be an array of characters with 1 dimension (the number of characters), and generally speaking the dimension of an array is a one dimensional list of non-negative integers. Many array languages also include an array type which is approximately the same as a C pointer to an array, with a bit of jargon thrown in to distinguish a reference to an array from the array itself.

Something like an SQL table in an array language would be implemented as a list of columns (rather than as a list of rows) and a corresponding list of column labels. This has some interesting benefits.

That said, functions in array language are typically not arrays (though they presumably would have a textual representation). So... not everything is an array.

rschovor 5 Jahren

From the J docs: https://www.jsoftware.com/help/dictionary/dx005.htm

  nub=: (i.@# = i.~) # ]
     5!:2 <'nub'
  +-------------------+-+-+
  |+--------+-+------+|#|]|
  ||+--+-+-+|=|+--+-+|| | |
  |||i.|@|#|| ||i.|~||| | |
  ||+--+-+-+| |+--+-+|| | |
  |+--------+-+------+| | |
  +-------------------+-+-+

... so J functions have an array representation, at least.

rklmnovor 5 Jahren

Yes, each J function has several array representations.

mjaniczekvor 5 Jahren

You could consider Pandas / numpy an array language, even though it's really a library for Python. The exposed API is IMHO what's important.

Wezlvor 5 Jahren

I agree, but you have to take extensibility into account. You either have an extensible language and extend it[0], you make it an array language to the core, or you just have a few slightly convenient functions (not a language). (I don't have enough experience to judge which category Pandas / numpy are in)

[0]: https://github.com/phantomics/april

mjburgessvor 5 Jahren

Surprising highly production values -- programming podcasts are typically extremely poor auido with very little prep going into to them.

slvervor 5 Jahren

These days having good video and audio quality doesn't require a lot of money. It just requires couple of hours educating yourself online what you need and how to use it.

nulldatavor 5 Jahren

Great first episode. I'd love to hear Aaron Hsu on this!

FemmeAndroidvor 5 Jahren

This is very exciting. I’ve been really inspired by the passion Conor Hoekstra has for APL and J. His other podcast (Algorithms + Data Structures = Programming) is a lot of fun, and his YouTube videos are very educational, but I’ve really wanted something like this where he can interact with other experts outside of the C++ world.

Instant subscribe from me.

chewxyvor 5 Jahren

this seems like it's very very niche (and very specific to the niche that I am interested in!)

bidirectionalvor 5 Jahren

True, but I also feel like array programming is seen as less prevalent in industry than it really is due to a lack of online community (e.g. Haskell as a language probably has an order of magnitude more content online, let alone the functional paradigm).

In any financial centre there are hundreds of (often very well-paying) jobs using these languages (well mainly k and its ilk).

an1sotropyvor 5 Jahren

I had thought of APL as something from computing pre-history, with its bizarro custom keyboard, but I learned that APL and other array languages are apparently alive and well. Will subscribe to the podcast.

Two quotes the hosts brought up stuck with me:

(at 15:05) "A language that doesn't change the way you think is not a language worth learning". From Alan Perlis [1], and his Epigrams in Programming (#19) [2]

(at 16:49) "it is a privilege to learn a language/ a journey into the immediate". From poet Marilyn Hacker [3]; totally captivating idea, even if not not about programming languages [4]

[1] https://en.wikipedia.org/wiki/Alan_Perlis [2] https://cpsc.yale.edu/epigrams-programming [3] https://poets.org/academy-american-poets/winner/prizes/james... [4] https://www.enotes.com/topics/marilyn-hacker/critical-essays

bidirectionalvor 5 Jahren

k (and the closely related q) is the main language used in industry, particularly at investment banks and hedge funds. It can be a bit of a shock to realise there are people in London earning in excess of £1000/day (pretty good for London) working in a language where well-written code looks like this[1]:

  us:{$[#i:&{(y~*K)&"*"~\*x}':x;@[x;i;:[;,"_"]];x]}

It's like discovering a whole different world of software development. Also I don't use that example to disparage k, I have come to appreciate the array language way-of-working. It just looks very alien.

[1] Real example found in a random script on https://nsl.com/: http://nsl.com/k9/sql.k

jiggawattsvor 5 Jahren

What the actual f?

It's like someone threw up the noise that modems make during initial connection onto an electric typewriter from the 1960s, and then explained their intention using quotes from a Lovecraft novel.

geocarvor 5 Jahren

I am going to tell you something fantastic, but first, I want to explain some things about this:

    us:{$[#i:&{(y~*K)&"*"~*x}':x;@[x;i;:[;,"_"]];x]}

The first is that there's a typo in what bidirectional wrote. The above is correct. The second, is what it is. Once I have explained that, I can tell you the fantastic thing.

k syntax is very simple. There's just a few forms you need to be aware of:

f x

which applies x to f.

    a f b

which is apply f to the two arguments a and b, and:

    f[a;b;c]

which allows you to do three arguments. You can write the first one as f[x] and the second as f[a;b] if you like even more consistency. f can be an "operator" -- that is a symbol. The symbols ' / and \ are special and called adverbs. These adverbs have a special form if followed by a colon, so ': is different than ' and has nothing to do with : or '. I think Arthur just ran out of keys on the keyboard. Once you have those, parenthesis () and braces {} have some special syntax, just like double-quotes " do.

With the syntax explained, let us try to understand what we are looking at.

us: is how we start assignment. You can say "us gets" if you like (the colon can be pronounced). {} braces surround a lambda, this one takes a single argument "x" (the first argument). $[a;b;c] is cond like in lisp; if a then b else c. # means count. i: is another assignment.

& means where -- the argument to which is going to be a bitmap like 000100b or 01101b or something like that, and where returns the indices of the set bits; the former example being the list 3, the latter example being the three-element list 1 2 4.

Another lambda comes next: We can see it takes two arguments because there's an x and a y in there (y is the second argument). We can get a clue as to what it expects because the following adverb ': means each-prior. This tells us "x" is going to be a list of things, and this lambda is going to consume them pairwise. If given the list {(x;y)}':"iliketacos" we get the result:

     {(x;y)}':"iliketacos"
    i 
    li
    il
    ki
    ek
    te
    at
    ca
    oc
    so

y is the "previous" value, and "x" is the current value. The "where" before it tells us we want to know the indices where the condition inside is true. Let's try and understand that condition.

y~*K is in parenthesis. Parenthesis group, so we execute them first (just like in other languages). We're looking for a situation where the previous value is the first (that's what asterisk means here) of K. What is K?

    K:("select";"distinct";"partition";"from";"where";"group";"having";"order";"limit")

So we're looking for a value (x) whose previous (y) is the first of K which is "select". The "&" that follows here is "and" - Arthur likes to overload operators since there aren't many symbols on the keyboard and this is something you get used to.

So you can read {(y~*K)&"*"~*x}':x as simply trying to find the sequences "select star" -- given a list ("select"; "*"; "from"; "potato") you get 0100b and from ("select"; "*"; "from"; "("; "select"; "*"; "from"; "potato; ")") you get 01000100b. I think the attempt is to disambiguate the asterisks in the sql:

    select * from tacos where cat=4*42

but sql is a strange and irregular language, so this kind of thing is necessary. Back to our query:

    us:{$[#i:&{(y~*K)&"*"~*x}':x;@[x;i;:[;,"_"]];x]}

i is going to be the locations of the asterisks following select. If the count of that is nonzero; we're going to do the @-part, and if not, we're just going to return x.

    @[x;i;f]

is called amend. It returns x, but at indices i, we apply them to f, so it's x[i]:f[x[i]] which is pretty cool. f in this case is a projection, of "gets" (the function colon) with the second-argument bound to an underscore. That is:

    :[;,"_"]

is just a function. That's how @[x;i;:[;,"_"] replaces all of the asterisks that follow select with an a "_"

Almost. It's actually a list of length one, rather than the scalar "_". I haven't read everything in sql.k but this is probably important elsewhere.

Ok. Now that I have explained what this is and what it does, I am ready to tell you something fantastic. I read this:

    us:{$[#i:&{(y~*K)&"*"~*x}':x;@[x;i;:[;,"_"]];x]}

as:

"us gets a function, that finds the indices of asterisk following the first element of K, and then replaces the things at those indices with underscores"

Literally. From left, to right. Just that fast. And I only program in k part-time. That's not the fantastic thing. The fantastic thing is that by learning to read k, I am almost miraculously able to read other languages faster. This:

    copied=False
    for i in range(1,len(a)):
      if a[i] == "*" and a[i-1] == "select":
        if not copied:
          copied = True
          a = a[:]
        a[i] = "_";

gives me some grief for being so irregular and gross, and I have to look up range/xrange and len and memorise a much more complex set of rules for syntax, and I have to track the order of things carefully and so on, but I have places in my brain, made by k, for those things, and so I am able to absorb code in other languages faster.

If that does not amaze you, I do not think you have considered the ramifications of what I said. I can suggest maybe reading it again (or maybe actually reading what I wrote instead of skipping to the punchline), but if after two or three tries you are still lost, maybe you can ask a question and I can try to answer it.

leephillipsvor 5 Jahren

Is the open source version (Kona) any good? Or do we have to go proprietary to try this out?

ithkuilvor 5 Jahren

I had much fun learning the klong language: https://t3x.org/klong/

I even bought the well written book, which was a pleasure to read.

RodgerTheGreatvor 5 Jahren

ngn-k is a different dialect from Kona, but actively-maintained and open source: https://codeberg.org/ngn/k

oK (which is also k6) can be tried here: http://johnearnest.github.io/ok/index.html

specialistvor 5 Jahren

Wow. That's some explanation. Thanks.

Do you use "rainbow brackets"?

This example is first hit I found: https://kristofferc.github.io/OhMyREPL.jl/latest/features/ra...

Such an obvious idea once you see it. Wish I had syntax coloring and rainbow brackets when I coded LISP for hire.

geocarvor 5 Jahren

> Wow. That's some explanation. Thanks.

Happy to help.

> Do you use "rainbow brackets"?

No. I flash brackets, but I tend to turn off other forms of syntax highlighting. I find it extremely distracting when the syntax highlighter "decides" wrong, and I've become convinced comments at the end of lines like //} to "fix" the highlighter deal with complicated stuff is hurting more than it's helping.

smirvor 5 Jahren

In J language single sided brackets are functions/verbs

kazinatorvor 5 Jahren

TXR Lisp:

  1> (defun rewrite (fun list)
      (build
        (while* list
          (let ((nlist [fun list]))
            (if (eq list nlist)
              (if list (add (pop list)))
              (set list nlist))))))
  rewrite
  2> (defmacro rewrite-case (sym list . cases)
      ^(rewrite (lambda (,sym)
                  (match-case ,sym
                    ,*cases))
                ,list))
  rewrite-case
  3> (rewrite-case x '(foo bar * select * fox select * bravo)
       ((select * . @rest) ^(select _ . ,rest))
       (@else else))
   (foo bar * select _ fox select _ bravo)

rewrite-case and rewrite appear in the TXR Lisp internals; they are used in the compiler for scanning instruction sequences for patterns and rewriting them.

E.g. a function early-peephole looks for one particular four instruction pattern (six items, when the labels are included). Rewriting it to a different form helps it disappear later on.

  (defun early-peephole (code)
    (rewrite-case insns code
      (((mov (t @t1) (d @d1))
        (jmp @lab2)
        @(symbolp @lab1)
        (mov (t @t1) (t 0))
        @lab2
        (ifq (t @t1) (t 0) @lab3)
        . @rest)
      ^((mov (t ,t1) (d ,d1))
        (jmp ,lab3)
        ,lab1
        (mov (t ,t1) (t 0))
        ,lab2
        ,*rest))
      (@else else)))

This is much more general and powerful than a hack which just looks at successive pairs for an ad-hoc match. rewrite-case can have multiple clauses, of different lengths, and arbitrary matching complexity.

kazinatorvor 5 Jahren

The original requirements should be addressed. The thing being matched is not just select, but actually any one of a set of symbols that appear in K.

We can stick that data into the pattern matching syntax using the or operator:

  3> (rewrite-case x '(foo bar * select * fox where * bravo)
       ((@(or select distinct partition from where
              group having order limit) * . @rest) ^(,(car x) _ . ,rest))
       (@else else))
  (foo bar * select _ fox where _ bravo)

Or put it into a variable:

  4> (defvarl K '(select distinct partition from where group having order limit))
  K
  5> (rewrite-case x '(foo bar * select * fox where * bravo)
       ((@(member @sym K) * . @rest) ^(,sym _ . ,rest))
       (@else else))
  (foo bar * select _ fox where _ bravo)

"If an object sym which is a member of K is followed by * and some remaining material, replace that by sym, underscore and that remaining material."

Hash table:

  6> (set K (hash-list '(select distinct partition from where group having order limit)))
  #H(() (select select) (where where) (limit limit) (order order)
     (having having) (distinct distinct) (partition partition) (group group)
     (from from))
  7> (rewrite-case x '(foo bar * select * fox where * bravo)
       ((@[K @sym] * . @rest) ^(,sym _ . ,rest))
       (@else else))
  (foo bar * select _ fox where _ bravo)

This code golfs moderately well: https://news.ycombinator.com/item?id=27227276

kazinatorvor 5 Jahren

Let's do this using the same approach, "find indices and assign over them in a copy of the sequence":

  This is the TXR Lisp interactive listener of TXR 259.
  Quit with :quit or Ctrl-D on an empty line. Ctrl-X ? for cheatsheet.
  Do not operate heavy equipment or motor vehicles while using TXR.
  1> (defun subst-select-* (list)
        (let ((indices (where (op starts-with '(select *))
                              (cons nil (conses list)))))
          (if indices
            (let ((list (copy list)))
              (set [list indices] (repeat '(_)))
              list)
            list)))
  subst-select-*
  2> (subst-select-* '(foo bar * select * fox select * bravo))
  (foo bar * select _ fox select _ bravo)

(conses list) gives us a list of the list's conses: e.g in (1 2 3) the conses are (1 2 3), (2 3) and (3), so the list of them is ((1 2 3) (2 3) (3)).

where applies a function to a sequence, and returns the 0-based indices of where the function yields true.

(op starts-with '(select *)) yields a lambda which tests whether its argument starts with (select *). No brainer.

If we naively applied that to the conses, we would get thew rong indices: the indices of the select symbols, not of the asterisks.

The workaround for that is (cons nil (conses list)): we cons an extra dummy nil element to shift the positions, and process the resulting list.

Once we have the indices list, if it isn't empty, we copy the original input, and assign underscore symbols into the indicated positions. To do that we generate an infinite lazy list of underscores; the assignment takes elements from this list and puts them into the specified index positions.

geocarvor 5 Jahren

If this were me, and I needed a function like us, I would have written this:

    us:{$[x~(z;y);,"_";y]}[(*K;,"*")]':

I would be interested in seeing anything that was shorter[1] and faster than that in any language, and I would be very curious to learn from anyone who could also do that faster than me.

But I'm not a fetishist: I didn't learn k because it was cute, and I don't wake up every day looking for ways to rewrite other people's code so that it is slower and bigger. Do you? Or is today special?

[1]: Measured in source-code bytes.

rschovor 5 Jahren

> Do you? Or is today special?

Today is the usual. He does that in all threads related to array languages.

kazinatorvor 5 Jahren

> I would be interested in seeing anything that was shorter

One way to do that would be to pose that as a problem on the Code Golf Stackexchange.

My rewrite-case solution condenses (by removal of all non-essential spaces) to 69 bytes if the symbols are in a hash table K, and the input list is in a variable y, rather than a big literal:

  (rewrite-case x y((@[K @sym]* . @rest)^(,sym _ . ,rest))(@else else))

A one-letter name could be chosen for the macro. Furthermore, one-letter names could be chosen for sym, rest and else:

  20> (r x y((@[K @s] * . @r)^(,s _ . ,r))(@e e))
  (foo bar * select _ fox where _ bravo)

Still working! Now down to 43 bytes.

(It's not because I cannot that I do not do this with all my code.)

Doh, why use . ,r in the backquote if we are golfing? That should be ,*r: using the splice operator, like Common Lisp's or Scheme's ,@, getting us to 42 bytes:

  20> (r x y((@[K @s] * . @r)^(,s _ ,*r))(@e e))
  (foo bar * select _ fox where _ bravo)

I suspect Code Golf Stackechange regulars could get it down to way in one of the dedicated golfing languages like Retina or what have you.*

What else can we do? The @[K @s] predicate syntax could be replaced by a custom operator defined by defmatch, looking like @(K s).

  21> (defmatch k (sym) ^@[K (sys:var ,sym)])
  k
  22> (r x y((@(k s) * . @r)^(,s _ ,*r))(@e e)) ;; 41
  (foo bar * select _ fox where _ bravo)

Just noticed the ) * is not fully golfed:

  23> (r x y((@(k s)* . @r)^(,s _ ,*r))(@e e)) ;; 40
  (foo bar * select _ fox where _ bravo)

The k pattern operator macro could be non-hygienic: it could implicitly bind a variable called s:

  24> (defmatch k () ^@[K @s])
  k
  25> (r x y((@(k)* . @r)^(,s _ ,*r))(@e e)) ;; 38
  (foo bar * select _ fox where _ bravo)

These last few feel like cheating because they are too special purpose. Defining anything you can only possibly use just once isn't making the overall program smaller.

kazinatorvor 5 Jahren

Also-rans:

  1> [window-map 1()(do if(and[K @1](eq'* @2))'_ @2)y]
  (foo bar * select _ fox where _ bravo)
  2> (mapcar(do if(and[K @1](eq'* @2))'_ @2)(cons()y)y)
  (foo bar * select _ fox where _ bravo)
  3> (mapcar(lambda(:match)((@[K] *)'_)((@a @b)b))(cons()y)y)
  (foo bar * select _ fox where _ bravo)

rschovor 5 Jahren

you'll never get J/K brevity though. Because in opposition to Lisp where control flow is explicit in the syntax, most of the control flow in array languages is implicit. I love lisp too, but for some (rare) use cases it's not the best.

kazinatorvor 5 Jahren

You will never get J/K brevity by writing a condensed for of normal Lisp because spaces are often required to separate tokens.

You can get implicit control flow via macros.

Furthermore, all control flow is implicit is some way.

Take basic old progn: (progn (foo) (bar)) evaluates (foo) and then control implicitly passes to (bar) because that's the next thing.

The only control flow which is not implicit is that of a state machine described by a table or graph, indicating the next state for every input and current state.

(How can you say you can't get implicit control flow in Lisp, when I'm using pattern matching to express if you see this prefix, replace it with that?)

Speaking of macros, you could always resort to a macro like this:

  (j"... line noise in character string ...")

The j macro expands the string to code at compile time. The code could be a bona fide J implementation, or something like it. That's still not down to J, because of the (j"") wrapping chewing up five characters. In a Lisp with a programmable read table, that could be reduced to two character framing or something.

Avshalomvor 5 Jahren

okay so it's not really smaller because it uses UTF8 also it relies on order of operations to get rid of the parens around ⍵≡'*' but

us←{('_'@(1+⍸2{(⍺≡⊃K)∧⍵≡'*'}/⍵))⍵}

is one(1) character shorter.

rak1507vor 5 Jahren

⍵≡'*' should be ⍵≡,'*' like in the original k

us←⊃,⍥⊆2{(⍺≡⊃K)∧⍵≡,'*':,'_'⋄⍵}/⊢ is shorter anyway

geocarvor 5 Jahren

    us←{'_'@{¯1⌽⍵⍷⍨(⊂⊃K),'*'}⍵}

rak1507vor 5 Jahren

Ah, ⍷, clever, don't even need the outer {...⍵},

  us←'_'@{¯1⌽⍵⍷⍨(⊂⊃K),'*'}

geocarvor 5 Jahren

Yes!

You could also do: ⌽'*',⊂⊃K to save another character. I don't know if that's "better" though.

rak1507vor 5 Jahren

Or alternatively '*',⍨⊂⊃K which I think I prefer over ⌽, but neither seem that 'better' in my mind anyway.

wanderer2323vor 5 Jahren

Is this the same language or a different one?

rak1507vor 5 Jahren

This is APL, k is a descendant of APL.

rak1507vor 5 Jahren

Why do you pass in (*K;,"*") as x instead of hard-coding it in x~(z;y)? Clarity?

geocarvor 5 Jahren

Great question!

     v:("select";,"*";"from";"tacos")

     us:{$[#i:&{(y~*K)&"*"~*x}':x;@[x;i;:[;,"_"]];x]}              
     \t:100000 us v
    233
     us:{$[(*K;,"*")~(y;x);,"_";x]}':
     \t:100000 us v
    206
     w:(*K;,"*");us:{$[w~(y;x);,"_";x]}':
     \t:100000 us v
    179
     us:{$[x~(z;y);,"_";y]}[(*K;,"*")]':
     \t:100000 us v
    129

I think it's important to remember just how simple k is: We the programmer know that (*K;,"*") isn't supposed to change, so we should be explicit so k "knows" this as well. In addition to never changing, we also know the value is only used once, so we really don't want to look anything up in the workspace every time we call us: Again, let us be explicit.

On the other hand, this:

     us:{$[#i:& XXX ;@[x;i;:[; YYY ]];x]}

is an extremely recognisable idiom. It occurs several times in sql.k so the reader is probably used to seeing it at this point. It also has a similar syntactic structure to the APL '@' so it may be more "obvious" to an APL programmer. Maybe the reason I write it this way is that I'm not a very experienced APL programmer :)

rak1507vor 5 Jahren

Thank you for the extensive answer, that's very interesting that there's such a performance difference. I would have naively assumed if anything keeping it as a constant would be faster.

Another question: Why do

  $[#i:& XXX ; @[x;i;:[; YYY]]; x]

when (if I'm not mistaken, which I could be) you can do

  @[x;i:& XXX;:[; YYY]]

Performance again? A quick test using ngn/k did seem to find the first one faster, but only marginally.

geocarvor 5 Jahren

That's a harder question. Why do other people do the things they do? One reason might be because it was convenient for them to write (and debug) things that way.

But it is possible they are doing it for performance reasons, so maybe it's worth considering why it should be faster? That is to say, should the programmer assume the latter is faster than the former?

To do that, I would suggest putting yourself in the shoes of the implementor: You have a choice of how it should be implemented. Knowing this decision will affect every use of @[x;i;f], should it begin with an if statement like this?

    if(y->n == 0)return x;

But before you answer, remember the next thing @[x;i;f] needs to do, is consider if x has any additional references, because if it does, it has make a copy of x. That means there already needs to be an if statement that looks like this:

    if(x->r > 0) x = copy(x);

So one way to think about this, is to ask if you are implementing @[x;i;f], should you choose one branch or two?

    $[#i:& XXX ; @[x;i;:[; YYY]]; x]

Also: Do you see the part that goes :[;YYY]? That's constructing an object. Do you think it is worth avoiding that allocation if possible?

rak1507vor 5 Jahren

Ok, that all makes sense, thank you.

And thanks for pointing out the pattern in the first place, it makes sense now how you were able to parse the original expression so quickly.

kazinatorvor 5 Jahren

> I have to look up range/xrange and len and memorise

All you're communicating here is that you don't regularly work with Python.

The claim that you have to look up len seems disingenuous; I might believe it if you didn't look like a speaker of English.

> & means where

And that's something any engineer would know, unlike having to look up what len means?

geocarvor 5 Jahren

> The claim that you have to look up len seems disingenuous

In what way?

I have to look up "how do I get the indices of a list" to get range(1,len(x)) -- I think I could have also used enumerate() and a bunch of other things, but this seemed the shortest.

> All you're communicating here is that you don't regularly work with Python.

I hope I'm communicating more than that because I put a lot of effort into my comment. I don't regularly work with k either.

What exactly do you think you are communicating?

kazinatorvor 5 Jahren

What I mean by that is that len is an obvious abbreviation for length that crops up in numerous languages. Not to mention code bases. It's simply not plausible you could forget what len means in Python after confirming one time that it is exactly what it looks like. (Unless you're a struggling non-native speaker of English forgetting the word length itself.)

jiggawattsvor 5 Jahren

To me, this is infinitely more readable:

    let mut input = ["select", "*", "from", "potato"];
    for i in 1..input.len() {
        if ["select","*"] == input[i-1..=i] {
            input[i] = "_";
        }
    }

If I could be bothered to dig up a Haskell compiler, I'm sure it's possible to do a one-liner list comprehension that is both terse and readable.

If I was doing this in Rust, it's easy to create an iterator extension that does something like "map_lookback" which explains the intention without requiring comments.

I'm going to be blunt: Unreadable array languages are popular with quants because they work in a highly competitive, cut-throat industry where "write only" languages provide job security. I've come across developers purposefully obfuscating code by using only one-character identifiers and zero comments. One of them literally blackmailed their employer, demanding their salary be doubled on the grounds that there was no chance their code could be maintained by anyone else. He should have gone to jail for that, but he had his managers by the balls, got what he wanted, and bought a house with cash soon after.

Why are we applauding this?

geocarvor 5 Jahren

> To me, this is infinitely more readable:

Yes, to you. Just like Chinese would be infinitely less readable to you, if you don't know Chinese.

Do you think this is a deep observation?

The real challenge is knowing whether it is worth it to learn k so that it becomes readable.

- How many characters is it? This is a useful metric when you realise bug/defect rate is proportional to the physical size (in rows and columns of source code) of a program given equivalent processes (Moore 1992; McConnell 1993) but that process matters more than anything else. Not tooling, not "memory safety", and certainly not "readability" by people unfamiliar with the language.

However "readable" you think your rust code is, did you notice the bug in your rust code? (hint: input is not supposed to be mutable)

- How fast is it? On my 2014 i7 macbook air, the supplied us averages 232 msec for 100k cycles. My version

    us:{$[x~(z;y);,"_";y]}[(*K;,"*")]':

is faster: 126msec for 100k cycles.

- How quickly was it written? I can't speak for sa/atw on us since I didn't see them write it. I wrote mine in about 30 seconds including testing. Yes really. How long did your rust program take to write? Did you think simply because you thought you understood the requirements that you didn't need to test it?

- How quickly can someone familiar with the language read it? Again, I can't speak for anyone other than myself, but I'm not a k expert -- I program in it very infrequently, and the new amend-syntax in k9 I had not run across previously. And yet I read it as quickly as I said.

These four values (less code, fast run, fast write, fast read) are the biggest most important things to me. And anyone who shows me all four of things will get my attention.

Rust? Simply does not impress.

> Unreadable array languages are popular with quants because they work in a highly competitive, cut-throat industry where "write only" languages provide job security.

That's another interesting opinion. This one might even be true amongst some quants (Most of the ones I know that use k don't particularly like k). But I suggest you try not to expect the worst in people. Yes, some people are assholes, but most people aren't. And for what it's worth, I'm not a quant (I work in Advertising).

jiggawattsvor 5 Jahren

> 126msec for 100k cycles.

Or to put it another way: 1,200 nanoseconds. That's about 3,000-5,000 instructions on a modern CPU. Believe it or not, that's actually pretty bad.

After jumping through some hoops to ensure that rustc doesn't just compile the whole thing down to a constant, I benchmarked my version as taking 15-20 nanoseconds per iteration. About 45-80 instructions!

I actually couldn't quite believe it myself, so I jumped through more hoops to ensure that it wasn't being optimised away, wasn't getting inlined too aggressively, etc... No change.

Ran it through Godbolt to inspect the assembly, and then I realised that, yes, modern languages, compilers, and CPUs really are this good!

Think about it: For the specific input example with 4 strings the algorithm boils down to: compare 7 bytes with 7 bytes, replace a pointer with another pointer, then compare 2 bytes with 2 bytes three times. That's about a hundred assembly instructions, or thereabouts.

Godbolt output:

    push   rbp
    push   r15
    push   r14
    push   r13
    push   r12
    push   rbx
    push   rax
    mov    r12,rdi
    mov    ebx,0x8
    xor    ebp,ebp
    lea    r14,[rip+0x3cb3c]        # 444e8 <anon.6527da4acb4810bb73692fa85a2e25ef.0.llvm.5506334730328533235+0x20>
    mov    r15,QWORD PTR [rip+0x3f3b5]        # 46d68 <bcmp@GLIBC_2.2.5>
    cs nop WORD PTR [rax+rax*1+0x0]
    nop    DWORD PTR [rax]
    cmp    rbp,0x2
    je     79f2 <example::testabc+0x62>
    mov    r13,rbp
    mov    rdx,QWORD PTR [rbx+r14*1]
    cmp    rdx,QWORD PTR [r12+rbx*1]
    jne    79ec <example::testabc+0x5c>
    lea    rbp,[r13+0x1]
    mov    rsi,QWORD PTR [r12+rbx*1-0x8]
    mov    rdi,QWORD PTR [rbx+r14*1-0x8]
    call   r15
    add    rbx,0x10
    test   eax,eax
    je     79c0 <example::testabc+0x30>
    cmp    r13,0x2
    jb     7a07 <example::testabc+0x77>
    lea    rax,[rip+0x3060e]        # 38007 <_fini+0xd13>
    mov    QWORD PTR [r12+0x10],rax
    mov    QWORD PTR [r12+0x18],0x1
    xor    ebx,ebx
    xor    ebp,ebp
    nop    DWORD PTR [rax+rax*1+0x0]
    cmp    rbp,0x2
    je     7a43 <example::testabc+0xb3>
    mov    r13,rbp
    mov    rdx,QWORD PTR [rbx+r14*1+0x8]
    cmp    rdx,QWORD PTR [r12+rbx*1+0x18]
    jne    7a3d <example::testabc+0xad>
    lea    rbp,[r13+0x1]
    mov    rsi,QWORD PTR [r12+rbx*1+0x10]
    mov    rdi,QWORD PTR [rbx+r14*1]
    call   r15
    add    rbx,0x10
    test   eax,eax
    je     7a10 <example::testabc+0x80>
    cmp    r13,0x2
    jb     7a58 <example::testabc+0xc8>
    lea    rax,[rip+0x305bd]        # 38007 <_fini+0xd13>
    mov    QWORD PTR [r12+0x20],rax
    mov    QWORD PTR [r12+0x28],0x1
    mov    ebx,0x8
    xor    ebp,ebp
    nop
    cmp    rbp,0x2
    je     7a93 <example::testabc+0x103>
    mov    r13,rbp
    mov    rdx,QWORD PTR [rbx+r14*1]
    cmp    rdx,QWORD PTR [r12+rbx*1+0x20]
    jne    7a8d <example::testabc+0xfd>
    lea    rbp,[r13+0x1]
    mov    rsi,QWORD PTR [r12+rbx*1+0x18]
    mov    rdi,QWORD PTR [rbx+r14*1-0x8]
    call   r15
    add    rbx,0x10
    test   eax,eax
    je     7a60 <example::testabc+0xd0>
    cmp    r13,0x2
    jb     7aa8 <example::testabc+0x118>
    lea    rax,[rip+0x3056d]        # 38007 <_fini+0xd13>
    mov    QWORD PTR [r12+0x30],rax
    mov    QWORD PTR [r12+0x38],0x1
    add    rsp,0x8
    pop    rbx
    pop    r12
    pop    r13
    pop    r14
    pop    r15
    pop    rbp
    ret    
    nop    WORD PTR [rax+rax*1+0x0]

Rust? It simply impresses.

geocarvor 5 Jahren

I'm not sure getting the wrong answer fast is something to be proud of.

Your rust code has a bug in it, is longer, and you spent more time on it.

jiggawattsvor 5 Jahren

Please elaborate on that bug. The rust function I provided replaces "select, *" correctly with "select, _". Is that not what it was supposed to do? I checked that it works with empty arrays, arrays with one entry, etc...

Thinking about it, this version boils down the logic to its barest essence:

    fn test(input: &mut [&str]) {
    
        let mut was_select = false;
        for i in input.iter_mut() {
            if was_select && *i == "*" { *i = "_"; was_select = false; }
            else { was_select = *i == "select"; }
        }
    }

It runs in 5.5 nanoseconds per iteration, which is just absurdly fast. That's about 200x faster than your K version!

What I like about this Rust version is that it reflects the approach that for a computer processor is optimal, yet it is high-level and readable. I could convert that "test" function easily enough to a generic "replace" function similar to a string search & replace, but one that can operate on any mutable iterator with the maximum possible efficiency, not just arrays.

Then, your K code that requires "careful reading" to slowly tease apart the meaning could be converted to a form that is practically prose:

    let mut input = ["select", "*", "bar", "potato"];
    replace( &mut input, &["select", "*"], &["select", "_"] );*

geocarvor 5 Jahren

> The rust function I provided replaces "select, *" correctly with "select, _". Is that not what it was supposed to do?

No. It is not supposed to modify its input but return a copy.

jiggawattsvor 5 Jahren

Granted, if replacing arbitrary sequences with arbitrary sequences, then copying is necessary, as this could result in the output length increasing.

However, for replacing scalars only, the in-place mutable version is in some sense superior: it doesn't force a memory allocation. Moreover it can be trivially converted into a copying version by simply wrapping it in a function that first copies the input, and then mutates it in-place. The reverse is not true: the copying version cannot be wrapped to create a non-allocating version.

To be honest, I wish more languages put this kind of effort into their standard libraries, but most have about 10% of what I would like to see. For example, this kind of "string matching" is really "sequence matching" and ought to be fully generic for any underlying comparable and copyable type, not just arrays of characters. String search algorithms like Boyer-Moore ought to be directly applicable to arrays of integers or enums, e.g.: for recognition of code patterns in lists of parsed tokens.

At least one person has done something like this for Rust: https://github.com/peterjoel/rust-iter-replace/blob/6c575eeb...

Similarly, there's the new InPlaceIterable, which is interesting but not quite enough to suit my taste: https://doc.rust-lang.org/std/iter/trait.InPlaceIterable.htm...

geocarvor 5 Jahren

> However, for replacing scalars only, the in-place mutable version is in some sense superior

I mean, we are talking about someone else's code and someone else's decisions. If we can change the rules, you're absolutely right we can do much much better.

But beware microbenchmarking too much: Giving the treatment you gave your rust to your entire program can be more than exhausting, it can actually end you up with a slower program simply because your program gets too big!

k makes a lot of compromises to stay small enough to keep both the interpreter and the application in L1, but whole-program speeds benefit from this treatment sometimes by factors of 1000x or more, and that's hard to show with these microbenchmarks as well.

> To be honest, I wish more languages put this kind of effort into their standard libraries, but most have about 10% of what I would like to see. For example, this kind of "string matching" is really "sequence matching" and ought to be fully generic for any underlying comparable and copyable type, not just arrays of characters

In APL, this is called ⍷ (pronounced "find") and sometimes even APL-ers momentarily forget it exists[1], but in k you always have to make it yourself, usually (as we did today) with ⍸ (where) and ≡ (match), but sometimes some other way[2], and this works on all the different data types k supports (including integers, enums, dates, times, symbols) and across multiple cores as well. You are right to predict it would be useful: It is useful :)

[1]: https://news.ycombinator.com/item?id=27229994

[2]: https://news.ycombinator.com/item?id=16851862

rschovor 5 Jahren

Because you're wrong. Do spend six months doing hobby array programming. Your view will change.

jiggawattsvor 5 Jahren

It's on my bucket list. But it seems so niche that I suspect I'll never use it.

My only motivation is that I've been tempted to develop a toy programming language myself that is as high-level as Rust or C++ but explicitly designed for "wide" processing platforms such as GPUs or many-core processors with SIMD instruction sets.

All I'm saying is that the concept of array programming -- like pure functional programming -- may be valuable, but the syntax is not.

klibertpvor 5 Jahren

I just can't understand how even experienced programmers can fixate on syntax. In my - repeated tens of times by this point - experience the syntax is important for a few months (tops!) at the beginning of using a language, and then stops to matter almost completely. What experiences would make someone convinced otherwise? There's an argument against too high complexity of syntax, but most general-purpose languages out there are very similar in this regard...

jiggawattsvor 5 Jahren

There's empirical evidence, loads of it, that this simply isn't true.

Syntax absolutely does matter, in all walks of life, not just programming.

We're not infinite, error-free computers like some sort of mathematical abstraction. We have squishy meat brains evolved to tell stories around a campfire.

As a random example: I learned Rust just for fun, and something that repeatedly caused hours of frustration is that unlike every other modern language, it allows identifier shadowing. This compiles and runs just fine:

    let a = 5;
    let a = "wat?"

Practically no other language allows this, because it is a recipe for errors.

Internally, compilers typically use single static assignment, so the above would be processed something like this:

    let a_1 = 5;
    let a_2 = "wat"

For a compiler to track this is no problem. For a human? It's a problem. Maybe not for trivial examples like this, but if dozens of identifiers are littered across a huge function it can be a challenge to keep track of which one is which if the names are the same but they're... not the same. Especially if the change is subtle.

We're not computers, that's why we make them out of silicon and metal and sell them for money.

klibertpvor 5 Jahren

> There's empirical evidence, loads of it, that this simply isn't true.

Source?

> unlike every other modern language, it allows identifier shadowing

1. All dynamically typed languages allow this, all/most REPLs even for statically typed languages allow this, even if regular code doesn't.

2. This is not syntax, at all, this is semantics of immutable value declaration+initialization.

> but if dozens of identifiers are littered across a huge function it can be a challenge to keep track of which one is which

Littering dozens of identifiers across huge functions is going to be a problem, no matter the syntax. It's a programmer's job to manage complexity by utilizing various kinds of techniques, including syntactic sugar (true), but also factoring the code into manageable chunks, using higher-level or better suited for the task at hand abstractions, and so on. Syntax on its own is, in my experience, the least impactful technique for managing complexity, at least before going into DSL-land (but then it's syntax+semantics).

Another thing worth mentioning is that lots of general-purpose languages give you exactly the same constructs, just spelled differently. Simple examples:

    for (x : list) { ... }
    foreach (x; list) { ... }
    for x <- list do ... end
    for x in list: ...
    for my $x (@list) { ... }
    for x := list { ... }
    list each(x, ...)
    list each: [ :x | ...]
    (loop for x in list do ...)
    (for [x list] ...)

All the above denote exactly the same construct, with very little variation in the number of tokens. Can you tell me if you think one of them is better than the others - and if so, why?

rschovor 5 Jahren

Syntax can certainly be more or less suitable to a particular problem or even to humans in general, but (dis)allowing identifier shadowing is part of the semantics of the language (i.e. what the syntax means). One could say: "there should be syntactic cues if we are to allow variable shadowing". That would be a well-defined criticism of syntactic issues. BTW, most modern languages do allow identifier shadowing. The particularity of Rust and the object of your criticism is allowing it in the same scope as the definition.

But again, the only way you'll be able to apprehend array language notation is by writing enough of it to become somewhat fluent. It doesn't mean you have to like it, but remember it was the object of a Turing award and that's usually a good indicator of something relevant.

rschovor 5 Jahren

https://futhark-lang.org/

Is the closest you'll get to that, I think. It took a whole team of big brains to make it, and the author stated it was highly non-trivial to make something like that.

jiggawattsvor 5 Jahren

Interesting, but not quite what I had in mind.

My idea was loosely based on a code search engine Google had for a while before they killed it off.

I'm not sure exactly how they did it, but I very strongly suspected that they implemented regular expressions over database indexes.

A simple b-tree index is just a set of sorted strings. If you have a lot of sorted strings and you squint at it, you can see how it picks out common prefixes, like a tree. (You can literally store it with the prefixes factored out, and then you have a trie.)

For a fairly wide range of regular expressions, and a tree of sorted strings, you can implement a search over the b-tree. E.g.: the search "[bx](a+)foo" can be implemented via finding the range starting with "b", then the range starting with "x". For each range, find the sub-range starting with "ba", "xa", etc...

Each step takes logarithmic time, and can be done in parallel. In theory, it takes only about a hundred times longer to find a ten matches in a million strings than it would take to match a single string.

Wouldn't it be nice if you could take an algorithm such as "match regex" that normally takes a 'scalar' string and have the compiler automatically create a version that can take an array of sorted values, like the database index?

The idea is that much like how array programming languages eschew scalars and pass arrays all over the place to gain efficiency, my language would pass sets all over the place, and gain a different type of efficiency.

geocarvor 5 Jahren

> the concept of array programming -- like pure functional programming -- may be valuable, but the syntax is not.

You would not be the first person tempted to try and separate the two, but I think it is impossible. Every attempt to do so I have ever seen has lost too much in the translation that the four-values I have for programming are no longer conserved.

userbinatorvor 5 Jahren

I bet Chinese and Japanese look like that to someone who knows only English too.

Avshalomvor 5 Jahren

As some one who only knows English. No that is not what Chinese and Japanese look like to me.

That's why I prefer APL and its special symbols. It's still utterly inscrutable if you don't know it but at it looks intentional. Those symbols shift your mindset or reframe what you're looking at.

rschovor 5 Jahren

it's extremely readable when used to it, though. I use J for exploratory data analysis. It's really, really good at that kind of thing.

jiggawattsvor 5 Jahren

I've lost count of the number of times I've heard some theoretical mathematician say that about impenetrable gibberish.

dangvor 5 Jahren

"Please don't post shallow dismissals, especially of other people's work. A good critical comment teaches us something."

Dismissing what you don't understand because it is unfamiliar is the essence of a shallow dismissal, no? And you did it twice in this thread. The first was a blessing in disguise because of geocar's excellent reply, but now you're just repeating it, with added name-calling. Please don't do that here.

https://news.ycombinator.com/newsguidelines.html

jiggawattsvor 5 Jahren

It's the essence of the thing in this case, and isn't a shallow criticism.

Syntax can be good or bad. I don't believe that it's just a matter of "getting used to it", there are objective metrics of readability, developer error rates, and speed of training that some languages do poorly at.

My point was that most array languages look like line noise. That's flippant, but true.

Ask yourself this question: Is it possible to develop an array-based language and use more "normal" syntax?

Of course it is! That was a rhetorical question.

Similarly, there are other branches of science or mathematics that through an accident of history have developed impenetrable syntax.

This is not dismissing something I don't understand. I understand several such branches of mathematics and still think the syntax is unnecessarily obtuse. Coughcategory theorycough.

PS: Geocar wrote about two pages to explain what is a trivial piece of code, and I still have no idea what it even does or how! I can write code in 10 languages and read about 20-30 without too much difficulty. This includes obscure languages like Mathematica, Haskell, and several flavours of assembly language.

Array based languages are the first time I've seen a high-level language that is less readable than the machine code that they are supposed to be abstracting away! It's also the second time that I've failed to understand a simple piece of code even with a detailed explanation. For reference, the only other case is quantum algorithms.

Okay, I tell a lie. Thinking back on it, I've also seen Perl scripts that are unreadable line noise, but that's about it...

geocarvor 5 Jahren

> It's the essence of the thing in this case, and isn't a shallow criticism.

Listen: I don't think you're qualified to speak to the essence of a thing that you don't understand, and I think you know that. I am telling you this is not very difficult and it is very useful, and I think you're angry at yourself for not "getting it". You are so smart you know ten programming languages, so you assume there's something wrong with "it" because to think there's something wrong with you is unthinkable.

Relax. There's nothing wrong with you either. You can learn this, but you aren't going to be able to skim it; the knowledge you have gained learning ten other languages is not going to help you very much. What is going on is stranger than you can realise at this point, so many array-people believe it is easiest if you try to forget everything you know about programming when you try to learn an array language. I don't think I agree with that, but the myth that children learn languages faster than adults is pervasive. Maybe that helps.

> Geocar wrote about two pages to explain what is a trivial piece of code, and I still have no idea what it even does or how!

Try reading it again! It is not complicated (nor optimal) but you are likely skimming or skipping important parts. If you can tell me where I lost you, I can probably help you find your way.

You might find it easier to follow along with a k reference guide. These typically fit on a note-card and can be helpful in memorising the k operators. These are all of the operators in the above program:

    $[a;b;c] cond (if a then b else c)
    @[a;i;f] amend i indices in a with operation f

    &x where (are the 1-bits in x)
    #x count (the length of x)
    *x first (element of X)

    x&y min/and
    x~y atom-equivalent (like lisp's equalp)

    f':x call f with each pair of x

A complete reference card isn't much longer and usually ships with the k interpreter. People learn in different ways, and that's ok! Help me help you learn and you will probably learn this very quickly.

> Is it possible to develop an array-based language and use more "normal" syntax?

"Array-based" isn't exactly the same thing as an "array" language - I think people get too hung up on the power of vector-operators (which are indeed powerful!) and miss the fact that there are so few operators which to learn.

I don't expect you to understand exactly what I mean by that, but I hope that if you take the plunge to learn this properly, you'll remember that I said this and when it does make sense, it will be useful.

> Array based languages are the first time I've seen a high-level language that is less readable than the machine code that they are supposed to be abstracting away!

You can't read Arabic either, can you? Why do you think this is different?

Try to stop thinking in terms of "readable": You can only say something is "readable" to you, or perhaps if you are a hiring manager, "to most people". I can read it perfectly fine. Worrying about how "readable" a programming language at this point speaks only to how easy it is for you to acquire the ability to learn it, not how valuable it will be when you actually learn it.

thrwaeasddsafvor 5 Jahren

> I don't believe that it's just a matter of "getting used to it", there are objective metrics of readability, developer error rates, and speed of training that some languages do poorly at.

If we had such objective metrics on APL or K, that would be an interesting discussion. But what I've seen here is just opinion and anecdote.

I also think that "looks like line noise" is a fair characterization of how J and K look on the surface, but that does not mean they are unreadable.

I'm hardly qualified to give any deep insight, but after reading the first half of Aaron Hsu's thesis and playing around with a Dyalog APL interpreter online for a few days, APL starts to look quite readable to me (much more so than J, thanks to the additional symbols) and I found it rather easy to learn (unlike mathematics, but for me the struggle there is not at all in syntax but in the concepts, methods, and meaning).

rschovor 5 Jahren

Geocar put in the effort of trying to make you see. But you are prejudiced with the equivalent of those who spite Lisp because of parentheses.

If you want a "more readable" array language, take a look at Nial.

PS: qualifying the Wolfram language (Mathematica) as obscure is a really, really myopic view of the programming world.

jiggawattsvor 5 Jahren

I've never seen anyone use Mathematica outside of academia other than myself. It also occupies a slightly unique corner of the language design landscape. One that I like!

I wish that Mathematica was modernised and made into a proper, compiled (or at least JIT-ed), heavyweight language with all the trimmings. A proper IDE, debugger, tracing tools, etc... The current version is a bit like Visual Basic -- a very old language that had grown past its original design capabilities and now needs a revamp. A bit like how VB.NET replaced VB, and was superseded in practice by C# on dotnet core.

[deleted]vor 5 Jahren

rschovor 5 Jahren

Then you have a different use case. Doesn't mean theoretical mathematicians are wrong for their use case. I do epidemiology, not pure maths though.

secondcomingvor 5 Jahren

What part of finance uses this language? Is it in widespread use, or is like Goldman Sachs' proprietary language (I forget the name)

bidirectionalvor 5 Jahren

Various parts, particularly in markets, from pricing quants to high-frequency traders. It is fairly widespread. Barclays, JPM, UBS, Morgan Stanley, HSBC are some of the big names, then you have loads of smaller firms.

the_only_lawvor 5 Jahren

I recall a discussion on array programming languages on here a while back where someone claimed that an acquaintance of theirs was earning nearly 7 figures working on q/kdb+.

mlang23vor 5 Jahren

The latest language which fits quote 1 for me was Haskell. Even though I already had some functional background (Lisp), it took me seemingly forever to actually grok purely functional programming. But once it clicked, it felt like stepping up on a ladder. My perspective on other languages changed as well.

geocarvor 5 Jahren

I have a common-lisp background, and I learned Haskell at the insistence of a former colleague. I also know k, and have since learned some APL and j. I would like to try and suggest to you my perspective:

The jump from Python to Haskell - or really anything along that way is like talking about a ladder of computing. You start at one end, and you are climbing upwards. And every step you take, you can look down and see all of the things you knew before, but with greater perspective.

And Haskell? Well, it's definitely pretty far up the ladder. If you get Haskell, you feel like you really understand what's going on. I know pg was talking about lisp when he was thinking blub, but in some blubish respects, Haskell is a better lisp than lisp.

But see, going from Haskell (or really anything) to Iverson is like, listen: Forget the ladder, because a ladder only goes up and down. Iverson is sideways. It is in this way, like adding depth to flatland, that Arrays are an even bigger deal than you can possibly imagine until you go there.

an1sotropyvor 5 Jahren

For me it was Prolog. I came to a class, which used Prolog, with a bad attitude of "whatever I can think of, I can program in C". Luckily for me I was schooled.

mamcxvor 5 Jahren

For my was Rust. I done like 12(?) Langs before, including F# that also was a change of mind, but the first 2 or 3 months of Rust I fell like an idiot looking intensely to a wall. I start to think my 20 years programming were a big fat lie.

I can't believe why it feels so hard? !I already know pascal and obj-c and F#!, kind of similar, no?

Now I feel rust so easy (as python easy!) that is weeeeeeird. (btw: I think is months now where I never think I have meet an error or situation that truly confuse me).

zanethomasvor 5 Jahren

https://github.com/abrudz/ngn-apl