In my experience, existing programmers struggle the most in learning APL because they struggle to internalize the Array Model and how to talk about things as Arrays. The results are, in essence, very poorly typed data, which then requires tons of mangling to make work.
It's the same thing as when you learn various other languages and you start to not appreciate how to appropriately model data in that language, which leads to excessively complex or inadequate types.
IMO, the elegance of a notation is not defined by its ability to prevent people from writing bad expressions, but by the capacity for good expressions to be practically and usefully leveraged. A good notation is about empowerment and clarity, rather than about prevention.
I'll comment a bit here. If you look into prior research, there are a number of papers on sparse representations in APL along with accompanying code in the literature. That's a good starting point.
Additionally, sparse matrices are on the roadmap for Co-dfns in the future, so there's hope for you there. If you have a specific problem that you wish would work, the best thing to do is to make it known to us so that we can actually prioritize it. I think myself and Dyalog often prioritize those things for which we know there are users. If you want to be able to do this stuff, it would be great to get some concrete programs that we can work off of.
Finally, licensing should, IMO, be a non-issue for most people in the vast majority of cases. Dyalog provides some of the best licensing terms that I've seen, and there are already models that scale well both to startups and to existing deployment in large-scale commercial enterprise situations. If something in the licensing terms doesn't seem to work, I would get in touch with Sales @ Dyalog and they will surely be able to work with you to figure out something that would be equitable.
Most people are surprised when they find out how easy it is to move forward with Dyalog on a commercial product, or how readily Dyalog is looking at making things work for customers or people who wish they could be customers.
Just a comment about the intellisense stuff. Dyalog's IDE(s) do provide name completion for methods, functions, variables, objects, and namespace bindings. You don't get any auto-complete for the ordering of parameters, but arguably, this is less of an issue with a 1 and 2-arity function requirement in which it is generally bad style to deviate too far from this for too long in too many places without good reason.
Rest assured that I haven't given up on the textbook / tutorial approach just yet, but I might have to "invent" my own publication technology to get the "book" that I want.
Thanks for the thoughts. While I am sympathetic to your position (and I don't necessarily disagree), I'm not sure that the education problem with APL is strictly technical in nature. Nor am I convinced that what needs to be learned in order to use APL is strictly technical in nature.
Part of this is a question of what the stamina level is for the current computing student. With a lot of technical documentation that people are learning, they're essentially consuming either thorough reference materials or walkthrough/how-to materials.
We can characterize both of these materials as efficiently transferring a certain sort of information about how to do things by virtue of a very wide base of shared prior knowledge. This is what allows reference materials and walkthroughs to work.
But this shared base of knowledge does not exist for most people learning about APL. Target audiences include non-computing students who are just learning computing, programmers who want to learn APL, and technical professionals who have little or limited prior computer science knowledge (but perhaps cursory programming knowledge) who need to learn APL to accomplish the work they want to do in their domain. In the cases of the technical professional and the uneducated student, reference materials and most tutorial books are wholly inadequate.
For the programmer, these tutorials and references work because most of the time you are adding to your existing knowledge as a programmer and have already trained numerous soft skills that enable you to navigate the sea of potential directions to integrate the new knowledge.
I worked on my thesis to try to make it as accessible to the general programming community as possible to teach a set of algorithmic concepts that most programmers already have a vague notion about (but almost never use), namely, SIMD/data parallel programming, using an extremely simple and easy to explain syntax and vocabulary (APL). My thesis is already pretty detailed and illustrative, with worked examples for every single programming statement in the entire compiler going over examples and prose detailing everything that is going on, a sufficient background introduction not only to the syntax and semantics of APL, but also the design and stylistic questions, reference material, and the architectural education necessary to understand the structure of the compiler as a whole.
Even with all of this, presented to experts in the C.S. fields of PL and parallel computation, who have seen this material multiple times over, there was a general feeling that my thesis could have benefited from even more exposition.
My thesis is around 70,000+ words and around 250+ double-spaced pages. For 17 lines of APL code. People wanted more. Had I added the additional exposition desired, I imagine the thesis easily could have been 80 - 100k words.
A typical short novel is around the same size, and those often take about 8 - 12 hours to read through aloud. If you give me around that amount of time in classroom time, I could easily have almost any programmer fully understanding my compiler within that time, and probably with more thoroughness than if someone were to take the same amount of time to read the thesis.
There is something about the higher bandwidth and richness of full video content that contributes better to the transmission of data in some ways, particularly to the understanding of certain ideas and concepts.
I began to think about what it would take to put together an introduction to APL as a textbook. These have been done before and are out there, and they would generally work, but I'm not convinced anyone would read them.
You can teach APL the language to a sufficient level in a very short document (maybe a few pages) and in maybe 2 hours of classroom time. It's relatively trivial to teach the language, as it's very, very easy to understand the semantics and syntax. There's even a walkthrough of APL at http://tutorial.dyalog.com already.
The challenge comes in its application, and that's where the real value of a good education rests. But that part of the education is a lot harder to teach in text, because there is an element of style, aesthetic, and practice that goes into it. It's something that you gain by seeing, watching, and doing. You have to bring your mind into the space and struggle, go in and out of exploration mode and absorption mode.
How do you teach the exploratory process of APL? The design aesthetic? The set of idiomatic patterns? How do you get people to intern those concepts into their practice so that they are productive?
You could write a textbook to do this, no doubt. But thinking about it, I think such a textbook would be very large. It would be not very "exciting" emotionally. It would have very little of the emotional hooks that help a person stay engaged in things. And moreover, it would be extremely intimidating.
How many people effectively learn without any teacher in a totally "new" field from a textbook that is devoid of any human feedback? You can do it. I've done it before. But it's infinitely improved by the introduction of someone who can help to inject energy and dynamism into the space.
I just don't think people will want to pick up a 400 page "introductory" book and then have the motivation to stick with it when there are so many other shiny things in this world today. The truth is that the world is filled with distracted people who are very bad about putting the time into learning things. I want APL education to be engaging, accessible, and satisfying, to say nothing of effective. For most people today, a textbook is unlikely to be that.
Another big challenge with the "experienced programmer" target audience is that they are used to reference and tutorial materials. Very few programmers actually sit down and read such materials. They skim them to find what they think they want to know, and then plug that into their existing framework. That's great for efficiently extracting new information that is similar enough to their old practices to work. If something breaks, they can just go back and do a little extra skimming to clarify. That doesn't work really well when what you need to learn is fundamentally conflicting with or altering how you think about problems.
Getting good with APL at the beginning, before you've learned to intern it, is about undoing or unwinding previous habits and thought patterns that worked for you in past programming experiences, but that won't work for you in APL. After you have interned the new methods, then you can switch back and forth at your leisure, but until then, learning APL is as much about unlearning as it is about adding new knowledge. For programmers who are used to just skimming what they read to get what they want to know without actually listening to what the author is saying (and that is how most programmers read that I've met), this makes textbooks a recipe for disaster. Either the information has to be presented in such a dense way that most people will not be able to figure it out, or it has to be presented in such a measured, hand-holding way that most people will just ignore it, get bored with it, or skim it and still have the issues.
It does little good to have a full, thorough textbook introducing everything if no one reads it, though they think they've read it, and then they complain about how hard APL is to learn because they couldn't just take what they thought they wanted to know and leave the rest.
If you could get a programmer who was motivated, eager, and willing to read from cover to cover, stopping at each point where a hard question is asked, do the problems, quizzes, challenges, and exercises, in order, linearly, from start to finish, and actually pay attention to what was said when, and then take the points in the book that indicate one should think about this or that, I would agree with you that a textbook could have some potential benefits over the video format; but, I don't think that this is reality, based on my experience.
Your typical APL workshop for technical professionals is 4 days about about 24 hours of work with probably around 12 of those hours being expository. After that, people are quite solidly grounded in APL. But a lot of people have gone through that same amount of time on their own with plenty of the same resources available to them in terms of problems, documentation, and the like, and they haven't been as successful. Guided, in-person education is hard to beat, and video is the closest thing online.
I haven't given up on the hope of a great APL book, and I could be convinced by someone that I should do a book first, but at the moment, I feel that video courses are more likely to be more effective for more people and present a larger ROI in terms of the difficulty in producing and creating them compared to a book of similar effectiveness for the same range of people. If anyone has an alternative, please do speak up, I'm all ears.
Except that you do. Python is a form of OOP/Iterative programming, involves memory allocation, pointers, references, and the like. Pandas requires the skilled application of a general purpose language coupled with a domain-specific vocabulary for your problem space.
I maintain that APL isn't fundamentally more difficult. If there is something that you have as an example that you think is just too "hard" to do in APL that is very "easy" in Python and Pandas, please do let me know via email ([email protected]) as I would love to make sure APL has a good answer to such things (I want to make APL accessible).
Since you are talking about statistics and data analysis type things, you may find TamStat of interest:
Additionally, don't forget that Dyalog APL also has a full Chart/Graphics system (two, actually), and a large suite of idiomatic expressions for computing many classic analysis problems. There's a number of other integrations that people seem to be unaware of often enough.
I very seriously would like to receive from you one example of something you have done that you find too difficult in APL, because if it really is, then we can do better and I'd like to make sure that we do.
I've updated the Getting Started page to include a link to the Dyalog APL download page. APL is extremely easy to install now for most platforms. Thanks to Nix users at our Workshops, it is now easy to install on Nix as well.
This should get you started, though some of the instructions are based on the less friendly download process of v17.0 instead of the streamlined process of v17.1.
It's really no more difficult than learning recursion, pointers, memory allocation, type theory, functional programming, iterative programming, OOP, or any other of the myriad techniques that we use for computing. As I mention in my talk on APL design patterns, it's largely a matter of what we have been trained to understand, and there is some evidence to suggest that APL is actually arranged in a manner more natural for the human faculty than most computing languages:
Dyalog APL is also free (as in beer) and easy to install now that distribution has been streamlined. While I am biased, I consider Dyalog APL to be the premier APL interpreter and getting it is a matter of a few clicks on Dyalog's download page:
Dyalog recently changed its distribution policies and licensing approaches to the APL interpreter, so it's easier than ever to get started on any platform.
I would encourage you to leverage this to write a Gentoo portage (or whatever they are using now) build for it yourself and send it out for the world. We have a number of Nix users that come to our APL workshops who have arranged for just such a system on Nix.
For typing the APL symbols, I use this: 'setxkbmap -layout us,apl -variant ,dyalog -option grp:lswitch'. I find this to be the most efficient for typing APL. The APL keyboard map comes default on modern Linux distributions.
Dyalog APL also has extensive documentation for configuring your APL system on your platform.
I hesitate to call J the successor to APL. I would currently say that J was/is a step within the evolution of array languages, but many of the concepts of J are being actively integrated into Dyalog APL, and Roger Hui is a part of Dyalog right now. I feel that Dyalog APL is the future of APL, learning from J, rather than J being a strict successor to APL.
Actually, there is movement in this general direction. My Thesis is now out and available publicly in its final version and should be indexed in the databases shortly entitled "A data parallel compiler hosted on the GPU." This provides a bit of an introduction to some aspects of APL, and includes a background on some of the ideas I've talked about before. Additionally, it is a "handbook" of sorts for doing tree transformations in APL, which is, to my knowledge, the first of its kind.
As for the beginner's book, I've come to the general feeling right now that a online video course will be a better fit for APL beginners than a textbook. This is an active project of mine that I anticipate completing some time within the year 2020.
You may also want to see my presentations on design patterns for APL and the upcoming videos at FnConf 2019 this year. I am not sure if the APL Training Wheels session was recorded, unfortunately, but the content from that work will be going into the video course.
I clarify this more above, but one of the reasons you don't see changes to that file is that much of the work of the repository was working on code that was deleted in favor of that file. And yes, the XML was edited by hand, as text, using ed(1). The vast majority of what I was exploring in the past I completely discarded in favor of that e.cd file that now represents the compiler.
I notice that my discussion about the churning on the GitHub repository has been repeated a number of times. I can provide some historical clarification on why there was so much churning there.
The documentation for those 17 lines of code are in a hopefully soon to be published thesis of about 70k words that includes performance information and the like.
However, during the development of this project, I didn't start writing it in APL on Dyalog. I explored a significant number of other architectural designs and programming methodologies. Some more popular I tried were habits like Extreme Programming, Agile methods, Java, Scheme, Nanopass, SML, Isabelle, and even a custom extension of Hoare Logic on top of dfns. I believe that I also explored implementing the compiler in C++/Boost and prototyped some stuff (I don't know if it ended up in this Git repo) and C.
In other words, the compiler has not been a single code base, but has been a series of complete rewrites using different methods, approaches, languages, techniques, and architectures. I have used heavyweight machine support (mostly around C++ with Visual Studio's infrastructure) as well as some very hardcore UNIX style low-level handiwork. Multiple different IDEs, text editors, and operating systems were all explored, as were multiple different backends, targets, and the like. At one time I had targeted LLVM, and another C, another OpenACC, and another ArrayFire.
The whole project has been a somewhat wide ranging exploration of the design space, to say the least.
What you are seeing of the XML stuff was from a particular design effort that was an attempt to apply strict Cleanroom Software Engineering as a methodology to the compiler design, to see what would happen. In the end, I abandoned the attempt, for what I hope will be obvious reasons, but during this time, I predominately worked on RHEL with the ed(1) text editor editing XML files for the DocBook publishing suite. Parts of the churning are the incorporation and removal of various dependencies that had to be brought in and out of the repository depending on what infrastructure I was relying on. In the case of DocBook, some of those files are large.
However, a significant amount of the work of Cleanroom Software Engineering is "coding via documented process." This includes the certification steps as well as the function specification, increment development, sequent analysis, and so forth.
Thus, for a very real portion of the Co-dfns work, I was literally programming in XML using ed(1) to model relatively complex state machines and function specifications that provided very fine-grained behaviors of the compiler. For example, a significant amount of work went into the following file:
This file is about 45k lines of XML, and was written and edited entirely by hand using ed(1). I had a video demonstration of this a while back which demonstrated how I did this, and particularly how I did a lot of this with ed(1), but I lost the script file recording.
Over time, as I continued to explore patterns and development approaches, I continued to discover that the code was faster, better, and easier to work with as I removed more and more "stuff" from the various processes and possibilities.
It wasn't until relatively late in the game that I actually realized that not only could the compiler be written in dfns well, but also that the compiler could be written in dfns in a way that was fully data parallel, which is the core insight of my Thesis. This had significant ramifications on the source code, because it meant that the compiler could not be tackled not only as a self-hosting project (at least in theory) but also in a fundamentally idiomatic way.
The result is that the compiler has generally continued to be more featureful, less buggy, and more dense at each major stage, with the latest leading to 17 lines of code. This is accomplishing essentially the same result as the 750 lines of code in a previous HN discussion, but does so partly by recognizing some passes as irrelevant and unnecessary to the current needs.
I do expect that after the publication of the thesis, the compiler will grow a little bit to add some new things that need to go in. However, at this point, I have a fairly efficient methodology.
So, the GitHub repository is not just a record of the code, but a record of a lot of different approaches to how to do what I was trying to do. Much of that XML you see was very much "coding" in the sense that I was providing for the core behavior of the system and was the primary specification of its behaviors in a formal, rigorous manner.
You may want to check out the Co-dfns compiler, which is proof positive that APL is a terrific language for doing things like tree manipulations, which is the traditional domain of Lisp and functional programming languages like Haskell.
You mean Dyalog APL? It already regularly outperforms normal, handwritten C code. It also has support for distributed computing, multi-threading, and the Co-dfns compiler can be used to compile your APL code to the GPU. For example, consider the following talk, which discusses sub-nanosecond lookups/search using the Dyalog APL interpreter.
This is an intense, APL-driven position working closely with both Dyalog, Ltd. and one of our major customers as a full-time consultant. You'll be working with both new development and large, legacy application code bases.
This position includes extensive training in "the APL way" that I've spoken about in the past, and we're looking for people with the aptitudes and the motivation to go far beyond a simple "code monkey" or software engineer.
Our hiring process is very hands-on and somewhat unique, and you should expect to be doing more hands-on programming and exploration throughout the interview and probationary process than just interviewing.
If that's something of interest to you, get in touch with me via email and we can see if we can't get Co-dfns supporting what you need to make it a reality! There are a number of things that could make this nice in Co-dfns, including the possibility of using idioms instead of library calls, potentially making adaptation of algorithms easier.
It's the same thing as when you learn various other languages and you start to not appreciate how to appropriately model data in that language, which leads to excessively complex or inadequate types.
IMO, the elegance of a notation is not defined by its ability to prevent people from writing bad expressions, but by the capacity for good expressions to be practically and usefully leveraged. A good notation is about empowerment and clarity, rather than about prevention.