Getting Started with APL (2017)(sacrideo.us)
sacrideo.us
Getting Started with APL (2017)
https://www.sacrideo.us/getting-started-with-apl/
40 comments
The Gnu APL interpreter is free, installs easily and the author is not only very responsive on the bug-apl mailing list but very knowledgeable about APL. Recommend you check out this interpreter.
Dyalog APL is also free (as in beer) and easy to install now that distribution has been streamlined. While I am biased, I consider Dyalog APL to be the premier APL interpreter and getting it is a matter of a few clicks on Dyalog's download page:
https://www.dyalog.com/download-zone.htm
https://www.dyalog.com/download-zone.htm
Nars2000 is also free as in beer and opensource, and is easy enough to install, and is nowhere near as comprehensive as Dyalog but interesting in its own ways (ball arithmetic, combinatorial functions, pi/prime number functions, simpler GUI)
NARS2000 is always doing some interesting stuff. Morten has said before that if he had to recommend an open-source APL, it would be NARS2000.
A big hurdle for me is that there's no APL interpreter available in my Linux distribution's (Gentoo's) package manager. So I can't easily install it as I can most other languages, using my distro's package manager. That means I'm going to have to search out how to get an APL interpreter -- something not even mentioned on in the Getting Started article linked to in this HN post, nor on the Getting Started article on dyalog.tv that the former article links to. That alone is just way too much trouble already.
Then there's the issue of typing in all the weird APL symbols. I've reconfigured my keyboard to type special symbols a long time ago, but I don't even remember how to do that now so it'd be nice if there was some straightforward documentation on how to do that for APL on a typical Linux system -- or, better yet, some easy plug-and-play way to do that. That would eliminate yet another major hurdle.
Then there's the issue of typing in all the weird APL symbols. I've reconfigured my keyboard to type special symbols a long time ago, but I don't even remember how to do that now so it'd be nice if there was some straightforward documentation on how to do that for APL on a typical Linux system -- or, better yet, some easy plug-and-play way to do that. That would eliminate yet another major hurdle.
Dyalog recently changed its distribution policies and licensing approaches to the APL interpreter, so it's easier than ever to get started on any platform.
I would encourage you to leverage this to write a Gentoo portage (or whatever they are using now) build for it yourself and send it out for the world. We have a number of Nix users that come to our APL workshops who have arranged for just such a system on Nix.
For typing the APL symbols, I use this: 'setxkbmap -layout us,apl -variant ,dyalog -option grp:lswitch'. I find this to be the most efficient for typing APL. The APL keyboard map comes default on modern Linux distributions.
Dyalog APL also has extensive documentation for configuring your APL system on your platform.
https://www.dyalog.com/documentation_171.htm
I would encourage you to leverage this to write a Gentoo portage (or whatever they are using now) build for it yourself and send it out for the world. We have a number of Nix users that come to our APL workshops who have arranged for just such a system on Nix.
For typing the APL symbols, I use this: 'setxkbmap -layout us,apl -variant ,dyalog -option grp:lswitch'. I find this to be the most efficient for typing APL. The APL keyboard map comes default on modern Linux distributions.
Dyalog APL also has extensive documentation for configuring your APL system on your platform.
https://www.dyalog.com/documentation_171.htm
Just try kdb+/q instead. It’s really the only APL derived language that is used in industry. It’s blazing fast and if you get good at it, you can make more money than maybe any other programming language. It’s a very nice environment and the language (q), comes with a great time series database (kdb+). It’s proprietary, but the 32-bit version is free to use, I believe.
> It’s proprietary, but the 32-bit version is free to use, I believe.
They've changed the licensing a few times. Originally the 32-bit version was free for commercial use, then it became non-commercial use only. Now you can get both the 32-bit and 64-bit versions free for non-commercial use, but you're not allowed to run the free 64-bit version "in the cloud".
https://kx.com/connect-with-us/licenses/
They've changed the licensing a few times. Originally the 32-bit version was free for commercial use, then it became non-commercial use only. Now you can get both the 32-bit and 64-bit versions free for non-commercial use, but you're not allowed to run the free 64-bit version "in the cloud".
https://kx.com/connect-with-us/licenses/
I don't have lots of money and my company passed on it as too proprietary, else I'd be all over it.
Did you actually see a quote from Kx before making that decision?
I've first-hand seen the cost/proprietary argument thrown into the face of KDB, to then see companies proceed down a path of other technologies, with a 10x spend in hardware and dev effort.
KDB is insanely tuned.
I've first-hand seen the cost/proprietary argument thrown into the face of KDB, to then see companies proceed down a path of other technologies, with a 10x spend in hardware and dev effort.
KDB is insanely tuned.
There are other costs. It is hard enough to find people in my industry with domain knowledge, ability to dig into problems, and SQL skills. Finding enough people to use an array language would be difficult as others in my industry have found. I'd personally be super pumped, but I'm an outlier here. Also, there are costs to getting kicked into something like this instead of something like tsdb or influx.
I've updated the Getting Started page to include a link to the Dyalog APL download page. APL is extremely easy to install now for most platforms. Thanks to Nix users at our Workshops, it is now easy to install on Nix as well.
https://github.com/markus1189/dyalog-nixos
This should get you started, though some of the instructions are based on the less friendly download process of v17.0 instead of the streamlined process of v17.1.
https://github.com/markus1189/dyalog-nixos
This should get you started, though some of the instructions are based on the less friendly download process of v17.0 instead of the streamlined process of v17.1.
https://github.com/fedeliallalinea/gallifrey/tree/master/dev...
I thought Gentoo people despise plug-and-play. This is what I have for xorg
I thought Gentoo people despise plug-and-play. This is what I have for xorg
Section "InputClass"
Identifier "KeyboardDefaults"
MatchIsKeyboard "on"
Option "XkbLayout" "us,apl"
Option "XkbModel" "pc104"
Option "XkbVariant" "dvp,dyalog"
Option "XkbOptions" "grp:shifts_toggle,caps:hyper"
EndSection"I thought Gentoo people despise plug-and-play."
If we did, would we have a package manager? Would Gentoo auto-identify hardware as it booted? Gentoo is mostly plug-and-play, like every other distro. We just have a lot of choice.
Thanks for the apl ebuild, by the way. That should make things easier, though it'd be much better if it was part of Gentoo's official portage repo.
Thank you also for the keyboard configuration. But how would I switch layouts from US to APL? Also, is there some graphic I can refer to to see which keys will output what APL symbols?
If we did, would we have a package manager? Would Gentoo auto-identify hardware as it booted? Gentoo is mostly plug-and-play, like every other distro. We just have a lot of choice.
Thanks for the apl ebuild, by the way. That should make things easier, though it'd be much better if it was part of Gentoo's official portage repo.
Thank you also for the keyboard configuration. But how would I switch layouts from US to APL? Also, is there some graphic I can refer to to see which keys will output what APL symbols?
It's trivial to write an e-build for most software, so if there's not already an APL overlay you can make one in a day.
Using your distro's package manager to setup your development environment is usually cumbersome. Perhaps it would be better to use something like Nix on top of your Gentoo to manage that, or even abusing your ~/.local directory with manual installs of the tools you want.
J is the successor to APL and uses standard ASCII and should be in your package manager (or you can install pretty easily with wget and bash) and it is free.
I like the APL symbols better though.
I like the APL symbols better though.
I hesitate to call J the successor to APL. I would currently say that J was/is a step within the evolution of array languages, but many of the concepts of J are being actively integrated into Dyalog APL, and Roger Hui is a part of Dyalog right now. I feel that Dyalog APL is the future of APL, learning from J, rather than J being a strict successor to APL.
But what’s the future of Dyalog? With the passing of John Scholes, then the CTO quitting to “work on his first love, compilers” while Dyalog talks of building compilers, then Gitte Christensen saying in a talk that one reason for changing the download terms is that there aren’t enough young people signing up, then after twenty or thirty years of development one person (Marshall) can go into the codebase and get enormous performance improvements..
I don’t at all mean this to be heckling from the peanut gallery, I have great respect for the people, the tools, the company; but from the distant outside there seems an air of slightly directionless faded grandeur, if you will, like looking at an old-timey cinema.
If Dyalog APL is the future of APL, how do you imagine that future?
I don’t at all mean this to be heckling from the peanut gallery, I have great respect for the people, the tools, the company; but from the distant outside there seems an air of slightly directionless faded grandeur, if you will, like looking at an old-timey cinema.
If Dyalog APL is the future of APL, how do you imagine that future?
Thanks for your interest in the future of Dyalog! You are absolutely right that 2019 marks a kind of watershed moment for Dyalog. The original two-man development team who built version 1.0 of the interpreter (released in 1983) recently celebrated their 70th birthdays. At the beginning of this year, both of them had reduced working hours to about half time. Geoff Streeter remains in good health, but sadly John Scholes passed away in February.
Before Dyalog was acquired in 2005 and Gitte and I came in as CEO and CTO, the team working on Dyalog APL consisted of 5 people. Today that number is heading towards 25, several of whom were hired over the past decade to be ready to take over the work of Scholes and Streeter.
Gitte's comment about "not enough" young people signing up needs some context. The problem was actually caused by significantly increased interest in APL, which made us realise that our "classical" registration process was putting potential new users off. It was often taking too long to get people a copy of Dyalog APL because of the manual processing that was required, people were confused by the licensing terms and afraid of using the software – and many young people are just allergic to providing any information about themselves online. Now, APL is available for Windows, Linux (including the Pi) and macOS with no questions asked – for non-commercial use.
Losing Jay as CTO was a bit of a blow, but I think we are on the way to recovery. Now that he has completed his PhD on the subject of a parallelizing compiler (research that Dyalog has funded for the last 5 years), Aaron Hsu will join young guns Marshall Lochbaum and Adam Brudzewsky at Dyalog, and we have one more C developer joining the team in the UK in January. We hired two fresh APL consultants in the USA in 2019, and will be looking to hire two more in 2020 to support both existing and new business. I've been in the APL business for 40 years now, and my opinion is that the "kids" at Dyalog are every bit as impressive as Scholes and Streeter were at the same age; the shoes will be filled. There is as much young talent at Dyalog as there has ever been working on any APL interpreter, throughout the 50+ year history of the language.
As the team grows, we are able to afford having one or two team members focusing most of their efforts on performance. These efforts have accelerated over time, as first Nic Delcros, then Roger Hui and now Marshall have spent significant time rewriting primitive functions to take advantage of vector instructions and new hashing, sorting and other algorithms that have been evolved over the last four decades. Hardware has evolved in ways which mean that many of the old algorithms are sub-optimal on modern chips where the relative speed of CPU cycles vs memory access is completely different from the Z8000. Yes, Marshall is very good indeed, but I can't see a problem there . Performance has improved consistently in most of the last ten versions of Dyalog APL, and I hope we may see even more spectacular improvements in the next decade, both in the interpreter and in GPU-based parallel compilers.
We have much work to do: In addition to language and performance work, we expect Aaron to work with Richard Park, another new recruit who was added to the team last year, to work on training materials and documentation – and with Marshall, Roger and myself on quote evangelism unquote; talks at Lambdaconf, Functionalconf and perhaps some new conferences in 2020 and beyond (invitations welcome!).
For decades, Dyalog was a company that made a very comfortable living providing a better APL interpreter to clients who converted to Dyalog APL after initially purchasing an APL system from one of the companies that did serious marketing of APL interpreters: companies like IBM, STSC and I.P.Sharp Associates. Thanks to our recent growth, and the fact that Dyalog APL now has decent tooling not only under Microsoft Windows, but Linux and macOS too, we are now starting to focus on developing new business and attracting the next generation of APL developers.
I'm sorry to hear that we appear "directionless" to you. I will admit that it is an interesting challenge to keep existing customers satisfied (which should always remain the first priority for any organisation – put on your oxygen mask before helping others) while also working on attracting completely new groups of users. The rate of development of new language features, development tools and libraries for Dyalog APL is as high as it has been for any APL interpreter in history, at the same time as we are making significant performance improvements. Is there anything in particular that you miss?
Aaron's work on using APL as a high-performance tool for manipulating tree structures demonstrates that APL is more relevant than ever before, as a programming language that has a natural mechanical sympathy with modern hardware, and allows relatively unskilled developers to write extremely efficient solutions without using complex tricks. I recently had the pleasure of sharing a podium with Aaron in Mumbai as part of the JioTalks series, where we covered some of the reasons why APL is a tool worth taking a look at in 2020, even though the core of the language is more than 50 years old: https://jiotalks.com/watch/204/home/Morten_Kromberg_&_Aaron_...
Thanks for asking!
Before Dyalog was acquired in 2005 and Gitte and I came in as CEO and CTO, the team working on Dyalog APL consisted of 5 people. Today that number is heading towards 25, several of whom were hired over the past decade to be ready to take over the work of Scholes and Streeter.
Gitte's comment about "not enough" young people signing up needs some context. The problem was actually caused by significantly increased interest in APL, which made us realise that our "classical" registration process was putting potential new users off. It was often taking too long to get people a copy of Dyalog APL because of the manual processing that was required, people were confused by the licensing terms and afraid of using the software – and many young people are just allergic to providing any information about themselves online. Now, APL is available for Windows, Linux (including the Pi) and macOS with no questions asked – for non-commercial use.
Losing Jay as CTO was a bit of a blow, but I think we are on the way to recovery. Now that he has completed his PhD on the subject of a parallelizing compiler (research that Dyalog has funded for the last 5 years), Aaron Hsu will join young guns Marshall Lochbaum and Adam Brudzewsky at Dyalog, and we have one more C developer joining the team in the UK in January. We hired two fresh APL consultants in the USA in 2019, and will be looking to hire two more in 2020 to support both existing and new business. I've been in the APL business for 40 years now, and my opinion is that the "kids" at Dyalog are every bit as impressive as Scholes and Streeter were at the same age; the shoes will be filled. There is as much young talent at Dyalog as there has ever been working on any APL interpreter, throughout the 50+ year history of the language.
As the team grows, we are able to afford having one or two team members focusing most of their efforts on performance. These efforts have accelerated over time, as first Nic Delcros, then Roger Hui and now Marshall have spent significant time rewriting primitive functions to take advantage of vector instructions and new hashing, sorting and other algorithms that have been evolved over the last four decades. Hardware has evolved in ways which mean that many of the old algorithms are sub-optimal on modern chips where the relative speed of CPU cycles vs memory access is completely different from the Z8000. Yes, Marshall is very good indeed, but I can't see a problem there . Performance has improved consistently in most of the last ten versions of Dyalog APL, and I hope we may see even more spectacular improvements in the next decade, both in the interpreter and in GPU-based parallel compilers.
We have much work to do: In addition to language and performance work, we expect Aaron to work with Richard Park, another new recruit who was added to the team last year, to work on training materials and documentation – and with Marshall, Roger and myself on quote evangelism unquote; talks at Lambdaconf, Functionalconf and perhaps some new conferences in 2020 and beyond (invitations welcome!).
For decades, Dyalog was a company that made a very comfortable living providing a better APL interpreter to clients who converted to Dyalog APL after initially purchasing an APL system from one of the companies that did serious marketing of APL interpreters: companies like IBM, STSC and I.P.Sharp Associates. Thanks to our recent growth, and the fact that Dyalog APL now has decent tooling not only under Microsoft Windows, but Linux and macOS too, we are now starting to focus on developing new business and attracting the next generation of APL developers.
I'm sorry to hear that we appear "directionless" to you. I will admit that it is an interesting challenge to keep existing customers satisfied (which should always remain the first priority for any organisation – put on your oxygen mask before helping others) while also working on attracting completely new groups of users. The rate of development of new language features, development tools and libraries for Dyalog APL is as high as it has been for any APL interpreter in history, at the same time as we are making significant performance improvements. Is there anything in particular that you miss?
Aaron's work on using APL as a high-performance tool for manipulating tree structures demonstrates that APL is more relevant than ever before, as a programming language that has a natural mechanical sympathy with modern hardware, and allows relatively unskilled developers to write extremely efficient solutions without using complex tricks. I recently had the pleasure of sharing a podium with Aaron in Mumbai as part of the JioTalks series, where we covered some of the reasons why APL is a tool worth taking a look at in 2020, even though the core of the language is more than 50 years old: https://jiotalks.com/watch/204/home/Morten_Kromberg_&_Aaron_...
Thanks for asking!
Hi Morten, I have watched your video of presenting with Aaron (and many more), taking an interest in APL for the past few months. It's been fun (if frustrating) to try and learn, so you mentioning training materials and documentation is particularly appealing.
> I'm sorry to hear that we appear "directionless" to you
I hoped for this to be quiet feedback in case you don't see it from inside, rather than criticism, I'm not a commercial user (yet?) and you have nothing to be apologetic about; I appreciate your detailed and comprehensive reply, am enthused about many of the things you've said, am glad Dyalog is thriving, and wish you all continued success!
> Is there anything in particular that you miss?
I miss the familiarity of an imperative/OOP language with intellisense completing the names of methods and ordering of parameters, but I'm sure that will pass with time. :)
> I'm sorry to hear that we appear "directionless" to you
I hoped for this to be quiet feedback in case you don't see it from inside, rather than criticism, I'm not a commercial user (yet?) and you have nothing to be apologetic about; I appreciate your detailed and comprehensive reply, am enthused about many of the things you've said, am glad Dyalog is thriving, and wish you all continued success!
> Is there anything in particular that you miss?
I miss the familiarity of an imperative/OOP language with intellisense completing the names of methods and ordering of parameters, but I'm sure that will pass with time. :)
Just a comment about the intellisense stuff. Dyalog's IDE(s) do provide name completion for methods, functions, variables, objects, and namespace bindings. You don't get any auto-complete for the ordering of parameters, but arguably, this is less of an issue with a 1 and 2-arity function requirement in which it is generally bad style to deviate too far from this for too long in too many places without good reason.
Thanks for taking the time to submit a lengthy reply Morten and I wish y'all the best of luck in 2020. I don't use APL/J/K professionally, but have played around with it for fun and found it to be quite enjoyable.
I keep up with Dyalog's twitter and conference videos, but wish there was a little more in the way of beginner material. The recent video on using the CSV/HTML/XML/JSON calls was great btw.
Two questions I have revolve around performance, numerical computing, and program distribution. To me, having to use Python + Numpy, Matlab, or low level languages for scientific and numerical computing is a shame as APL is basically math notation and should excel here. I hate wasting time writing loops (I parse lots and lots of text files) and how it makes my code harder to view and keep up with (APL fixes this). However, will Dyalog ever invest in writing some libraries or code snippets or primitives for things like sparse matrices and matrix factorizations that I probably don't want to implement myself and that make more sense to be done by the language implementers in C (I guess similar to how Dyalog has native support for JSON & CSV)? I'm guessing you can call out to BLAS/LAPACK (like J) or use Python or R's libraries via APL's bridge libraries, but that defeats most of the reason why I would want to use APL to begin with and I would end up with a franken-system and would be better off just using Python.
I guess what I'd like to see as a user is a way for me to write APL code in a tacit manner (I really like the organization of Aaron's compiler) and the compile it as a binary that could be shared with other users at work. It would be nice to have some built-in scientific primitives. Honestly though, the licensing is the real problem last I checked. It seems like running in production would be a non-starter, so I could only use as a single user tool (probably not worth the fight). I know this might be counter to how y'all currently make money, so I'm not exactly expecting a response on this note, but felt you should know what the few deal-breakers are from someone who has considered using it professionally but has shied away.
I keep up with Dyalog's twitter and conference videos, but wish there was a little more in the way of beginner material. The recent video on using the CSV/HTML/XML/JSON calls was great btw.
Two questions I have revolve around performance, numerical computing, and program distribution. To me, having to use Python + Numpy, Matlab, or low level languages for scientific and numerical computing is a shame as APL is basically math notation and should excel here. I hate wasting time writing loops (I parse lots and lots of text files) and how it makes my code harder to view and keep up with (APL fixes this). However, will Dyalog ever invest in writing some libraries or code snippets or primitives for things like sparse matrices and matrix factorizations that I probably don't want to implement myself and that make more sense to be done by the language implementers in C (I guess similar to how Dyalog has native support for JSON & CSV)? I'm guessing you can call out to BLAS/LAPACK (like J) or use Python or R's libraries via APL's bridge libraries, but that defeats most of the reason why I would want to use APL to begin with and I would end up with a franken-system and would be better off just using Python.
I guess what I'd like to see as a user is a way for me to write APL code in a tacit manner (I really like the organization of Aaron's compiler) and the compile it as a binary that could be shared with other users at work. It would be nice to have some built-in scientific primitives. Honestly though, the licensing is the real problem last I checked. It seems like running in production would be a non-starter, so I could only use as a single user tool (probably not worth the fight). I know this might be counter to how y'all currently make money, so I'm not exactly expecting a response on this note, but felt you should know what the few deal-breakers are from someone who has considered using it professionally but has shied away.
I'll comment a bit here. If you look into prior research, there are a number of papers on sparse representations in APL along with accompanying code in the literature. That's a good starting point.
Additionally, sparse matrices are on the roadmap for Co-dfns in the future, so there's hope for you there. If you have a specific problem that you wish would work, the best thing to do is to make it known to us so that we can actually prioritize it. I think myself and Dyalog often prioritize those things for which we know there are users. If you want to be able to do this stuff, it would be great to get some concrete programs that we can work off of.
Finally, licensing should, IMO, be a non-issue for most people in the vast majority of cases. Dyalog provides some of the best licensing terms that I've seen, and there are already models that scale well both to startups and to existing deployment in large-scale commercial enterprise situations. If something in the licensing terms doesn't seem to work, I would get in touch with Sales @ Dyalog and they will surely be able to work with you to figure out something that would be equitable.
Most people are surprised when they find out how easy it is to move forward with Dyalog on a commercial product, or how readily Dyalog is looking at making things work for customers or people who wish they could be customers.
Additionally, sparse matrices are on the roadmap for Co-dfns in the future, so there's hope for you there. If you have a specific problem that you wish would work, the best thing to do is to make it known to us so that we can actually prioritize it. I think myself and Dyalog often prioritize those things for which we know there are users. If you want to be able to do this stuff, it would be great to get some concrete programs that we can work off of.
Finally, licensing should, IMO, be a non-issue for most people in the vast majority of cases. Dyalog provides some of the best licensing terms that I've seen, and there are already models that scale well both to startups and to existing deployment in large-scale commercial enterprise situations. If something in the licensing terms doesn't seem to work, I would get in touch with Sales @ Dyalog and they will surely be able to work with you to figure out something that would be equitable.
Most people are surprised when they find out how easy it is to move forward with Dyalog on a commercial product, or how readily Dyalog is looking at making things work for customers or people who wish they could be customers.
Yeah I'll yield to your expertise in this matter.
Dyalog APL does seem to be nice.
Dyalog APL does seem to be nice.
Unfortunately I don't think Aaron has made any progress on coming up with an APL beginners book yet. I'd certainly be excited to buy a copy though. The only book I've read is the free one on Dyalog's website which is good, but doesn't really help me think in terms of arrays for everything.
Actually, there is movement in this general direction. My Thesis is now out and available publicly in its final version and should be indexed in the databases shortly entitled "A data parallel compiler hosted on the GPU." This provides a bit of an introduction to some aspects of APL, and includes a background on some of the ideas I've talked about before. Additionally, it is a "handbook" of sorts for doing tree transformations in APL, which is, to my knowledge, the first of its kind.
As for the beginner's book, I've come to the general feeling right now that a online video course will be a better fit for APL beginners than a textbook. This is an active project of mine that I anticipate completing some time within the year 2020.
You may also want to see my presentations on design patterns for APL and the upcoming videos at FnConf 2019 this year. I am not sure if the APL Training Wheels session was recorded, unfortunately, but the content from that work will be going into the video course.
As for the beginner's book, I've come to the general feeling right now that a online video course will be a better fit for APL beginners than a textbook. This is an active project of mine that I anticipate completing some time within the year 2020.
You may also want to see my presentations on design patterns for APL and the upcoming videos at FnConf 2019 this year. I am not sure if the APL Training Wheels session was recorded, unfortunately, but the content from that work will be going into the video course.
> As for the beginner's book, I've come to the general feeling right now that a online video course will be a better fit for APL beginners than a textbook.
Other than generating emotional responses, I don't find video better than text at anything, and most especially not technical education.
Other than generating emotional responses, I don't find video better than text at anything, and most especially not technical education.
Thanks for the thoughts. While I am sympathetic to your position (and I don't necessarily disagree), I'm not sure that the education problem with APL is strictly technical in nature. Nor am I convinced that what needs to be learned in order to use APL is strictly technical in nature.
Part of this is a question of what the stamina level is for the current computing student. With a lot of technical documentation that people are learning, they're essentially consuming either thorough reference materials or walkthrough/how-to materials.
We can characterize both of these materials as efficiently transferring a certain sort of information about how to do things by virtue of a very wide base of shared prior knowledge. This is what allows reference materials and walkthroughs to work.
But this shared base of knowledge does not exist for most people learning about APL. Target audiences include non-computing students who are just learning computing, programmers who want to learn APL, and technical professionals who have little or limited prior computer science knowledge (but perhaps cursory programming knowledge) who need to learn APL to accomplish the work they want to do in their domain. In the cases of the technical professional and the uneducated student, reference materials and most tutorial books are wholly inadequate.
For the programmer, these tutorials and references work because most of the time you are adding to your existing knowledge as a programmer and have already trained numerous soft skills that enable you to navigate the sea of potential directions to integrate the new knowledge.
I worked on my thesis to try to make it as accessible to the general programming community as possible to teach a set of algorithmic concepts that most programmers already have a vague notion about (but almost never use), namely, SIMD/data parallel programming, using an extremely simple and easy to explain syntax and vocabulary (APL). My thesis is already pretty detailed and illustrative, with worked examples for every single programming statement in the entire compiler going over examples and prose detailing everything that is going on, a sufficient background introduction not only to the syntax and semantics of APL, but also the design and stylistic questions, reference material, and the architectural education necessary to understand the structure of the compiler as a whole.
Even with all of this, presented to experts in the C.S. fields of PL and parallel computation, who have seen this material multiple times over, there was a general feeling that my thesis could have benefited from even more exposition.
My thesis is around 70,000+ words and around 250+ double-spaced pages. For 17 lines of APL code. People wanted more. Had I added the additional exposition desired, I imagine the thesis easily could have been 80 - 100k words.
A typical short novel is around the same size, and those often take about 8 - 12 hours to read through aloud. If you give me around that amount of time in classroom time, I could easily have almost any programmer fully understanding my compiler within that time, and probably with more thoroughness than if someone were to take the same amount of time to read the thesis.
There is something about the higher bandwidth and richness of full video content that contributes better to the transmission of data in some ways, particularly to the understanding of certain ideas and concepts.
I began to think about what it would take to put together an introduction to APL as a textbook. These have been done before and are out there, and they would generally work, but I'm not convinced anyone would read them.
You can teach APL the language to a sufficient level in a very short document (maybe a few pages) and in maybe 2 hours of classroom time. It's relatively trivial to teach the language, as it's very, very easy to understand the semantics and syntax. There's even a walkthrough of APL at http://tutorial.dyalog.com already.
The challenge comes in its application, and that's where the real value of a good education rests. But that part of the education is a lot harder to teach in text, because there is an element of style, aesthetic, and practice that goes into it. It's something that you gain by seeing, watching, and doing. You have to bring your mind into the space and struggle, go in and out of exploration mode and absorption mode.
How do you teach the exploratory process of APL? The design aesthetic? The set of idiomatic patterns? How do you get people to intern those concepts into their practice so that they are productive?
You could write a textbook to do this, no doubt. But thinking about it, I think such a textbook would be very large. It would be not very "exciting" emotionally. It would have very little of the emotional hooks that help a person stay engaged in things. And moreover, it would be extremely intimidating.
How many people effectively learn without any teacher in a totally "new" field from a textbook that is devoid of any human feedback? You can do it. I've done it before. But it's infinitely improved by the introduction of someone who can help to inject energy and dynamism into the space.
I just don't think people will want to pick up a 400 page "introductory" book and then have the motivation to stick with it when there are so many other shiny things in this world today. The truth is that the world is filled with distracted people who are very bad about putting the time into learning things. I want APL education to be engaging, accessible, and satisfying, to say nothing of effective. For most people today, a textbook is unlikely to be that.
Another big challenge with the "experienced programmer" target audience is that they are used to reference and tutorial materials. Very few programmers actually sit down and read such materials. They skim them to find what they think they want to know, and then plug that into their existing framework. That's great for efficiently extracting new information that is similar enough to their old practices to work. If something breaks, they can just go back and do a little extra skimming to clarify. That doesn't work really well when what you need to learn is fundamentally conflicting with or altering how you think about problems.
Getting good with APL at the beginning, before you've learned to intern it, is about undoing or unwinding previous habits and thought patterns that worked for you in past programming experiences, but that won't work for you in APL. After you have interned the new methods, then you can switch back and forth at your leisure, but until then, learning APL is as much about unlearning as it is about adding new knowledge. For programmers who are used to just skimming what they read to get what they want to know without actually listening to what the author is saying (and that is how most programmers read that I've met), this makes textbooks a recipe for disaster. Either the information has to be presented in such a dense way that most people will not be able to figure it out, or it has to be presented in such a measured, hand-holding way that most people will just ignore it, get bored with it, or skim it and still have the issues.
It does little good to have a full, thorough textbook introducing everything if no one reads it, though they think they've read it, and then they complain about how hard APL is to learn because they couldn't just take what they thought they wanted to know and leave the rest.
If you could get a programmer who was motivated, eager, and willing to read from cover to cover, stopping at each point where a hard question is asked, do the problems, quizzes, challenges, and exercises, in order, linearly, from start to finish, and actually pay attention to what was said when, and then take the points in the book that indicate one should think about this or that, I would agree with you that a textbook could have some potential benefits over the video format; but, I don't think that this is reality, based on my experience.
Your typical APL workshop for technical professionals is 4 days about about 24 hours of work with probably around 12 of those hours being expository. After that, people are quite solidly grounded in APL. But a lot of people have gone through that same amount of time on their own with plenty of the same resources available to them in terms of problems, documentation, and the like, and they haven't been as successful. Guided, in-person education is hard to beat, and video is the closest thing online.
I haven't given up on the hope of a great APL book, and I could be convinced by someone that I should do a book first, but at the moment, I feel that video courses are more likely to be more effective for more people and present a larger ROI in terms of the difficulty in producing and creating them compared to a book of similar effectiveness for the same range of people. If anyone has an alternative, please do speak up, I'm all ears.
Part of this is a question of what the stamina level is for the current computing student. With a lot of technical documentation that people are learning, they're essentially consuming either thorough reference materials or walkthrough/how-to materials.
We can characterize both of these materials as efficiently transferring a certain sort of information about how to do things by virtue of a very wide base of shared prior knowledge. This is what allows reference materials and walkthroughs to work.
But this shared base of knowledge does not exist for most people learning about APL. Target audiences include non-computing students who are just learning computing, programmers who want to learn APL, and technical professionals who have little or limited prior computer science knowledge (but perhaps cursory programming knowledge) who need to learn APL to accomplish the work they want to do in their domain. In the cases of the technical professional and the uneducated student, reference materials and most tutorial books are wholly inadequate.
For the programmer, these tutorials and references work because most of the time you are adding to your existing knowledge as a programmer and have already trained numerous soft skills that enable you to navigate the sea of potential directions to integrate the new knowledge.
I worked on my thesis to try to make it as accessible to the general programming community as possible to teach a set of algorithmic concepts that most programmers already have a vague notion about (but almost never use), namely, SIMD/data parallel programming, using an extremely simple and easy to explain syntax and vocabulary (APL). My thesis is already pretty detailed and illustrative, with worked examples for every single programming statement in the entire compiler going over examples and prose detailing everything that is going on, a sufficient background introduction not only to the syntax and semantics of APL, but also the design and stylistic questions, reference material, and the architectural education necessary to understand the structure of the compiler as a whole.
Even with all of this, presented to experts in the C.S. fields of PL and parallel computation, who have seen this material multiple times over, there was a general feeling that my thesis could have benefited from even more exposition.
My thesis is around 70,000+ words and around 250+ double-spaced pages. For 17 lines of APL code. People wanted more. Had I added the additional exposition desired, I imagine the thesis easily could have been 80 - 100k words.
A typical short novel is around the same size, and those often take about 8 - 12 hours to read through aloud. If you give me around that amount of time in classroom time, I could easily have almost any programmer fully understanding my compiler within that time, and probably with more thoroughness than if someone were to take the same amount of time to read the thesis.
There is something about the higher bandwidth and richness of full video content that contributes better to the transmission of data in some ways, particularly to the understanding of certain ideas and concepts.
I began to think about what it would take to put together an introduction to APL as a textbook. These have been done before and are out there, and they would generally work, but I'm not convinced anyone would read them.
You can teach APL the language to a sufficient level in a very short document (maybe a few pages) and in maybe 2 hours of classroom time. It's relatively trivial to teach the language, as it's very, very easy to understand the semantics and syntax. There's even a walkthrough of APL at http://tutorial.dyalog.com already.
The challenge comes in its application, and that's where the real value of a good education rests. But that part of the education is a lot harder to teach in text, because there is an element of style, aesthetic, and practice that goes into it. It's something that you gain by seeing, watching, and doing. You have to bring your mind into the space and struggle, go in and out of exploration mode and absorption mode.
How do you teach the exploratory process of APL? The design aesthetic? The set of idiomatic patterns? How do you get people to intern those concepts into their practice so that they are productive?
You could write a textbook to do this, no doubt. But thinking about it, I think such a textbook would be very large. It would be not very "exciting" emotionally. It would have very little of the emotional hooks that help a person stay engaged in things. And moreover, it would be extremely intimidating.
How many people effectively learn without any teacher in a totally "new" field from a textbook that is devoid of any human feedback? You can do it. I've done it before. But it's infinitely improved by the introduction of someone who can help to inject energy and dynamism into the space.
I just don't think people will want to pick up a 400 page "introductory" book and then have the motivation to stick with it when there are so many other shiny things in this world today. The truth is that the world is filled with distracted people who are very bad about putting the time into learning things. I want APL education to be engaging, accessible, and satisfying, to say nothing of effective. For most people today, a textbook is unlikely to be that.
Another big challenge with the "experienced programmer" target audience is that they are used to reference and tutorial materials. Very few programmers actually sit down and read such materials. They skim them to find what they think they want to know, and then plug that into their existing framework. That's great for efficiently extracting new information that is similar enough to their old practices to work. If something breaks, they can just go back and do a little extra skimming to clarify. That doesn't work really well when what you need to learn is fundamentally conflicting with or altering how you think about problems.
Getting good with APL at the beginning, before you've learned to intern it, is about undoing or unwinding previous habits and thought patterns that worked for you in past programming experiences, but that won't work for you in APL. After you have interned the new methods, then you can switch back and forth at your leisure, but until then, learning APL is as much about unlearning as it is about adding new knowledge. For programmers who are used to just skimming what they read to get what they want to know without actually listening to what the author is saying (and that is how most programmers read that I've met), this makes textbooks a recipe for disaster. Either the information has to be presented in such a dense way that most people will not be able to figure it out, or it has to be presented in such a measured, hand-holding way that most people will just ignore it, get bored with it, or skim it and still have the issues.
It does little good to have a full, thorough textbook introducing everything if no one reads it, though they think they've read it, and then they complain about how hard APL is to learn because they couldn't just take what they thought they wanted to know and leave the rest.
If you could get a programmer who was motivated, eager, and willing to read from cover to cover, stopping at each point where a hard question is asked, do the problems, quizzes, challenges, and exercises, in order, linearly, from start to finish, and actually pay attention to what was said when, and then take the points in the book that indicate one should think about this or that, I would agree with you that a textbook could have some potential benefits over the video format; but, I don't think that this is reality, based on my experience.
Your typical APL workshop for technical professionals is 4 days about about 24 hours of work with probably around 12 of those hours being expository. After that, people are quite solidly grounded in APL. But a lot of people have gone through that same amount of time on their own with plenty of the same resources available to them in terms of problems, documentation, and the like, and they haven't been as successful. Guided, in-person education is hard to beat, and video is the closest thing online.
I haven't given up on the hope of a great APL book, and I could be convinced by someone that I should do a book first, but at the moment, I feel that video courses are more likely to be more effective for more people and present a larger ROI in terms of the difficulty in producing and creating them compared to a book of similar effectiveness for the same range of people. If anyone has an alternative, please do speak up, I'm all ears.
An interesting perspective. I personally didn't enjoy lectures in college when it came to technical classes (engineering, math, physics) as I had trouble keeping up. Videos help solve that though as you can go at your own pace...pause and rewind. I'm sure it will be a lot faster and reach more people faster with videos. There are so few in the APL/J world. The few ones that exist are very popular though.
Rest assured that I haven't given up on the textbook / tutorial approach just yet, but I might have to "invent" my own publication technology to get the "book" that I want.
Would it be hard to just create a Dyalog-APL notebook (I recall Dyalog added support for that recently) and export as PDF?
I find Jupyter notebooks in Python and Wolfram's Mathematica to be nearly perfect for explaining complicated content where you can easily format text, images, and inline code snippets.
I find Jupyter notebooks in Python and Wolfram's Mathematica to be nearly perfect for explaining complicated content where you can easily format text, images, and inline code snippets.
I have started something along these lines.
github.com/rikedyp/learnapl
It is currently very rough around the eges and also in the middle. Work will continue in the new year.
There is also now a set of introductory problem sets that have been used in a couple of schools on problems.tryapl.org
github.com/rikedyp/learnapl
It is currently very rough around the eges and also in the middle. Work will continue in the new year.
There is also now a set of introductory problem sets that have been used in a couple of schools on problems.tryapl.org
Thanks for the response Aaron. I'll make sure to give the videos a go when you release them.
A shot in the dark, but learning Clojure got me thinking in terms of sequences for everything. Not quite the same, but I suspect there'd be non-trivial transfer.
AFAIK APL itself is easy and beautiful but bending you mind to interpret complex logic in its model can be tricky.
It's really no more difficult than learning recursion, pointers, memory allocation, type theory, functional programming, iterative programming, OOP, or any other of the myriad techniques that we use for computing. As I mention in my talk on APL design patterns, it's largely a matter of what we have been trained to understand, and there is some evidence to suggest that APL is actually arranged in a manner more natural for the human faculty than most computing languages:
https://www.youtube.com/watch?v=v7Mt0GYHU9A&feature=youtu.be
https://www.youtube.com/watch?v=v7Mt0GYHU9A&feature=youtu.be
> It's really no more difficult than learning recursion, pointers, memory allocation, type theory, functional programming, iterative programming, OOP, or any other of the myriad techniques
You don't really need anything of this to solve a problem with Python and Pandas.
I always feel like I wish I'd solve it with APL (because it's beautiful) but end-up just getting the thing done this way :-]
Thank you for the lecture, perhaps it is going to inspire me to actually learn to solve real problems with APL.
You don't really need anything of this to solve a problem with Python and Pandas.
I always feel like I wish I'd solve it with APL (because it's beautiful) but end-up just getting the thing done this way :-]
Thank you for the lecture, perhaps it is going to inspire me to actually learn to solve real problems with APL.
Except that you do. Python is a form of OOP/Iterative programming, involves memory allocation, pointers, references, and the like. Pandas requires the skilled application of a general purpose language coupled with a domain-specific vocabulary for your problem space.
I maintain that APL isn't fundamentally more difficult. If there is something that you have as an example that you think is just too "hard" to do in APL that is very "easy" in Python and Pandas, please do let me know via email ([email protected]) as I would love to make sure APL has a good answer to such things (I want to make APL accessible).
Since you are talking about statistics and data analysis type things, you may find TamStat of interest:
http://www.tamstat.com/
Additionally, don't forget that Dyalog APL also has a full Chart/Graphics system (two, actually), and a large suite of idiomatic expressions for computing many classic analysis problems. There's a number of other integrations that people seem to be unaware of often enough.
I very seriously would like to receive from you one example of something you have done that you find too difficult in APL, because if it really is, then we can do better and I'd like to make sure that we do.
I maintain that APL isn't fundamentally more difficult. If there is something that you have as an example that you think is just too "hard" to do in APL that is very "easy" in Python and Pandas, please do let me know via email ([email protected]) as I would love to make sure APL has a good answer to such things (I want to make APL accessible).
Since you are talking about statistics and data analysis type things, you may find TamStat of interest:
http://www.tamstat.com/
Additionally, don't forget that Dyalog APL also has a full Chart/Graphics system (two, actually), and a large suite of idiomatic expressions for computing many classic analysis problems. There's a number of other integrations that people seem to be unaware of often enough.
I very seriously would like to receive from you one example of something you have done that you find too difficult in APL, because if it really is, then we can do better and I'd like to make sure that we do.
Just seeing this. I'm not the original poster that you replied to, but their point is somewhat valid as is yours.
Python certainly requires one to learn iterative programming which is definitely not a walk in the park. I learned it over a decade ago, but vividly recall how non-intuitive it was at the time (global versus local scope had me scratching my head once upon a time). Then there is lists, tuples, dictionaries, pass by reference, iterators, file I/O...just soo much to learn.
One interesting thing is that I used OO methods in Python for ages before understanding how to write and use classes. To me, a = list.sort() was just a weird way to read a = sort(list). You can almost write Python entirely in an imperative manner and for short scripts I almost always do.
Learning all the OO aspects of Java/C# and all the fundamentals of FP (monads, currying... etc) is much harder for me than APL which is basically just learn what these symbols do and the very straightforward way to combine them and you're done. The only harder part of APL is somehow learning to replace imperative programming with array operations. Sure it is easy to do some things like sum up an array (+/ array), but that is also easy in Python (list.sum()). The idioms as APL people call them are not at all obvious to me, nor is the path of enlightenment outside of stop talking and go write some APL haha.
Python certainly requires one to learn iterative programming which is definitely not a walk in the park. I learned it over a decade ago, but vividly recall how non-intuitive it was at the time (global versus local scope had me scratching my head once upon a time). Then there is lists, tuples, dictionaries, pass by reference, iterators, file I/O...just soo much to learn.
One interesting thing is that I used OO methods in Python for ages before understanding how to write and use classes. To me, a = list.sort() was just a weird way to read a = sort(list). You can almost write Python entirely in an imperative manner and for short scripts I almost always do.
Learning all the OO aspects of Java/C# and all the fundamentals of FP (monads, currying... etc) is much harder for me than APL which is basically just learn what these symbols do and the very straightforward way to combine them and you're done. The only harder part of APL is somehow learning to replace imperative programming with array operations. Sure it is easy to do some things like sum up an array (+/ array), but that is also easy in Python (list.sum()). The idioms as APL people call them are not at all obvious to me, nor is the path of enlightenment outside of stop talking and go write some APL haha.