Fair. I agree Arrow is still more of a vision than anything else.
> it didn't even provide a shape attribute
I suspect this has to do with the project's focus. I think they aspire to be a back-end to DataFrame libraries, which are generally 2d. I think they (correctly) are ceding the "n-dimensional tensor computation" space to the current incumbents.
What is your point? (I honestly do not understand.)
Blackbear's comment was that writing libraries in C allows those libraries to be deployed broadly in many compute environments.
Jakob's reply (as I understood it) was that outside of the big Deep Learning libraries, this has not really happened. There is no C implementation of Pandas that allows for redeployment in other non-python compute contexts.
My point was that, with Arrow, this type of cross platform compatibility is coming to python dataframe libraries. You can prototype Dask code that runs on your laptop, then deploy it to a production Spark cluster, knowing the same Arrow engine is underpinning both. Or at least that's the vision. Obviously Arrow is still relatively young. But the point is, it's far from certain that the long-term global optimum for the ecosystem isn't sticking with "all libraries are written in C".
> for library writers, how many people might use it can be a deciding factor on investing the time
Yes. Especially given that a high percentage of library development these days is being done by industry, not just academia. (At least in machine learning.) They're not writing packages for fun or for academic citations, they want users.
Makes sense. I don't use TF Eager, but I do use Jax, and Jax lets you arbitrarily compose JITed and non-JITed code, which made me think that might be a viable pattern. I guess I wondered if there might be something like "nonstatic_jit(foo)" that would do "julia style" compiling on function foo, in addition to "jit(foo)" that compiles foo to optimized XLA ops. Probably impractical. Thanks.
If the next machine learning killer-app model requires autodiff'ed dynamic control flow, do you think Google/Facebook will build that capability into XLA/TorchScript? Seems like if NLP SOTA requires dynamic control flow, Google will build it? Maybe they let you declare some subgraph as "dynamic" to avoid static optimizations? But maybe the static graph assumption is so deeply embedded into the XLA architecture, they'd be better off just adopting Julia? (I honestly don't know the answer, asking your opinion!)
Always happy to help. Especially the last day or so - I've been waiting on some long training loops so it's been a pleasant diversion.
To be 100% honest with you, there's pretty much 0% chance of me adopting Julia in the next 12 months. I evaluated it before embarking on my current project, but ended up going with Python, and now I have several thousand lines of Python code that work fine, and I'm not going to rewrite it all in the near future. At some point in the medium term, I'll re-evaluate Julia. But until then, I don't want to lead anyone on any wild goose chase. Even if you solved this problem, and all my problems, I'm just not in a position to switch right now. So for that reason I think I'll hold off on issuing a community wide call for help. But I'm cautiously optimistic that at some point in the future I'll be writing Julia professionally.
A lot of this is also probably cultural rather than language features. The first thing they teach any Python data science boot camp initiate is: never "from numpy import *", always "import numpy as np" and yet in Julia "using" appears more common than "import"...
I also wouldn't read too much into my one example. It was initially meant just as an illustrative point, but somehow I was so bad at explaining it that it took tons of comments for me to get my minor point across. I do think the MLJ guys are properly on the right track, and that should work fine for most people who don't mind Macros. Maybe I'm in the minority in hating Macros.
The more commonly cited issues around compile time caching, tooling, etc. are boring to list off yet again, but probably the right areas of focus for the community, in terms of both impact and ease of implementation.
More generally, I really do think you're better off talking to people like @systemvoltage, who have actually given Julia more of a chance than I have. If I worked for Julia Computing I'd be reaching out to him and begging to get his brain dump on all the challenges he faced. In any business, it's always easier to reduce churn and learn from your (unhappy) customers, then it is to convert new prospects, whether that business is programming languages or investment banking.
> since `fit` would've likely come from the same source
No.
As described in the great-great-great-great-great-great-great-great-great-parent comment, the problem I am describing is that of trying to combine models from multiple software authors. You may not have that problem. It may not be a common problem among Julia's academic user base. I do have that problem.
Just to be clear, I like Julia, and think it has advantages over Python. I'm writing all this as someone who is cheering for Julia to break out of its HPC niche. Thanks.
I know in Julia I can just precede every function call and object instantiation with "modulename." and solve the namespace problem that way. What I want to do instead, what I do in python, is bind one namespace of function methods to each object, so that as I code, I don't have to remember which module each each object came from. That is the appeal to me of "model.fit" over "module.fit(model)".
EDIT: This is not some "shave a few seconds off coding time" quality of life issue. This is a mission critical requirements in many enterprise contexts.
Scenario A: Model Development Team trains a model, serializes it, and sends the serialized model object to Production. You want Production to have to lookup which software Package the Model Development Team used for each model, just so they know which "predict" function to call? No, "predict" should just be packaged with the model object as a method, along with "retrain", "diagnose", etc.
Scenario B: I want to fit a dozen different types of models, from multiple packages, on the same data, to evaluate how they each do, and build build some sort of ensemble meta model. So I need to loop over each model object in my complicated ensembling logic. You want me to also programmatically keep track of which module each model comes from?
These are important, happens-every-day use cases in the enterprise.
I think the best solution for this in Julia is to package the model state with all the "method" functions in one struct. Again, this is what MLJ does. This is the closest Julia gets to OOP. But then you either need a lot of boilerplate code, or a Macro to unpack it all for you behind the scenes.
Never made it that far. This was a feature I use all the time in Python ML development (both consuming open source packages via an OOP interface and also writing in-house model classes) that I consider essential for my productivity and that Julia was missing.
<edit>I'm also nervous of relying on the recourse of asking package maintainers to edit their variable names to improve compatibility with the random third package I want to use; maybe the culture is different in Julia but in Python that's a good way to get laughed out of the issue tracker :) </edit>
If I were you I would maybe ask people like @systemvoltage who took the plunge and wrote a big project in Julia only to find they had trouble maintaining the project. Maybe one reason he can't upgrade without breaking everything is because of namespace collisions amongst his many dependencies? If not that, it's something like that.
I know nothing about Lisp, so at the risk of talking past eachother....
I never said macros were required. I said implementing this type of code without OOP required more boilerplate, and MLJ uses macros to reduce that boilerplate.
As I understand module imports in Julia: Each module developer exports a list of publicly facing objects. Obviously "fit" and "model" would be among them. If you import two modules that both export a new "nn" subtype of shared parent type "model", and both extend "fit" and "predict" and etc to accept their own subtype "nn", then you have to manually specify which module you are referring to every time you call nn, or fit, or predict, or whatever. Is that wrong? If I just import PyTorch, import TensorFlow, and then call "mymodel = TensorFlow.nn; fit(mymodel, mydata)" then Julia doesn't know that the "fit" I am calling is the TensorFlow implementation and not the PyTorch implementation; what if I had WANTED to use module A's "fit" on module B's model, and they intentionally adopted the same abstract type system to enable this interoperability? So instead I have to write "mymodel = TensorFlow.nn; TensorFlow.fit(mymodel, mydata); TensorFlow.predict(mymodel, mynewdata)". Obviously the extra typing is mildly annoying but the bigger problem is potentially introducing bugs by mismatching modules, and the developer's cognitive overload of having to keep track of modules. Python style OOP is a more elegant solution to the namespace problem and results in more readable, maintainble code, at least in my opinion. Anyways, maybe Julia has a more elegant solution I'm not aware of, if so I'd love to hear it.
I agree you can achieve same benefits with Macros. Indeed, I see that MLJ, Julia's attempt at a SciKit type project, makes extensive use of Macros. But I personally think macros are an antipattern. In large projects, they can introduce subtle bugs. Especially if you're using multiple modules that are each editing your code before compile time and that don't know about each other. I know others in Julia community agree that Macros are dangerous.
I think abstract types are a brittle solution. The "can of worms" I alluded to is something like this: Library TensorFlow implements model "nn" and Library PyTorch also implements model "nn" and they both want to override "fit" to handle the new type "nn"... Good luck combining them in the same codebase. This problem is less pronounced in OOP where each development team controls their own method. Julia devs can solve this by having every developer of every "fit" function and every developer of every "model" struct agree beforehand on a common abstraction, but that's an expensive, brittle solution that hurts innovation velocity.
I think the closest I can do in Julia via pure structs is for the developer to define and expose their preferred fit function as a variable in the struct, something like "fit = model['fit_function']; fit(model,X,Y)" but that introduces a boilerplate tax with every method I want to call (fit, predict, score, cross validate, hyperpameter search, etc). (EDIT: indeed, I think this is pretty much what MLJ is doing, having each model developer expose a struct with a "fit" and "predict" function, and using the @load macro to automate the above boilerplate to put the right version of "fit" into global state when you @load each model... but as described above, I don't like macro magic like this.)
I'll give you my two cents, recognizing that I very well might just be ignorant about Julia and multiple dispatch, and if so please continue to educate me.
Consider if we want to run many different types of models. Logistic regression, gradient boosting, NNs, etc. We want the ability to easily plug in any type of model into our existing code base. That's why model.fit(X,Y) is attractive. I just need to change "model = LogisticRegressionModel" to "model = GradientBoostingModel" and the rest of the code should still Just Work. This is a big part of SciKit's appeal.
But all these different models have very different training loops. So with "fit!(model,X,Y)" I need to make sure I am calling the compatible "fit" function that corresponds to my model type.
You might now say "Ah! Multiple dispatch handles this for you. The 'fit' function can detect the type of its 'model' argument and dispatch execution to the right training loop sub function." And I suppose that's theoretically correct. But in practice I think it's worse.
It should be the responsibility of the developer of "model" to select the "fit" algorithm appropriate for "model." (They don't have to implement it, but they do have to import the right one.) The developer of "fit" should not be responsible for handling every possible "model" type. You could have the developer of "model" override / extend the definition of "fit" but that opens up its own can of worms.
So is it possible to do the same thing with "fit!(model,X,Y)"? Yes of course it is. It's possible to do anything with any turing complete language. The point is, which system provides the best developer ergonomics via the right abstractions? I would argue, in many cases, including this one, it's useful to be able to bundle functions and state, even if that is in theory "less flexible" than pure functions, because sometimes programming is easier with less flexibility.
A lot of problems are fixable with time and money. Maybe the Series A will help!
But some problems might be related to Julia's design choices. One thing I really missed in Julia is Object Oriented Programming as a first class paradigm. (Yes I know you can cobble together an approximation with structs.)
OOP gets a lot of hate these days. Mostly deserved. But in some large complex projects it's absolutely the right abstraction. I've used OOP in several Python projects. Most of the big Python DS/ML packages use OOP.
Maybe you think PyTorch, SciKit, etc are all wrong, and would be better off with a more functional style. I know it's fashionable in some circles to make fun of the "model.fit(X,Y); model.predict(new_data)" style of programming but it works well for us busy professionals just trying to ship features.
I don't think Julia is wrong for enforcing a more functional style. It probably makes it easier for the compiler to generate great performance.
But Python has achieved its success because of its philosophy of being the "second best tool for every job" and that requires a more pragmatic, multiparadigm approach.
I am a python developer who has dabbled with Julia but it never stuck for me.
I think Julia was built by academics for other academics running innovative high performance computing tasks. It excels at the intersection of 1) big data, so speed is important, and 2) innovative code, so you can't just use someone else's C package. Indeed, Julia's biggest successful applications outside academica closely resemble an academic HPC project (eg Pumas). I think it will continue to have success in that niche. And that's not a small niche! Maybe it's enough to support a billion dollar company.
But most of us in industry are not in that niche. Most companies are not dealing with truly big data, on our scale, it is cheaper to expand the cluster than it is to rewrite everything in Julia. Most who ARE dealing with truly big data, do not need innovative code; basic summary statistics and logistic regression will be good enough, or maybe some cloud provider's prepackaged turn key neural nets scale out training system if they want to do something fancy.
I think for Julia to have an impact outside of academia (and academia-like things in industry) it will need to develop killer app packages. The next PyTorch needs to be written in Julia. Will that happen? Maybe! I hope so! The world would be better off with more cool data science packages.
But I think the sales pitch of "it's like Pandas and scikit but faster!" is not going to win many converts. So is Jax, Numba, Dask, Ray, Pachyderm, and the many other attempts within the Python community of scaling and speeding Python, that require much less work and expense on my part for the same outcome.
Again, congrats to the team, I will continue to follow Julia closely, and I'm excited to see what innovative capabilities come out of the Julia ecosystem for data scientists like me.
Did PyTorch increase or decrease demand for ML developers?
Did Ruby on Rails increase or decrease demand for web developers?
Did C increase or decrease demand for systems programmers?
Automation and abstraction is a complement, not a substitute.
If CoPilot works and we get to focus on higher level goals and developer throughout increases, it will be good for developers and good for the world. I hope CoPilot succeeds.
https://www.slideshare.net/wesm/pycon-colombia-2020-python-f...
Slide 43: The "Arrow C++ Platform" encompasses a "Multi-core Work Scheduler" and a "Query Engine"
Slide 38: "It would be more productive (long-term) to have a reusable computational foundation for data frames"
Again, I agree that, today, it's more data format, and the shared compute stuff is more a vision.
EDIT: See also https://ursalabs.org/tech/