The reason it is easier for programmers to transform into software engineers is the same reason we are having this conversation. For the most part, I think this is because software is so cheap and accessible. Anyone can become a software artisan with just a laptop and time. For traditional engineering, you need resources.
Here's a quote from the second post:
"When it comes to things like making circuit boards, it’s pretty common for things not to work the first time. You have to do it again, and you have to send it out to the factory and do it again. You know it costs you another however many £1000 and another two weeks on the schedule. -Mike (electrical)"
If building software was expensive like engineering in the physical world, we wouldn't have to have this argument. It's too bad that physical engineering is so expensive, because we'd probably have a lot more people trying it out.
I thought I was over this argument because it always hits a dead end, but Hillel took a really entertaining approach with this series.
Looking back, I'm almost always having this argument with other software people, but never traditional engineers. I hadn't thought much about how our stereotypes are wrong.
A great example from the second post in the series is how software people don't have experience with the unpredictability of traditional engineering projects: "Part of this misconception comes from us seeing the results of the engineering process, not the process itself. We don’t see the friction, the overruns, the delays that happen because someone built the wall one inch to the left, or when a critical supplier goes out of business. To assume that software is uniquely unpredictable is a special kind of arrogance."
This has me wondering why the software world is so unpredictable in the first place and why we aren't working on that? Or at least getting better at dealing with it. Also, why are we so inclined to think the physical world is so predictable? Probably because we spend so much time on our computers...
There seem to be a lot of components like this that need to be developed.
For example, there are plenty of tools to create ML model endpoints (send data, receive prediction) but many times, those models are not customer/user facing. They will be hit by some other program/system (often on a regularly timed schedule).
One solution is to embed the model in the application that needs the predictions, but this can get hairy when it comes to updates/maintanence/etc.
Over the next few years, we will see some generalized "model invocation" components that integrate with common data sources for these use cases.
Have you shared the code or any more details for that custom search engine anywhere? I'd love to see how you did that. I'm considering doing the same thing soon
my biggest problem with this idea is that many people are already hesitant to write/post online. limiting the number of posts they are allowed to make might make their post anxiety even worse.
I'd love a high latency app as well, but there is undeniable magic in instant communication. My best writing is often when I'm "hot" on a topic and can quickly go back and forth with others on the idea. Steam could quickly run out for that type of thing without instant communication.
Turns out an easy google search answered my question as well... this post https://medium.com/distributed-computing-with-ray/ray-for-th... does say that ray uses the actor model, and ray is listed on The Actor Model wikipedia page as an Actor Model library.
Snapchat also has some sort of phenomena that makes it feel like a much lower barrier to entry than a text message.
Snapping a new person is much lower pressure than texting them.
Pictures also make it much easier for people to be involved, because you don't need to be so descriptive with your writing. That is part of the reason snap groups tend to be so much more active on snap than text groups.
There really needs to be some sort of mix between chat and a collaborative note-taking app.
How can we transform chats into a reasonable, permanent reference materials?
Maybe it would be possible to build a slack-bot that could identify which part of these threads are the most important, and even move them to a more permanent storage.
Or, slack channels could add functionality to allow users to vote for which messages are most pertinent, or should be moved to longer-term storage or a featured position for everyone.
I think there is probably value in these rapid fire IM's, but I feel the Mike's (the author's) pain. I skip over threads everyday because reading them never feels worth it.
I recently started sending long form voice recordings to friends as a new way to stay connected (I explain that idea here https://nicktorba.com/brain/voice-memo-experiment). I'm excited to send those friends a link to this article as a way to further convince them they should send recordings back.
I did not expect the personal thought benefits that come from speaking to yourself in long form. After starting to send recordings to friends, I quickly became more comfortable speaking for long periods and found it led to more interesting ideas and better understanding of my thoughts and ideas. Plus, it helped me better structure thoughts verbally. I've also started making a recording right after I listen to any podcast. It has made listening to them much more engaging.
I was also surprised by how energizing recording audio is. I find myself refreshed after going for a walk and recording my thoughts, or creating a recording for a friend (much more so than if I listened to a podcast on the same walk, or just walked in silence). It's also a fantastic way to make time in the car creative and productive.
Voice recording or self talking is definitely an underutilized tool for thought and creation.
Based on this thread, the comparison should include
* metaflow (model training on AWS Batch)
* polyaxon (model training on kubernetes)
* pachyderm (experimentation)
* hopsworks (model training/serving/ and more, mostly on kubernetes)
* cortex (model serving on kubernetes)
* seldon-core (model serving and monitoring on kubernetes)
and likely more that I missed.
I can see why it would be so hard to put together this comparison.
Even with all these tools, there is still a lot of manual work for data scientists or DevOps engineers the data scientists pass their models off to.
It also seems there is yet to be a fully open source DevOps stack. Most companies still build custom software to glue together manual processes (like integrations between different tools for training, deploying, monitoring, etc). This could be one factor why more comparisons of these tools and stack discussions have not been more popular - they can't share them yet.
I've played with cortex before, and it is easy to use, but I am still questionable if automating kubernetes deployments through an easy code interface, without much kubernetes know-how, is safe.
In my experience, even when you have a tool automating a lot of kubernetes for you, you will still run into trouble that will be best handled if you are familiar with kubernetes.
I'm not sure what debugging utilities cortex has, but I think the ultimate solution to this problem will be a tool that truly allows users to not think about the fact their deployments are running on kubernetes at all.
I'm also interested in the similarities of Cortex and Seldon-core. Of course, seldon-core does not automate infra provisioning, but based on my previous point, I think many teams are better off being more hands on with this infra.
Lastly, there is a third tool missing from the mix - monitoring. I think cortex offers some tools in this area, but I wish they would make a part two showing how the monitoring functionality they offer can integrate into a retraining pipeline within metaflow. This post shows you how to get started, but it doesn't show you how to maintain applications long term.