Tackling Webdev as a Bioinformatician: why is it so hard?(jessimekirk.com)
jessimekirk.com
Tackling Webdev as a Bioinformatician: why is it so hard?
https://jessimekirk.com/blog/web_for_biofx/
193 comments
There used to be a time when web frameworks strived for simplicity.
I fully agree with the article's opinion that things have become needlessly complicated.
The new best practice seems to be to install NodeJS, then use the npm repository manager to install bower, then use that to download your packages. And 5000 dependencies and 500 MB in downloads later, you have bootstrapped a Hello World template.
Or name me one person who can, from memory, correctly configure the rails sprockets asset pipeline :p
There's no denying that most of the tutorials and contemporary web development advice boils down to "follow these instructions carefully + magic happens"
I fully agree with the article's opinion that things have become needlessly complicated.
The new best practice seems to be to install NodeJS, then use the npm repository manager to install bower, then use that to download your packages. And 5000 dependencies and 500 MB in downloads later, you have bootstrapped a Hello World template.
Or name me one person who can, from memory, correctly configure the rails sprockets asset pipeline :p
There's no denying that most of the tutorials and contemporary web development advice boils down to "follow these instructions carefully + magic happens"
This 100%. The vast majority of websites (not webapps) don't need anything more than HTML, CSS and some JS/JQuery.
> Or name me one person who can, from memory, correctly configure the rails sprockets asset pipeline
I feel that one, it's insane that the most basic "I need to execute a few lines of very specific JS on this specific page" must devolve into a DRY absolutist abstraction madness, previously with sprockets, now with webpacker.
> Or name me one person who can, from memory, correctly configure the rails sprockets asset pipeline
I feel that one, it's insane that the most basic "I need to execute a few lines of very specific JS on this specific page" must devolve into a DRY absolutist abstraction madness, previously with sprockets, now with webpacker.
Your second paragraph neatly sums up the state of things. It's gotten downright cartoonish.
I remember when Rails first hit the ground. It was touted as a revolution in Web simplicity (and was relative to then, when what we expected of our apps was relatively simple).
But, someone got it in their heads that web apps should be near-indistinguishable from desktop apps. But, let's not address the impedance mismatch there. Instead, let's create a bunch of layers out of new tools and concepts to obscure the fact that these things don't fit together.
So our projects/apps now demand more and have just grown extremely complex. Turns out that making the Web act just like a native app (while battling all its Web semantics) is hard. We want it to be X, when all it wants to be is the Web.
Add to this all of the other disparate pieces that we want to work together: countless technologies, languages, libraries, 3rd party services, etc. And if you're talking full stack (and who's really not fullstack these days?), the number of universes that you must make coexist is insane. And, much of this is to layer over existing technologies (like Rails) that weren't intended to be used in this way. Enter entire concepts like transpiling, webpacking, etc.
Then to deploy, You're probably dealing with no less than 6-10 cloud provider services, each with its own set of concepts and requirements to be satisfied. Even if you're not the one deploying, you have to understand their implications on your code.
Web development is now less about writing logic and much more about wiring a bunch of existing stuff together, with discrete chunks of logic serving as the business/domain layer glue. I mean the app still has to serve some business purpose, so the code that supports that must be written. But that's barely the start these days.
I remember when Rails first hit the ground. It was touted as a revolution in Web simplicity (and was relative to then, when what we expected of our apps was relatively simple).
But, someone got it in their heads that web apps should be near-indistinguishable from desktop apps. But, let's not address the impedance mismatch there. Instead, let's create a bunch of layers out of new tools and concepts to obscure the fact that these things don't fit together.
So our projects/apps now demand more and have just grown extremely complex. Turns out that making the Web act just like a native app (while battling all its Web semantics) is hard. We want it to be X, when all it wants to be is the Web.
Add to this all of the other disparate pieces that we want to work together: countless technologies, languages, libraries, 3rd party services, etc. And if you're talking full stack (and who's really not fullstack these days?), the number of universes that you must make coexist is insane. And, much of this is to layer over existing technologies (like Rails) that weren't intended to be used in this way. Enter entire concepts like transpiling, webpacking, etc.
Then to deploy, You're probably dealing with no less than 6-10 cloud provider services, each with its own set of concepts and requirements to be satisfied. Even if you're not the one deploying, you have to understand their implications on your code.
Web development is now less about writing logic and much more about wiring a bunch of existing stuff together, with discrete chunks of logic serving as the business/domain layer glue. I mean the app still has to serve some business purpose, so the code that supports that must be written. But that's barely the start these days.
Is anyone still using bower?
Also webpacker for rails is somehow both insanely complex and alright to use.
Also webpacker for rails is somehow both insanely complex and alright to use.
A perfect proof for GP's/TFA's point.
To this I'll only add that "follow these instructions carefully + magic happens" works only for ~6 months after a tutorial is published, longer than that and some dependency or toolchain update somewhere is bound to break the instructions in a way incomprehensible to a novice.
To this I'll only add that "follow these instructions carefully + magic happens" works only for ~6 months after a tutorial is published, longer than that and some dependency or toolchain update somewhere is bound to break the instructions in a way incomprehensible to a novice.
Not only to a novice. I was recently contracted to update a client from rails 4 to 5.2, which included updating sprockets, which ended with us downloading the source code of all the gems to because nobody could figure out why the html would contain the correct url for the asset, but rake would first produce the file during asset precompilation and then later delete it during cleanup. 2 days wasted on broken magic...
Oh I agree, my current preferred way to use Rails is as an API with no frontend at all.
Fun bit of trivia: the first two major web libraries - cgi-lib.pl and CGI.pm - were written by bioinformaticians. Steven E. Brenner and Lincoln Stein, respectively.
Cgi.pm was great, really nice little abstraction. Create react app is also a really nice abstraction, although I wouldn’t call it little :-)
> It is a genuine skillset and people should not expect to just be able to jump into something qithout knowledge training or experience and be proficient at it
Sure, but this also applies to understanding the U.S. tax code, which I think is a needlessly overcomplicated, poorly designed system that requires layers of inefficiencies to extract value from, much like web development.
Sure, but this also applies to understanding the U.S. tax code, which I think is a needlessly overcomplicated, poorly designed system that requires layers of inefficiencies to extract value from, much like web development.
You can still make a webpage that is just pure, simple, structured HTML. In fact, the HTML5 standard arguably made it easier to write a simple HTML webpage.
The complexity in webdev exists for good reasons much (most?) of the time.
If you want interactivity, you need an actual programming language. If you're using an actual programming language, then there is a lot of tooling that can be used to make your life easier in the long run. Not intially, but definitely from a maintenance / bug-fixing / etc perspective. If you want to work on a large, complex project with a large team, this "complexity" is invaluable.
That being said, JS as a language kinda sucks still, but "complexity" has been added to attempt to ameliorate that too, such as with TypeScript / CoffeeScript / etc.
The complexity in webdev exists for good reasons much (most?) of the time.
If you want interactivity, you need an actual programming language. If you're using an actual programming language, then there is a lot of tooling that can be used to make your life easier in the long run. Not intially, but definitely from a maintenance / bug-fixing / etc perspective. If you want to work on a large, complex project with a large team, this "complexity" is invaluable.
That being said, JS as a language kinda sucks still, but "complexity" has been added to attempt to ameliorate that too, such as with TypeScript / CoffeeScript / etc.
As I read the original post, Javascript the language isn’t the major problem. It’s things like configuration, certificates and (lack of) inter-operability of frameworks. I agree that this stuff is difficult in web development and I think it’s largely accidental complexity.
(That is, the web is a distributed system and distributed programming is hard, sure. But the configuration hell you often run into is separate from that.)
(That is, the web is a distributed system and distributed programming is hard, sure. But the configuration hell you often run into is separate from that.)
It's also a decentral system. Even for the points of interaction we have protocols that not everybody implements the same (often by mistake, sometimes by choice), but for internals of frameworks and tools, there's very little standardization even within one language, much less intra-language.
> Should we expect anyone to just jump in cold with minimal experience and not find it difficult?
My take on this is that the author had some (relevant) programming experience and thought it would transfer better, hence the frustration.
That was my experience, as someone in the same boat (computational scientist trying to put together a webapp). I have a CS degree and my job involves a lot of diverse coding, from low-level stuff to interface with hardware to graphics/ML/data stuff, not to mention gluing together other people's code.
I nevertheless found it very frustrating to get started with web dev. There seem to be a million slightly different ways to do things and people have strong opinions about which ones are best, so it's hard to settle on authoritative source, especially one that's up to date. Many tutorials bounce between basic web stuff (which I wanted) and basic programming (which makes me zone out). Plus, there's a lurking fear that I'm gonna screw something up in a way that exposes a bunch of data or costs me a fortune in hosting (etc) while trying to make something that I sketched out on three pieces of paper.
This seems a little different from bioinformatics, where a lot of the complexity is baked into the subject matter. For background, textbooks certainly vary in how they present information, but they all pretty much agree: Molecular Biology of the Cell isn't going to tell you that DNA is where it's at, with RNA being a fad vs. The Cell claiming that DNA is outmoded and it's all RNA, baby!
My take on this is that the author had some (relevant) programming experience and thought it would transfer better, hence the frustration.
That was my experience, as someone in the same boat (computational scientist trying to put together a webapp). I have a CS degree and my job involves a lot of diverse coding, from low-level stuff to interface with hardware to graphics/ML/data stuff, not to mention gluing together other people's code.
I nevertheless found it very frustrating to get started with web dev. There seem to be a million slightly different ways to do things and people have strong opinions about which ones are best, so it's hard to settle on authoritative source, especially one that's up to date. Many tutorials bounce between basic web stuff (which I wanted) and basic programming (which makes me zone out). Plus, there's a lurking fear that I'm gonna screw something up in a way that exposes a bunch of data or costs me a fortune in hosting (etc) while trying to make something that I sketched out on three pieces of paper.
This seems a little different from bioinformatics, where a lot of the complexity is baked into the subject matter. For background, textbooks certainly vary in how they present information, but they all pretty much agree: Molecular Biology of the Cell isn't going to tell you that DNA is where it's at, with RNA being a fad vs. The Cell claiming that DNA is outmoded and it's all RNA, baby!
OP’s comment still seems pretty on point though even with your example. Taking the exampling to another field in programming that’s also diverse like gamedev and you run into a similar “too many choices” problem. The solution of course is to just pick a popular tutorial and start. You find and fill in the gaps along the way. Chasing for the elusive best practice is just setting yourself up for failure as a beginner.
This shouldn’t be the top comment. Yes, switching fields and expecting to just work professionally is naive and insulting towards the other profession. Has web dev a lot of legacy cruft and suffers from over-engineering? Absolutely. Even for seasoned programmers starting web dev is harder than it needs to be. Front end development has JS frameworks, CSS or compile-to-CSS languages, templating, web packing, JS transpilers, tree pruning, multiple package managers and what not. But in the end it’s just a bunch of text, images and buttons.
You can reduce any field to "it's a just bunch of...": All coding is just a bunch of binary numbers. All chemistry and biology is just a bunch of molecules. All construction is just a bunch of concrete, bricks and steel. It's nonsense, and is exactly the kind of naive and insulting statement you talked about.
Complexity is emergent. Look below any complex system and you'll find a simpler one, but that doesn't mean the higher-level complexity is unnecessary. All the things you complained about have good reasons to exist, invented by people who actually work as professional web developers and need to solve problems.
The real problem is "seasoned programmers" who don't work in web development, but who expect to be able to jump in an master it in a day, based on the fact that they put together a GeoCities homepage twenty years ago, and are enraged and disgusted to find that things aren't so simple anymore. It's absurd. Developers wouldn't expect to be able to jump into kernel development, or database development, and find it to be a breeze.
Complexity is emergent. Look below any complex system and you'll find a simpler one, but that doesn't mean the higher-level complexity is unnecessary. All the things you complained about have good reasons to exist, invented by people who actually work as professional web developers and need to solve problems.
The real problem is "seasoned programmers" who don't work in web development, but who expect to be able to jump in an master it in a day, based on the fact that they put together a GeoCities homepage twenty years ago, and are enraged and disgusted to find that things aren't so simple anymore. It's absurd. Developers wouldn't expect to be able to jump into kernel development, or database development, and find it to be a breeze.
So, the thing is... I'm not a webdev, though I've done web development in the distant past (early noughties). I'm not a desktop dev, though again done it in the late nineties and early noughties. I messed around with modern desktop/mobile dev recently (specifically Cocoa) and it's basically fine; in many ways simpler and better abstracted than what I was used to (mostly GTK+ and some win32 stuff back in the day). I looked at modern webdev stuff... and I'm sorry, it's insane. Vast increase in complexity for, for most purposes, no real gains.
I can't think of anywhere else this is the case. If you want to access an SQL database, that's easier than it used to be, and better abstractions on top are available if you want them. If you want to write a web API, that's easier than it used to be, and again nice abstractions are available. If you want to process tens of terabytes of data, that's so much easier it's not even funny. But if you want to write a webapp... unless you totally ignore the modern stuff it's vastly, vastly more complex than it used to be.
I suppose this may be my grumpy old man side coming out, but I really don't find modern single page apps to be a better experience than mid-noughties webapps in the vast majority of cases. They're usually worse; in particular they're usually slower, and mess with UI expectations more. And yet they are _vastly_ more difficult to produce. The whole thing seems like it took a wrong turn somewhere.
I can't think of anywhere else this is the case. If you want to access an SQL database, that's easier than it used to be, and better abstractions on top are available if you want them. If you want to write a web API, that's easier than it used to be, and again nice abstractions are available. If you want to process tens of terabytes of data, that's so much easier it's not even funny. But if you want to write a webapp... unless you totally ignore the modern stuff it's vastly, vastly more complex than it used to be.
I suppose this may be my grumpy old man side coming out, but I really don't find modern single page apps to be a better experience than mid-noughties webapps in the vast majority of cases. They're usually worse; in particular they're usually slower, and mess with UI expectations more. And yet they are _vastly_ more difficult to produce. The whole thing seems like it took a wrong turn somewhere.
This is not talked about enough. We frequently ask whether it's too complex, but that assumes the objective is even sound.
Anyone who has done significant work on a full scale SPA using any framework (or no framework) would have to weigh whether it was worth it. And that's on two levels: 1) Was the additional effort and complexity worth it for the perceived improvements in user experience that were sought? And 2) were there actually any such improvements or, even, is the user experience actually worse as a result?
Because I have to say that I'm not convinced. There does seem to be a period when AJAX first became a thing wherein the added responsiveness gained from judicious use of dynamic operations with partial page updates improved the experience over full page reloads exclusively.
Then, someone said, "hey, if that's cool, then why don't we make the whole thing dynamic, fighting Web semantics (and standard browser behavior like back buttons and history) every step of the way?
Not so sure that's a good premise or has led to a good result.
Anyone who has done significant work on a full scale SPA using any framework (or no framework) would have to weigh whether it was worth it. And that's on two levels: 1) Was the additional effort and complexity worth it for the perceived improvements in user experience that were sought? And 2) were there actually any such improvements or, even, is the user experience actually worse as a result?
Because I have to say that I'm not convinced. There does seem to be a period when AJAX first became a thing wherein the added responsiveness gained from judicious use of dynamic operations with partial page updates improved the experience over full page reloads exclusively.
Then, someone said, "hey, if that's cool, then why don't we make the whole thing dynamic, fighting Web semantics (and standard browser behavior like back buttons and history) every step of the way?
Not so sure that's a good premise or has led to a good result.
> Developers wouldn't expect to be able to jump into kernel development, or database development, and find it to be a breeze.
Yes, that’s correct. In my experience, in those fields the complexity of the solution scales with the complexity of the problem. However, modern web development usually tries to squash every problem under a huge stack of complex frameworks and libraries. How many “full stack” kernel or database developers are there? Complexity is not always emergent. In the case of web development it’s in large parts self inflicted. My point was that modern web development is more complex than it needs to be and you didn’t rebut it.
Yes, that’s correct. In my experience, in those fields the complexity of the solution scales with the complexity of the problem. However, modern web development usually tries to squash every problem under a huge stack of complex frameworks and libraries. How many “full stack” kernel or database developers are there? Complexity is not always emergent. In the case of web development it’s in large parts self inflicted. My point was that modern web development is more complex than it needs to be and you didn’t rebut it.
>The real problem is "seasoned programmers" who don't work in web development
Problem is some of those seasoned developers didn't just "put together a GeoCities homepage twenty years ago" (which is itself a weird knock on some devs in the midst of a rant that appears to be knocking people for knocking devs). They worked on "actual" hard stuff that even you might consider hard--let's say kernel or database development or even challenging Web development problems that actually existed prior to the current state of things (audience gasps).
So the question is whether there is a mismatch between the objective and the level of difficulty/complexity? Some stuff is just hard by consensus (and the relative number of people who can actually do it). Other stuff is needlessly complex. Another category is both hard and needlessly complex.
So, instead of parsing the complaints here into a personal attack on certain developer skillsets, then rebutting with the same, it might be more useful to earnestly consider whether things have been made overly complex and can be improved.
Problem is some of those seasoned developers didn't just "put together a GeoCities homepage twenty years ago" (which is itself a weird knock on some devs in the midst of a rant that appears to be knocking people for knocking devs). They worked on "actual" hard stuff that even you might consider hard--let's say kernel or database development or even challenging Web development problems that actually existed prior to the current state of things (audience gasps).
So the question is whether there is a mismatch between the objective and the level of difficulty/complexity? Some stuff is just hard by consensus (and the relative number of people who can actually do it). Other stuff is needlessly complex. Another category is both hard and needlessly complex.
So, instead of parsing the complaints here into a personal attack on certain developer skillsets, then rebutting with the same, it might be more useful to earnestly consider whether things have been made overly complex and can be improved.
But you don't need any of that. A static HTML page and a bit of CSS for styling and you have a web site.
If you don't think package management or transpilers makes your particular project simpler, then don't use them. They are tools.
If you don't think package management or transpilers makes your particular project simpler, then don't use them. They are tools.
Yes, that’s correct. And also missing the point.
No, that is quite relevant to the point. Web development have become easier over time. For example styling is much easier with better and more consistent CSS support in browsers. But at the same time people are attempting much more complex stuff which requires complex tools.
Perhaps there is a problem in the educational resources available if people think they need npm and typescript to build a simple website.
Perhaps there is a problem in the educational resources available if people think they need npm and typescript to build a simple website.
The point is that for someone starting out, web development looks complex. And it is. I know it because I teach this stuff. Ignoring 99% of the industry and use the bare minimum to write a web app is something an expert does. When you’re a beginner you lack the knowledge to recognize expertise. And when most tutorials suggest a heavy stack you‘ll likely stick with the majority. And even when you’re lucky and learn how to write simple web apps from scratch, CSS, HTML and JS are anything but sensible technologies, even with the latest additions that admittedly improved things a lot.
[deleted]
Every field has cruft. Part of gaining experience in a field is not only learning how to do things, but also learning how to navigate the field and differentiate the cruft from that which is of value.
> Even for seasoned programmers starting web dev is harder than it needs to be.
Maybe, but is it harder than it "can be", or is it significantly different to other fields in this respect. I don't believe so.
If you think the extra layers of unnecessary complexity noone needs is either (a) unique to web dev or (b) easily removable, then I think you may have a slightly naïve perspective on humans & society in general. A perfectly organised and navigable field is pretty utopic.
> Even for seasoned programmers starting web dev is harder than it needs to be.
Maybe, but is it harder than it "can be", or is it significantly different to other fields in this respect. I don't believe so.
If you think the extra layers of unnecessary complexity noone needs is either (a) unique to web dev or (b) easily removable, then I think you may have a slightly naïve perspective on humans & society in general. A perfectly organised and navigable field is pretty utopic.
To me there's hard and hard. (reminiscent of simple vs easy).
Hard concepts are ok for anybody with adult desire to learn. It's the ad-hoc transient social choices (package managers, conventions, ecosystems) that are overwhelming. It's draining.
Hard concepts are ok for anybody with adult desire to learn. It's the ad-hoc transient social choices (package managers, conventions, ecosystems) that are overwhelming. It's draining.
Why should you decide which comment should be the tlc?
I second this.
I have some experience in both web development and scientific computing in various fields (not bioinformatics). Things felt just as hard at the beginning in all cases—I don't recall things being any better in scientific computing compared to web development.
I agree that documentation is terrible most of the time, but that's not something limited to web development in my own experience. (On the contrary, check out the documentation of the website in question in this thread: https://news.ycombinator.com/item?id=23031408, it's absolutely fantastic).
Some things become second nature when you have developed expertise in that field, and it's easy to criticise something that you have comparatively less knowledge in and pass it off as "too difficult", particularly when the web development part seems to be a means to an end (which seems to be the case here).
Edit: missing conjunction, tidied up the last sentence for clarity.
I have some experience in both web development and scientific computing in various fields (not bioinformatics). Things felt just as hard at the beginning in all cases—I don't recall things being any better in scientific computing compared to web development.
I agree that documentation is terrible most of the time, but that's not something limited to web development in my own experience. (On the contrary, check out the documentation of the website in question in this thread: https://news.ycombinator.com/item?id=23031408, it's absolutely fantastic).
Some things become second nature when you have developed expertise in that field, and it's easy to criticise something that you have comparatively less knowledge in and pass it off as "too difficult", particularly when the web development part seems to be a means to an end (which seems to be the case here).
Edit: missing conjunction, tidied up the last sentence for clarity.
Oh, interesting. I just wrote a reply saying the exact opposite. I remember learning scientific computing as being much less frustrating. Sometimes you'd have some error spawning out of some ancient C/Fortran library. But that was rare for me. Even after a couple of years on the web, I still feel like I encounter issues where I just don't even know how to start.
Same, and they're not.
If anything, programming for academia is more of a mess than web development. Case in point - I spent the better part of a day rewriting what's essentially a parallelized for loop because the library I'm using in Julia to open HDF5 files has a well known memory leak. I have the impression that something similar in any popular web dev framework or library would have been ruthlessly purged a long time ago.
If anything, programming for academia is more of a mess than web development. Case in point - I spent the better part of a day rewriting what's essentially a parallelized for loop because the library I'm using in Julia to open HDF5 files has a well known memory leak. I have the impression that something similar in any popular web dev framework or library would have been ruthlessly purged a long time ago.
Bugs in libraries and code that overpromises are maddening (and too common), but I think it's different.
You can get the data out of an HDF5 file with
You can get the data out of an HDF5 file with
using HDF5
c = h5open(filename, "r") do file
read(file, "data")
end
HDF5.jl suffers from a memory leak when HDF5 files are opened in succession.
Bug report: https://github.com/JuliaIO/HDF5.jl/issues/349
Bug report: https://github.com/JuliaIO/HDF5.jl/issues/349
Right, but that's a bug.
It's annoying it isn't fixed and working around it is undoubtedly annoying. This seems different from a system where there's lots of complexity in just choosing what frameworks to start with and if/how they play well together, and all of that rapidly changing over time. (I'd bet, for example, that many of those examples used to work).
It's annoying it isn't fixed and working around it is undoubtedly annoying. This seems different from a system where there's lots of complexity in just choosing what frameworks to start with and if/how they play well together, and all of that rapidly changing over time. (I'd bet, for example, that many of those examples used to work).
IMO because of the poor building blocks of web dev (we are trying to build applications with document markup) things that should be simple are too complicated. As an example say I give you an url that will respond with a json of an array of persons and you want to view this json in a table( just view some raw data, I do not want fancy themes or features).
So if we ask web developers how to solve the above problem we get at least 25 different ways to sovle with with JS , and some other 25 ways if we use some backend PHP,Python, Java. If you ask some Qt/Flex developers you will get the same response, Drag and drop the DataGrid and set the data source to your data (and you get a ton of fancy features like adjustable columns, sorting,,,) and you don't have to install 300 dependencies.
So if we ask web developers how to solve the above problem we get at least 25 different ways to sovle with with JS , and some other 25 ways if we use some backend PHP,Python, Java. If you ask some Qt/Flex developers you will get the same response, Drag and drop the DataGrid and set the data source to your data (and you get a ton of fancy features like adjustable columns, sorting,,,) and you don't have to install 300 dependencies.
Yes. I think I was first paid for a website I developed around 1997. As I continued along the path, it was clear that we're trying to implement a kind of remote generic programming environment on top of systems that were not originally designed as such, and that this happens on almost every level.
HTML, at heart, is still a markup language designed to separate a stream of text into semantics and display them. CSS ... I am not even going to go into my feelings about CSS except to say that it suggests that people who look at design principles like C.R.A.P. were not near the discussion and were not in the room for some time. A programming language named after Java, in which it no way resembles, that had to be pushed out in almost no time at all given the irrational deadlines, dominates the landscape ... and we must transpile to it. And we're pushing video over HTTP.
All of the first-mover advantages have locked us into this path and many very intelligent people are busy using their prowess to perform work under some very arbitrary conditions, like not being able to use the letter "e" in writing. While it is a challenge and stretches the abilities, no respite is offered, and so there's a tremendous amount of waste occurring as these initial conditions, propagated down through time, must be constantly worked around.
HTML, at heart, is still a markup language designed to separate a stream of text into semantics and display them. CSS ... I am not even going to go into my feelings about CSS except to say that it suggests that people who look at design principles like C.R.A.P. were not near the discussion and were not in the room for some time. A programming language named after Java, in which it no way resembles, that had to be pushed out in almost no time at all given the irrational deadlines, dominates the landscape ... and we must transpile to it. And we're pushing video over HTTP.
All of the first-mover advantages have locked us into this path and many very intelligent people are busy using their prowess to perform work under some very arbitrary conditions, like not being able to use the letter "e" in writing. While it is a challenge and stretches the abilities, no respite is offered, and so there's a tremendous amount of waste occurring as these initial conditions, propagated down through time, must be constantly worked around.
But Qt/Flex is just one way of building desktop UIs :)
Comparing the entire gamut of web development against a single library seems unfair.
What about asking the same question of .net developers, VB classic developers, golang developers, electron developers, python developers, Java developers, c++ developers, swift developers? They'd all do it differently than QT developers would because there are different GUI frameworks that need to be used.
This problem is not unique to web dev.
Comparing the entire gamut of web development against a single library seems unfair.
What about asking the same question of .net developers, VB classic developers, golang developers, electron developers, python developers, Java developers, c++ developers, swift developers? They'd all do it differently than QT developers would because there are different GUI frameworks that need to be used.
This problem is not unique to web dev.
I am not sure is true, For example the .Net Windows Forms have a DataGrid widget that works exactly the same , the only differences between competent GUI frameworks is that Widget names and functions might differ.
so in summary in any GUI framerwork the reaponse is the same drag the DataGrid and connect it to the data(the names can differ), maybe some Delphi,GTK,JavaFX,WPF devs can confirm or disprove me.
so in summary in any GUI framerwork the reaponse is the same drag the DataGrid and connect it to the data(the names can differ), maybe some Delphi,GTK,JavaFX,WPF devs can confirm or disprove me.
I think you are ignoring a lot of the complexity and assuming it is trivial in desktop GUI development, but complex for web development.
I do not think this is the case - e.g. just look at the two different options just for Java (Swing vs JavaFx):
https://docs.oracle.com/javase/tutorial/uiswing/components/t...
https://docs.oracle.com/javafx/2/ui_controls/table-view.htm
This is not just dragging and dropping some common component - each thing is implemented differently, and so has a different API to control it (and require huge tutorials to work out how to use them) so everyone has to do it differently. They're not just different APIs to the same underlying OS-provided control. A Java Swing developer will not just be able to do the same thing they always do just in Qt, TK, Wx, WPF, WinForms, Gtk etc etc etc - it is still totally different. Sure the .net languages are all the same (i.e. same API ... but then do you use WinForms? WPF? Gtk#? others?), but there is more to GUI development than .net.
I do not think this is the case - e.g. just look at the two different options just for Java (Swing vs JavaFx):
https://docs.oracle.com/javase/tutorial/uiswing/components/t...
https://docs.oracle.com/javafx/2/ui_controls/table-view.htm
This is not just dragging and dropping some common component - each thing is implemented differently, and so has a different API to control it (and require huge tutorials to work out how to use them) so everyone has to do it differently. They're not just different APIs to the same underlying OS-provided control. A Java Swing developer will not just be able to do the same thing they always do just in Qt, TK, Wx, WPF, WinForms, Gtk etc etc etc - it is still totally different. Sure the .net languages are all the same (i.e. same API ... but then do you use WinForms? WPF? Gtk#? others?), but there is more to GUI development than .net.
Do you think if I ask how to solve this problem with angularN I get 1 response like I get with Qt? I think I will get still at least 5 responses, one on how to implement it from scratch, other how ti implement it from scrratch but using a different abstraction, other how to wrap X jquery library in angularN, other on how to use the existing wrapper for the jQuery library, other on how to use a pure angular library, probably some response would say to just buy something from X and at least get a working thing etc.
Replace angularN from above with React or even better Vanila? Make an Ask HN if you want on how to solve the problem with Vanila + vanila library (optional) and make a new question on how to solve it with Qt or WPF (optional with libraries) , as optional ask for sane funtionality like resizing columns, sorting data, re-arranging columns and performance when displaying more then 100-1000 rows.
Replace angularN from above with React or even better Vanila? Make an Ask HN if you want on how to solve the problem with Vanila + vanila library (optional) and make a new question on how to solve it with Qt or WPF (optional with libraries) , as optional ask for sane funtionality like resizing columns, sorting data, re-arranging columns and performance when displaying more then 100-1000 rows.
As a webdev, that is actually the beauty of the web, you get to solve / create / build in all these ways. Neither are wrong.
Another point I really like is that I can view a webpage from the 90’s and it will look fine (maybe not great)
And there are frameworks, platforms and plugins which allow you to do what you described without little coding. You can go up and down on the stack, switch stacks and still get to the same end result.
Another point I really like is that I can view a webpage from the 90’s and it will look fine (maybe not great)
And there are frameworks, platforms and plugins which allow you to do what you described without little coding. You can go up and down on the stack, switch stacks and still get to the same end result.
>you get to solve / create / build in all these ways.
Most of the time your solution is broken. Some examples I hit recently from giant companies, the PayPal amount input keyboard handeling is broken (Del button was not working), Youtube search inpout with it's dropdown can get stuck open and you are forced to reload the page , when you re-invent the wheel you probably are skiping or gorgetting to implement everything so your custom popup might appear outside the screen, or it does not implement Tab ordering correctly or is not accessible, or is not navigable with arrow keys etc. Where in other GUI frameworks you use the built-in one and on top of that all those GUI framework will let you provide/override a custom Paint function if you want (on top of existing customization) so you are not limited either in any way.
Most of the time your solution is broken. Some examples I hit recently from giant companies, the PayPal amount input keyboard handeling is broken (Del button was not working), Youtube search inpout with it's dropdown can get stuck open and you are forced to reload the page , when you re-invent the wheel you probably are skiping or gorgetting to implement everything so your custom popup might appear outside the screen, or it does not implement Tab ordering correctly or is not accessible, or is not navigable with arrow keys etc. Where in other GUI frameworks you use the built-in one and on top of that all those GUI framework will let you provide/override a custom Paint function if you want (on top of existing customization) so you are not limited either in any way.
> And there are frameworks, platforms and plugins which allow you to do what you described without little coding.
Name three.
I'm not joking. I'd love to know about a data table library that meets following requirements:
- Doesn't commit me, the user, to one of the infinite combinations of ridiculous JS build chains. Instead, it should be accessible as a single-file JS dependency in transpiled form. It may use a ridiculous JS build chain for building itself, but I don't want to care about that.
- Doesn't commit me to one of the few big JS SPA frameworks (which each comes with its own combination of ridiculous JS build chains). I want a table, not the kitchen sink, and definitely not the whole meat processing plant.
- Doesn't commit me to particular styling.
- Can handle thousands of rows on a single page.
- Has the most basic ability to: sort each column, and sort by multiple columns.
- Has the ability to do a per-column search across all data (that all matters if it paginates the data)
- Doesn't interfere with browser search, and/or allows a full-table search on all data if paginated.
- Has the ability to filter out ranges of values. Bonus points for extra type-specific affordances.
- Bonus points for the ability to dump its contents into CSV (file or clipboard). Though this could presumably be handled with a small amount of code on my side.
- Extra bonus points: has some minimal pivoting ability.
I haven't found any meeting these requirements, where IMO these should be the default (and if browsers are ever going to standardize on some data table implementation, I'm going to petition for all of these to be included).
Name three.
I'm not joking. I'd love to know about a data table library that meets following requirements:
- Doesn't commit me, the user, to one of the infinite combinations of ridiculous JS build chains. Instead, it should be accessible as a single-file JS dependency in transpiled form. It may use a ridiculous JS build chain for building itself, but I don't want to care about that.
- Doesn't commit me to one of the few big JS SPA frameworks (which each comes with its own combination of ridiculous JS build chains). I want a table, not the kitchen sink, and definitely not the whole meat processing plant.
- Doesn't commit me to particular styling.
- Can handle thousands of rows on a single page.
- Has the most basic ability to: sort each column, and sort by multiple columns.
- Has the ability to do a per-column search across all data (that all matters if it paginates the data)
- Doesn't interfere with browser search, and/or allows a full-table search on all data if paginated.
- Has the ability to filter out ranges of values. Bonus points for extra type-specific affordances.
- Bonus points for the ability to dump its contents into CSV (file or clipboard). Though this could presumably be handled with a small amount of code on my side.
- Extra bonus points: has some minimal pivoting ability.
I haven't found any meeting these requirements, where IMO these should be the default (and if browsers are ever going to standardize on some data table implementation, I'm going to petition for all of these to be included).
My first reaction (because you have a lot of stipulations) would be to use Drupal, which comes with views (a db query engine) in core. I am not up to date with details, but it should be able to get you very far. I did a booking system that did some impressive queries very fast, and allowed data export.
I think he mentioned that it must be 1 thing, no dependency, imagine he is already using soem other backend and now he has to switch to Drupal for a good Dratagrid and tomorrow when he needs a rich text editor he needs to switch to Wordpress and the new day when he needs a decent chart he has to use GWT.
The ideal is that Web Technology would have built-in or if not community made building blocks. If made by community it must have no licensing issue, no dependency hell, no 100 alternatives for each widget. In my experience jQuery UI is the one that got close to this. You had only 1 dependency on jQuery, the community was not fragmented to have 100 alternatives for each thing and the included widgets are not incomplete.
The ideal is that Web Technology would have built-in or if not community made building blocks. If made by community it must have no licensing issue, no dependency hell, no 100 alternatives for each widget. In my experience jQuery UI is the one that got close to this. You had only 1 dependency on jQuery, the community was not fragmented to have 100 alternatives for each thing and the included widgets are not incomplete.
The backend should be your database, and Drupal is one of the more successful CMSes that takes care of the model/view. You can build views very easily with them and use that as your search engine or table (or both), you can hook it up with its api for more flexibility if you so wish.
I don’t know in this exact use case, but there are probably building blocks (modules) for his to get very close.
I’ll absolutely agree Jquery is an excellent library that I use to this day. It’s extremely elegant and indeed, it is a good example how it is suppose to be done.
I don’t know in this exact use case, but there are probably building blocks (modules) for his to get very close.
I’ll absolutely agree Jquery is an excellent library that I use to this day. It’s extremely elegant and indeed, it is a good example how it is suppose to be done.
Well, if I have a backend serving views, then sky is the limit :).
But I really meant a self-contained, fully front-end piece of JS. One you could feed a data table and let the user play with it.
A side issue I'm hinting at is that I believe the features I mentioned should be present by default in any kind of tabular display. Without them, the users are getting shafted - they're served data in front of them (be it item prices, virus stats, or a restaurant menu), but they have no way to explore it on their own. This is but a small piece in a great set of ways in which our industry ensures the users can't ever grow and become more proficient, and then complains that users are dumb. Can't have a bicycle for the mind if pedals and handlebar are not provided.
But I really meant a self-contained, fully front-end piece of JS. One you could feed a data table and let the user play with it.
A side issue I'm hinting at is that I believe the features I mentioned should be present by default in any kind of tabular display. Without them, the users are getting shafted - they're served data in front of them (be it item prices, virus stats, or a restaurant menu), but they have no way to explore it on their own. This is but a small piece in a great set of ways in which our industry ensures the users can't ever grow and become more proficient, and then complains that users are dumb. Can't have a bicycle for the mind if pedals and handlebar are not provided.
Webdev is more like a trade than a science; you benefit more from remembering a large body of facts and details that comes with experience. There's no theory to learn, you just have to get practise in things like databases, browsers, HTTP, frameworks, etc.
This is probably why the author is struggling. This is a real world industry tackling real world problems. It is not the neat and cut world of science research.
This is probably why the author is struggling. This is a real world industry tackling real world problems. It is not the neat and cut world of science research.
> It is not the neat and cut world of science research.
You had me up until this last sentence. Everything I've experienced in science research is the opposite of neat or clean cut. Actually, I think I do my best programming and debugging with a treat it like a science experiment.
You had me up until this last sentence. Everything I've experienced in science research is the opposite of neat or clean cut. Actually, I think I do my best programming and debugging with a treat it like a science experiment.
I don't have any experience in scientific research, so that's a fair justification.
On the contrary, learning the fundamentals of CS has made me a much better dev, more so than any fact or tidbit.
Its a testament to the awesomeness of the web however that so so many people even without any formal training can go about creating value.
You _can_ be a webdev tradesman, fixing things till it works, kinda, or you can delve into the fundamentals and be much more productive. But you don’t have to and I think thats awesome.
For example I still remember that conference that introduced Ruby on Rails to the world so many years ago. Most webdevs did not know there was an underlying theory and standard (REST) that can be used to make web apis bearable! After rails, we kinda figured huh, so this RFC for the web was sitting there all those years gathering dust and only weirdos and graybeards talked about it. But it was actually very very useful.
Its a testament to the awesomeness of the web however that so so many people even without any formal training can go about creating value.
You _can_ be a webdev tradesman, fixing things till it works, kinda, or you can delve into the fundamentals and be much more productive. But you don’t have to and I think thats awesome.
For example I still remember that conference that introduced Ruby on Rails to the world so many years ago. Most webdevs did not know there was an underlying theory and standard (REST) that can be used to make web apis bearable! After rails, we kinda figured huh, so this RFC for the web was sitting there all those years gathering dust and only weirdos and graybeards talked about it. But it was actually very very useful.
Good points though I considered REST to be more like best practises than theory. One could see it either way i think.
When 99% of people say "REST" what they mean is "RESTful" in that its a ball of JSON garbage. That's just a trade and 99% of people are in that bucket.
That last 1% actually have read the OG paper and when they say REST they mean REST and its real theory.
That last 1% actually have read the OG paper and when they say REST they mean REST and its real theory.
You're absolutely right. Webdev is certainly its own fully fleshed out fields. Oddly, I think I can play off your facetious quote and turn it into a more analytical way of framing what the post was about.
"I am a biochem major but I decided to work in bioinformatics/scientific-programming despite having minimal skills or experience and I ran into loads of problems and issues that I was able to steadily solve because the ecosystem felt sane, the tracebacks were clear, and the debugging usually had a clear path forward. Then I was a bioinformatics professional doing work in webdev [...] I ran into loads of problems [...] that felt consistently more opaque and magical than when I was learning scientific programming"
I think that's the heart of it. The transition to the web feels so much more chaotic than the transition to bioinformatics.
"I am a biochem major but I decided to work in bioinformatics/scientific-programming despite having minimal skills or experience and I ran into loads of problems and issues that I was able to steadily solve because the ecosystem felt sane, the tracebacks were clear, and the debugging usually had a clear path forward. Then I was a bioinformatics professional doing work in webdev [...] I ran into loads of problems [...] that felt consistently more opaque and magical than when I was learning scientific programming"
I think that's the heart of it. The transition to the web feels so much more chaotic than the transition to bioinformatics.
I would suggest that it's really easy to envision some much less painful abstractions for web development.
Sure, learning an entire field from first principals is going to be hard, and to do science you need to understand those principals. But the general concept of "make a dynamic web page" is not computer science! What most people (and the author of the blog post) need is a clean abstraction that allows them to glue a few APIs together with some custom forms.
Sure, learning an entire field from first principals is going to be hard, and to do science you need to understand those principals. But the general concept of "make a dynamic web page" is not computer science! What most people (and the author of the blog post) need is a clean abstraction that allows them to glue a few APIs together with some custom forms.
That's the premise of low-code, right? "I don't need a fully general language with configurable everything; I just need a simple tool for this one common use case."
I'm excited about what Dark (https://darklang.com/) is doing with this idea. As you said, "a clean abstraction that allows them to glue a few APIs together with some custom forms."
I'm excited about what Dark (https://darklang.com/) is doing with this idea. As you said, "a clean abstraction that allows them to glue a few APIs together with some custom forms."
I am a software engineer who works closely with a team of bioinformaticians. Your reply suggests you only have one side of the story.
A bioinformatician is someone who has quite serious programming skills on top of their biology knowledge. They are frequently bottlenecked by compute and so tend to have a good understanding of data structures and algorithms. They usually know 3 or more languages and are no strangers to databases, version control, containers, et cetera.
And yet I recognize the feeling from the article: they get stuck in the heap of complexity we (as an industry) have created around a bog standard web frontend. The majority of us are not doing rocket science - the applications I’m working on today do similar things to the ones I worked on five years ago. Yet somehow, the ‘how’ of it, the whole Rube Goldberg machine, tends to get more convoluted every passing year.
I often get the feeling a nonzero percentage of it is introduced purely so web developers can feel they’re “real engineers” whatever that may be. There’s nothing to prove, now stop piling on junk and reinventing wheels.
A bioinformatician is someone who has quite serious programming skills on top of their biology knowledge. They are frequently bottlenecked by compute and so tend to have a good understanding of data structures and algorithms. They usually know 3 or more languages and are no strangers to databases, version control, containers, et cetera.
And yet I recognize the feeling from the article: they get stuck in the heap of complexity we (as an industry) have created around a bog standard web frontend. The majority of us are not doing rocket science - the applications I’m working on today do similar things to the ones I worked on five years ago. Yet somehow, the ‘how’ of it, the whole Rube Goldberg machine, tends to get more convoluted every passing year.
I often get the feeling a nonzero percentage of it is introduced purely so web developers can feel they’re “real engineers” whatever that may be. There’s nothing to prove, now stop piling on junk and reinventing wheels.
> ... or to be less facetious, modern web/frontend development is a genuine skill set that benefits from experience and knowledge.
I mean, there's a strong argument that it has become too complex for no particularly good reason. It is _far_ more complex today than a decade or so ago, and the results in many cases are arguably worse for the user.
If you compare the _average_ modern react-y website to a Ruby on Rails one with minimal or no Javascript from 2007 or so, the react-y one is far more difficult (and slower) to work on, but is it really delivering extra value to the end user? There are exceptions, of course, but I think for most CRUD web apps things have gone in a very bad direction.
I mean, there's a strong argument that it has become too complex for no particularly good reason. It is _far_ more complex today than a decade or so ago, and the results in many cases are arguably worse for the user.
If you compare the _average_ modern react-y website to a Ruby on Rails one with minimal or no Javascript from 2007 or so, the react-y one is far more difficult (and slower) to work on, but is it really delivering extra value to the end user? There are exceptions, of course, but I think for most CRUD web apps things have gone in a very bad direction.
Funny thing is, those rails and simple Django apps are just as effective today as they were 10 years ago. In fact, not letting your app get bogged down in the mess that is modern front ends is likely a competitive advantage these days. Keep it simple. Django and Rails are the best they have ever been. Most software is about data, not front ends. It takes dedicated teams to tame “modern” front ends. Let the data show you the way and avoid front end stuff until you absolutely can’t. Maybe, after you need something to do with a lot of VC money or you can justify a certain UX in your app.
100 percent this.
And yet, even within the web development community, we hear a lot of complaints about how this or that isn't beginner-friendly; or how such-and-such is gatekeeping, and so on. This has been baffling to me. Is it the same in other disciplines? Are engineering, bioinformatics or medicine, for example, expected to be beginner-friendly?
And yet, even within the web development community, we hear a lot of complaints about how this or that isn't beginner-friendly; or how such-and-such is gatekeeping, and so on. This has been baffling to me. Is it the same in other disciplines? Are engineering, bioinformatics or medicine, for example, expected to be beginner-friendly?
If you have any doubt that software takes real skill, wade into the muck that is software written by scientists. You will discover classes of errors at all levels that you didn't even think possible. Testing? What's that? Exponential run time? Does that matter? And why does my program give different answers now than it did last year? Those numbers worked for my paper--make the program give them again!
IMO web dev isn't that hard, given one has a CS background. I came out of college, immediately started to teach a bootcamp (they had a lot of faith in me). What I learned about the difficulty of web dev:
- Finding out the right curriculum by yourself is hard
- Once you have the right curriculum, then you can learn it yourself provided one has the right background. In fact, they even can teach that curriculum to people that don't know how to program or do web dev yet
When I was done teaching NodeJS/ReactJS, I went on to freelance for a company that had a microservice backend in Java. Since my CS degree was a Java school, I picked up Spring Boot quite quickly. I needed to build on top of code and not create it from scratch. That really helped, starting from scratch would've been more difficult and does require a lot more experience (so altering an open-source project might be a good way to practice). After 3 months the job was done and I left.
Now, I know that I don't have a lot of experience, but I honestly haven't seen any hard challenges yet when it comes to web dev at the 3 year old startup level. There were no scalability or security challenges to take care of. The web dev rabbit hole can go deep (and that is hard), but in practice it mostly doesn't (unless you work at Google scale, I presume). Deployment with IBM and Kubernetes was just: read the docs and know how to debug and troubleshoot.
It's not easy per se, but if you start with the right background (CS and a lot of programming courses in such a CS curriculum) and get a good coding school curriculum after that, then I would argue that web dev isn't that hard. There are enough startups where you'd be productive.
Though, to be fair: if I couldn't use a debugger, I'd be totally lost. Using a debugger makes it not that hard. This is also why I had a harder time with learning Webpack.
What I found way harder is wrapping your mind around concepts that seem completely alien. I have this still a bit with functional programming (not covered in my CS degree) or back in the day with databases or certain forms of math.
Reading API documentation isn't part of it which is what I feel web dev is mostly nowadays. Even when you learn another programming language, it's all procedural/object oriented, so you just need to learn the quirks of that particular language. When I first started out I also had to deal with emotional fear of being inadequate. But after your fifth language of feeling that way, you know it's part of the process and you shouldn't take it to mean that you are inadequate.
- Finding out the right curriculum by yourself is hard
- Once you have the right curriculum, then you can learn it yourself provided one has the right background. In fact, they even can teach that curriculum to people that don't know how to program or do web dev yet
When I was done teaching NodeJS/ReactJS, I went on to freelance for a company that had a microservice backend in Java. Since my CS degree was a Java school, I picked up Spring Boot quite quickly. I needed to build on top of code and not create it from scratch. That really helped, starting from scratch would've been more difficult and does require a lot more experience (so altering an open-source project might be a good way to practice). After 3 months the job was done and I left.
Now, I know that I don't have a lot of experience, but I honestly haven't seen any hard challenges yet when it comes to web dev at the 3 year old startup level. There were no scalability or security challenges to take care of. The web dev rabbit hole can go deep (and that is hard), but in practice it mostly doesn't (unless you work at Google scale, I presume). Deployment with IBM and Kubernetes was just: read the docs and know how to debug and troubleshoot.
It's not easy per se, but if you start with the right background (CS and a lot of programming courses in such a CS curriculum) and get a good coding school curriculum after that, then I would argue that web dev isn't that hard. There are enough startups where you'd be productive.
Though, to be fair: if I couldn't use a debugger, I'd be totally lost. Using a debugger makes it not that hard. This is also why I had a harder time with learning Webpack.
What I found way harder is wrapping your mind around concepts that seem completely alien. I have this still a bit with functional programming (not covered in my CS degree) or back in the day with databases or certain forms of math.
Reading API documentation isn't part of it which is what I feel web dev is mostly nowadays. Even when you learn another programming language, it's all procedural/object oriented, so you just need to learn the quirks of that particular language. When I first started out I also had to deal with emotional fear of being inadequate. But after your fifth language of feeling that way, you know it's part of the process and you shouldn't take it to mean that you are inadequate.
All programming has a barrier, but I think web development is a unique situation where there’s a lot of complexity because it’s so open to abuse. If you don’t do HTTPS right bad things can happen, so setting that up is hard. If you process payments bad things can happen, so setting that up is hard. If you send people email bad things can happen, so setting that up is hard. With things like game programming the worst that can happen is your game is slow or buggy or laggy but usually you’re not exposing it to the huge, often malicious audience. With web development you make one mistake and someone is stealing thousands of dollars from you and running cryptomining on your VPS while defacing your website.
> With things like game programming the worst that can happen is your game is slow or buggy or laggy but usually you’re not exposing it to the huge, often malicious audience
Not my domain of expertise, but cheating and lags in online real-time games (like FPS games) can ruin a game quickly. Also, in-game shops, currency, micro-transactions probably are targets for crackers and exploiters.
Not my domain of expertise, but cheating and lags in online real-time games (like FPS games) can ruin a game quickly. Also, in-game shops, currency, micro-transactions probably are targets for crackers and exploiters.
Generally that’s not something you run into until you’re actually much deeper into the field.
So because the webapp is exposed to the public over the internet and there's a risk of hacking the the developer story can be poor? That doesn't sound right. The developer story is just poor because the frameworks and developer Tools are poor, badly documented and don't work well together holistically. Part of this is just due to the continuous churn in web development that creates insane amount of busywork and makes sure that almost all of your available documentaton is always put of date. i find that most of this "complexity" that the author mentions is just man made.
But things like HTTPS, payment processing, emails / account management and even game programming, all of these have already been made easier - nobody has to build any of that on their own anymore, at best they have to configure it.
To make a safe website, stop wanting to do everything yourself. Use a webhost that doesn't give you SSH access, or a service like Netlify.
To make a safe website, stop wanting to do everything yourself. Use a webhost that doesn't give you SSH access, or a service like Netlify.
And linking all these heterogeneous pieces together and configure them to work is like cat herding: each component is so different and the developers are so proud of their "innovative" way of doing things that one cannot master it. A few years ago I started to believe that modern web developers are more about showing off and hipstering than having a genuine desire to make things better.
Weirdly, I think the same. I noticed an uptick of twitter programmers.
I just checked the author’s website, and he could have just used an installer to get him a wordpress website with an elearning system plugin attached to get to the same result. Buy an SSL and he would have been done. Any other path is just making it harder if it is not his field of expertise.
Yeah and it results in so many small, seemingly insiginificant things that can bite you. Like the last point in which I guess the author had hardcoded their redis uri rather than using the environment variable provided by Heroku. How would anyone know about that unless they had been told by someone or been bitten in the past or had some understanding of how Heroku worked underneath.
Edit: from the comments it's clear that OP didn't actually hardcode the connection string.
Edit: from the comments it's clear that OP didn't actually hardcode the connection string.
Can't really agree with that. Although most of my experience and expertise is with mobile game development, I think that I've read at least 5 blog posts and articles that explained that all connection strings, usernames and passwords (aside from something really trivial for local testing) should always be read from environment variables and never be hardcoded, and arguments were convincing enough. So, even if I haven't worked with Heroku before, I would look for a way to get this url in a service-provided environment variable.
Also, I would look to configure all components to fail loudly, so that if something wouldn't connect to Redid, my server just wouldn't launch.
I'm not trying to say that I'm better or smarter than OP: I've actually tried learning bioinformatics, but dropped out of uni 14 years ago, realising that my brain is just not created for anything biology-related. Rather my point is, many of these problems (and problems I've seen junior developers struggling with) are not some arcane knowledge that you should just learn by heart (like all of those awful genus names in Latin, or all the enzymes in the dreadful ATP scheme that I still have nightmares of sometimes), but a logical extension of more basic and universal principles that actually make total sense once you think about them.
Also, I would look to configure all components to fail loudly, so that if something wouldn't connect to Redid, my server just wouldn't launch.
I'm not trying to say that I'm better or smarter than OP: I've actually tried learning bioinformatics, but dropped out of uni 14 years ago, realising that my brain is just not created for anything biology-related. Rather my point is, many of these problems (and problems I've seen junior developers struggling with) are not some arcane knowledge that you should just learn by heart (like all of those awful genus names in Latin, or all the enzymes in the dreadful ATP scheme that I still have nightmares of sometimes), but a logical extension of more basic and universal principles that actually make total sense once you think about them.
Two quick points here:
1. A few years ago, I probably would have made the mistake of actually hard-coding my connection strings, usernames and passwords. I'm pretty sure I did that on my first ever webapp, cause I didn't know any better. So, not defending my intelligence, or taking any offense at your statement. 2. But... That's not what's happened here. I'm setting the redis url inside Heroku's "Config Vars" section, and then reading it in with Django's `env()` functionality, as is appropriate. But the url has seemingly been changed from underneath me, for a reason I still don't fully understand.
Side-note: I used to be under the impression that CS was "harder" than biology, and that anyone who understood CS could pick up biology at will, if they decided. Turns out that's not the case, haha.
1. A few years ago, I probably would have made the mistake of actually hard-coding my connection strings, usernames and passwords. I'm pretty sure I did that on my first ever webapp, cause I didn't know any better. So, not defending my intelligence, or taking any offense at your statement. 2. But... That's not what's happened here. I'm setting the redis url inside Heroku's "Config Vars" section, and then reading it in with Django's `env()` functionality, as is appropriate. But the url has seemingly been changed from underneath me, for a reason I still don't fully understand.
Side-note: I used to be under the impression that CS was "harder" than biology, and that anyone who understood CS could pick up biology at will, if they decided. Turns out that's not the case, haha.
Appologies for my earlier comment then I misunderstood your last point. I think my point still stands though. Thanks for the article!
> But the url has seemingly been changed from underneath me, for a reason I still don't fully understand.
Is the Redis add-on attached to the Heroku instance? This is how Heroku knows which instances' environment variables need updating when rotating credentials.
Is the Redis add-on attached to the Heroku instance? This is how Heroku knows which instances' environment variables need updating when rotating credentials.
Sure but it's something thats easily overlooked or neglected, many tutorials might tell a beginner to do it but not _why_ to do it which is important. It's also easy to hardcode during testing or development and forget about. Why would someone without a software background expect that a url could change from under them?
> Also, I would look to configure all components to fail loudly, so that if something wouldn't connect to Redid, my server just wouldn't launch.
In this case though the OP did not have control over the redis connection handling it was through a dependency that silently timed out.
> Also, I would look to configure all components to fail loudly, so that if something wouldn't connect to Redid, my server just wouldn't launch.
In this case though the OP did not have control over the redis connection handling it was through a dependency that silently timed out.
If you redirect users, bad things can happen. If you consume third party dependencies, bad things can happen. If you handle sensitive or personally identifiable information, bad things can happen. If you are not aware of privacy protection laws, bad things can happen. If you configure S3 buckets, bad things can happen. If you deploy static assets, better exclude sensitive information like version control data or bad things can happen. If you construct database queries, bad things can happen. If you manipulate files, bad things can happen. If your site can be put in an iframe, bad things can happen... and the list goes on and on.
I don't think that this is what makes it hard.
Setting up https, or online payments, etc. is actually quite simple. People with engineering or scientific background have most likely faced much more complex tasks. Security and safety aspects can be more stringent in many other industries, including in software development.
The difficulty of the web, I think, is the proliferation of standards (or not so standards) and technologies: networking, https and friends, APIs (payments, etc), html, css, javascript, php, ruby, java, aws, react, vue, redis, SQL, etc. And they are all evolving independently all the time. This is a right mess and can be overwhelming.
Setting up https, or online payments, etc. is actually quite simple. People with engineering or scientific background have most likely faced much more complex tasks. Security and safety aspects can be more stringent in many other industries, including in software development.
The difficulty of the web, I think, is the proliferation of standards (or not so standards) and technologies: networking, https and friends, APIs (payments, etc), html, css, javascript, php, ruby, java, aws, react, vue, redis, SQL, etc. And they are all evolving independently all the time. This is a right mess and can be overwhelming.
This is a fantastic point. Payments in particular have been intimidating for me. Nobody wants to pay for something and then have some goofy mistake on the backend mess up processing it. Or even worse, make them feel like their credit card is susceptible to theft.
I don't know how to explain it, exactly, but worrying about this definitely made development more difficult, even when initially setting up and testing out.
I don't know how to explain it, exactly, but worrying about this definitely made development more difficult, even when initially setting up and testing out.
If it makes you feel any better, I have been programming for almost half a dozen years, the latter portion of which I spent learning how to do so formally, and I am deathly frightened to work on any of the things you had trouble with. I’ll take securing an ancient obfuscated firmware binary any day over trying to secure your website :)
I feel like interacting with there real world, and with users in particular, just inevitably leads to orders of magnitude more complexity.
You can see that even in simple cases. Writing a basic single-propose script is trivial; it may end up being just a line or two. As soon as you decide to make it general and reusable, though, complexity explodes: positional arguments are confusing, better name them. Users are gonna expect a familiar format. Need a library for that. Reasonable and understandable names are tricky! What arguments are required versus optional? What if _this_ option is specified without _that_ one? Gotta add proper errors for every bad state! What if the user starts using globbing? Or wants to specify multiple inputs and one output, or vice versa?
Your three-line script is now 100+ lines. And you're not even worried about hostile or abusive users yet!
Writing code for the web is all that--and much more besides.
You can see that even in simple cases. Writing a basic single-propose script is trivial; it may end up being just a line or two. As soon as you decide to make it general and reusable, though, complexity explodes: positional arguments are confusing, better name them. Users are gonna expect a familiar format. Need a library for that. Reasonable and understandable names are tricky! What arguments are required versus optional? What if _this_ option is specified without _that_ one? Gotta add proper errors for every bad state! What if the user starts using globbing? Or wants to specify multiple inputs and one output, or vice versa?
Your three-line script is now 100+ lines. And you're not even worried about hostile or abusive users yet!
Writing code for the web is all that--and much more besides.
In a related article: "Tackling bioinformatics as a Webdev: Why is it so hard?"
Maybe Webdev is just complex? But tbh, as a bioinformatician (in Next Generation Sequencing mainly) I actually find Webdev easier. I mean, we face Bam files, Sam files, Cram files and annotate them with Bed files (with many different poorly documented standards) or GTF files depending on your Ensembl or UCSC references persuasion... Some file types are 0-based inclusive, some are 1 based exclusive, some call chromosome just ints (1,2, and then error when X, Y if you really assume they are ints) some call them chr1, chr2, chrX etc. We have fastq files with different quality encodings depending on their age, I recently had some strange errors, turns out the R2 of a fastq file is reverse complementary from the standard (why?!)... It's a mess and many people that spend years in the field still don't know more than just "choose one side and stick with it"... We also have a huge amount of frameworks for data analysis pipelines, Snakemake, Bpipe, Broad's WDL, etc, etc, etc. Not to mention the insance amounts of read mappers (BWA, RSEM, Tophat 1 and 2, Kallisto etc). Need I go on?
Maybe Webdev is just complex? But tbh, as a bioinformatician (in Next Generation Sequencing mainly) I actually find Webdev easier. I mean, we face Bam files, Sam files, Cram files and annotate them with Bed files (with many different poorly documented standards) or GTF files depending on your Ensembl or UCSC references persuasion... Some file types are 0-based inclusive, some are 1 based exclusive, some call chromosome just ints (1,2, and then error when X, Y if you really assume they are ints) some call them chr1, chr2, chrX etc. We have fastq files with different quality encodings depending on their age, I recently had some strange errors, turns out the R2 of a fastq file is reverse complementary from the standard (why?!)... It's a mess and many people that spend years in the field still don't know more than just "choose one side and stick with it"... We also have a huge amount of frameworks for data analysis pipelines, Snakemake, Bpipe, Broad's WDL, etc, etc, etc. Not to mention the insance amounts of read mappers (BWA, RSEM, Tophat 1 and 2, Kallisto etc). Need I go on?
That's super interesting. If someone asked me if bioinformatics was complex, I'd say "Yes, massively so." But then, I would take it in a totally different direction. To me, the complexity of bioinformatics come from the biology. How do you think about weird biological edge cases? What do your distributions look like and how do you choose thresholds? How do you even form a good question?
Thinks like [B|S|Cr]am files are complex in their own way, but mostly they're just kinda annoying to have to figure out. You can dig in as deep as you need pretty easily.
On the other hand, maybe I'm over-simplifying because I feel more comfortable in this domain, which I think is your point.
Thinks like [B|S|Cr]am files are complex in their own way, but mostly they're just kinda annoying to have to figure out. You can dig in as deep as you need pretty easily.
On the other hand, maybe I'm over-simplifying because I feel more comfortable in this domain, which I think is your point.
Well, I come from Biology, always had Webdev as a hobby and then moved into bioinformatics which now turns towards software development. So Biology feels most familiar, but it is indeed also always complex.
Btw, the files are ok and with effort can be understood and maybe some could even be considered elegant. It's just the shear amount of "standards" or rather the lack thereof.
Btw, the files are ok and with effort can be understood and maybe some could even be considered elegant. It's just the shear amount of "standards" or rather the lack thereof.
Hah bam files are the pinnacle of engineering compared to the clusterfuck that is VCF: GT or PL? Ploidy? Enumerating all possible ALTs but some with 0 AF? And then you have mutations in cis, try to merge them. Which assembly - you can guess because it doesn't have to be specified. You also don't have to specify your transcripts and of course you have better defaults than everyone else. How many different formats for annotating mutation effects are there 10? 20?
With webdev I can at least stick with jquery and do no harm
With webdev I can at least stick with jquery and do no harm
> With webdev I can at least stick with jquery and do no harm
The amount of time saved by doing this would be immense based on my experience inheriting maintenance projects in Anglaur / React / whatever flavour of the month framework was used. They may seem productive when developing, but trying to fix bugs in them can be an absolute nightmare.
The amount of time saved by doing this would be immense based on my experience inheriting maintenance projects in Anglaur / React / whatever flavour of the month framework was used. They may seem productive when developing, but trying to fix bugs in them can be an absolute nightmare.
Documentation is frighteningly bad across the industry and nobody has a good solution yet other than hoping that someone else will do something about it.
And if you think Django is murky, try Spring :). Django is one of the easiest frameworks to work with simply because it includes so many functions you would otherwise need to somehow integrate. And even then there are still these issues.
Good post.
And if you think Django is murky, try Spring :). Django is one of the easiest frameworks to work with simply because it includes so many functions you would otherwise need to somehow integrate. And even then there are still these issues.
Good post.
I think Django's documentation is fantastic. 90% of the time I have to look up something, google redirects me there anyway.
Django's documentation is great for developers who know their way around documentation like that as it is very comprehensive. It drills down nicely and is quick and easy to understand if you otherwise know what else is going on.
What it misses is mapping that documentation to the specific problem a person wants to solve or the feature they want to build. When I was learning Django a couple years ago I found the examples and best practices sparse.
What it misses is mapping that documentation to the specific problem a person wants to solve or the feature they want to build. When I was learning Django a couple years ago I found the examples and best practices sparse.
Compared to what I have seen with React + React add ons its fantastic.
("2 Scoops of Django" is an excellent book to describe best practices. It's something I haven't seen the equivalent of elsewhere).
("2 Scoops of Django" is an excellent book to describe best practices. It's something I haven't seen the equivalent of elsewhere).
A long long time ago, there was a really great website called djangobook.com. It had like 20 chapters and projects explaining how things are done in Django, and each paragraph had a comments section where people could ask questions specific to that paragraph. That site probably did a lot for Django's success in the community.
Unfortunately, giving away a free resource like that doesn't balance the books and years of neglect later the site no longer represented modern Django. It's entirely possible the new Django book site is just as good, but the paywall discourages novices and experts alike. My friends who stayed webdevs think Two Scoops of Django is close in scope, and comes with free updates. It's not quite the everbook that I had hoped the original site would be, but is highly focused on best practices.
Unfortunately, giving away a free resource like that doesn't balance the books and years of neglect later the site no longer represented modern Django. It's entirely possible the new Django book site is just as good, but the paywall discourages novices and experts alike. My friends who stayed webdevs think Two Scoops of Django is close in scope, and comes with free updates. It's not quite the everbook that I had hoped the original site would be, but is highly focused on best practices.
The Two Scoops book is good but I found it difficult to read due to its extremely contrived examples. Maybe that is just me though.
Not just you. Also annoyed at their pricing model; I bought the original book, 3 months later it was out of date and "buy our updated copy" was their answer. I got no use out of it.
I was so disappointed with other frameworks since Django was my first. I've come to realize this is widespread as I've learned other languages, but every time I start a new project and setting up the important boilerplate I'm constantly thinking in my head "Django would have just included this or there atleast be a battle tested plugin"
I think it depends on who you are as a dev. A lot want to create from scratch or have it be a handcrafted product, which I can respect.
I just like producing cool stuff and don't want to be manually running SQL queries in both dev and prod.
I just like producing cool stuff and don't want to be manually running SQL queries in both dev and prod.
Your comment comes off as haughty to me, as if you were saying that if you choose not to use a framework like Django its because you want to have a "handcrafted product" and because you don't care about "producing cool stuff", which sounds like a textbook definition of a bad (unproductive) programmer.
I assure you it's not, not using frameworks can also be a purely pragmatic choice. If you deviate from their intended use, they can very quickly become an annoyance instead of a time gainer.
I assure you it's not, not using frameworks can also be a purely pragmatic choice. If you deviate from their intended use, they can very quickly become an annoyance instead of a time gainer.
I am thinking more about the people who reflexively use Flask as Django is "bloated."
I never really got that unless people were complaining about Django's ORM. The framework is flexible enough that you can use as little or as much of it as you need.
Django has been a joy to work with. As other's have mentioned, I'm a huge fan of their documentation. I think they do a great job of letting you quickly hop between Tutorial and Reference docs. Lots of links help me quickly dive into more detail when I need, then jump back out.
Is there a better documented framework than Django? I am not aware of one.
My original comment misses some nuance.
Django's documentation is great for developers who know their way around documentation like that as it is very comprehensive. It drills down nicely and is quick and easy to understand if you otherwise know what else is going on.
What it misses is mapping that documentation to the specific problem a person wants to solve or the feature they want to build. When I was learning Django a couple years ago I found the examples and best practices sparse.
Django's documentation is great for developers who know their way around documentation like that as it is very comprehensive. It drills down nicely and is quick and easy to understand if you otherwise know what else is going on.
What it misses is mapping that documentation to the specific problem a person wants to solve or the feature they want to build. When I was learning Django a couple years ago I found the examples and best practices sparse.
Django is a big framework, it isn't aimed at beginners.
It's funny, when I was first learning programming 15 years ago, the thing I heard everywhere was "webdev is easy and for noobs", and sure, it was 15 years ago and webdev has come a long way... but when I interview the fresh-out-of-college candidates for junior positions, there's still the elitism of "If you aren't managing memory and flipping bits you aren't a REAL programmer."
No, most web developers aren't implementing the most complex and intricate algorithms or 'close to the metal' or whatever other gatekeeping people want to claim, but that doesn't mean it's easy work. I've never had a job where I didn't see some green engineer laugh off a feature request with "Oh, that's easy, it's just calling an API and putting the information into some HTML, what's so hard about that?" and then getting stuck when their first implementation fails QA, or the product manager explains that the customer needs it to be ever so slightly different, and the complexity of those edge cases turn out to not be so easy to deal with as the world of perfect computer science might suggest.
Is it hard in the same way? No, I will fully admit that coming up with a novel algorithmic solution requires a different mindset and brilliance that very few, if any, really good web developers I've met really have (although I'm certain there are people who excel at both, they're rare enough that I haven't been lucky enough to encounter them). But being really good at building a solid, secure, modern website isn't easy either, and it's also valuable.
This post focused on the complexity of the tooling, which yes, is definitely a thing, but if you don't know web development, and you know that you don't know web development, why are you attempting to build something using complex high level tools designed for and used by advanced web developers? Then again, if you don't know web development and you don't know what you don't know, and you've just heard 'Django is good' how would you know to maybe pick Flask or something simpler to understand?
I don't have an answer or a larger point other than this: writing software is difficult and complex and none of us should look down on anyone else because what they do seems easy on the outside.
No, most web developers aren't implementing the most complex and intricate algorithms or 'close to the metal' or whatever other gatekeeping people want to claim, but that doesn't mean it's easy work. I've never had a job where I didn't see some green engineer laugh off a feature request with "Oh, that's easy, it's just calling an API and putting the information into some HTML, what's so hard about that?" and then getting stuck when their first implementation fails QA, or the product manager explains that the customer needs it to be ever so slightly different, and the complexity of those edge cases turn out to not be so easy to deal with as the world of perfect computer science might suggest.
Is it hard in the same way? No, I will fully admit that coming up with a novel algorithmic solution requires a different mindset and brilliance that very few, if any, really good web developers I've met really have (although I'm certain there are people who excel at both, they're rare enough that I haven't been lucky enough to encounter them). But being really good at building a solid, secure, modern website isn't easy either, and it's also valuable.
This post focused on the complexity of the tooling, which yes, is definitely a thing, but if you don't know web development, and you know that you don't know web development, why are you attempting to build something using complex high level tools designed for and used by advanced web developers? Then again, if you don't know web development and you don't know what you don't know, and you've just heard 'Django is good' how would you know to maybe pick Flask or something simpler to understand?
I don't have an answer or a larger point other than this: writing software is difficult and complex and none of us should look down on anyone else because what they do seems easy on the outside.
I wouldn't count Flask as simpler than Django. These 5 lines hello world examples are incredibly misleading.
Django is an enormous project that dwarfs Flask in size and complexity.
Yes, Django takes care of the complexity so the developer does not have to. Flask is bare bones, so the complexity is left to the developer. I have done similar applications and websites with both and I know which one I'm more productive with. But of course YMMV and we are all lucky to have the choice.
Exactly.
You end up cobbling together half a dozen plugins (auth, forms, orm, admin) and adding dependencies that, albeit useful, are complex themselves: Celery, SQLAlchemy, etc.
You end up cobbling together half a dozen plugins (auth, forms, orm, admin) and adding dependencies that, albeit useful, are complex themselves: Celery, SQLAlchemy, etc.
Having a single source of documentation and components perfectly integrated out of the box make a huge difference.
You start with HTML. Write your HTML in an editor. You now have what we started with and I won't complain about your un-styled site. Right now I'm working on a simple personal thing with unstyled pages and a static site generator (Hugo.)
The first Google search for hosting should be fine. There should be a simple tutorial for uploading your HTML file.
That's all there is to it. Beyond that, maybe you're just trying to bite off more than you can chew because you think you need things. It would be like going to ACME engineering supply and grabbing a bunch of stuff off the shelf because it sounds good. Then you get home and you have no idea what these things do.
A lot of this stuff looks deceptively like it all goes under the category of "web development." Sending an email isn't web development. That's dealing with a different protocol and different server software. You don't need to go far down that path before you start to wonder how any of this mess can work at all. I'm not even talking about web development, I feel this way all the way down to the metal. Yesterday I spent hours trying to troubleshoot an issue with Ryzen 3rd generation chips apparently having a problem when you load up all the slots with RAM (I threw in the towel because I can get by with half the RAM.)
That has always been the case. Software hasn't gotten worse, it has always been bad. But functional. Maybe it has become more complex.
The first Google search for hosting should be fine. There should be a simple tutorial for uploading your HTML file.
That's all there is to it. Beyond that, maybe you're just trying to bite off more than you can chew because you think you need things. It would be like going to ACME engineering supply and grabbing a bunch of stuff off the shelf because it sounds good. Then you get home and you have no idea what these things do.
A lot of this stuff looks deceptively like it all goes under the category of "web development." Sending an email isn't web development. That's dealing with a different protocol and different server software. You don't need to go far down that path before you start to wonder how any of this mess can work at all. I'm not even talking about web development, I feel this way all the way down to the metal. Yesterday I spent hours trying to troubleshoot an issue with Ryzen 3rd generation chips apparently having a problem when you load up all the slots with RAM (I threw in the towel because I can get by with half the RAM.)
That has always been the case. Software hasn't gotten worse, it has always been bad. But functional. Maybe it has become more complex.
Anecdotally I have a Ryzen 3600 on a B450 board with all RAM slots populated and have yet to notice an issue that isn't there when using only half the RAM.
Probably the wrong place for the discussion. Do a search and you'll find all sorts of people with this issue. I had the problem after I first built the computer. Then I was able to boot by lowering the speed of the memory in the BIOS, now I have the problem again after a reboot. I don't know what changed. There's hundreds of threads which cover a lot of ground on this issue. I'm just going to make this a long term project to understand the issue until I finally cave and buy something which has already been built.
Edit: Also 3600 with B450 board. Kingston DDR4-2666 Ram (8x4) which is all the computer store here in smallish city in the Philippines had in stock.
Edit: Also 3600 with B450 board. Kingston DDR4-2666 Ram (8x4) which is all the computer store here in smallish city in the Philippines had in stock.
I 100% feel your pain.
I used to work for a very large financial news company. They were all javascript. I though to my self: I learnt JS in 2008, this should be easy (it was 2014). Apparently very hard.
HTML isn't HTML anymore JS is forced everywhere, and everything has a different name.
Anyway, I disappeared back into HPC/VFX/machinelearning, to re-appear into webdev in 2020.
I was forced to do some front end work as part of a bootcamp. It took the best part of a week to make a bit of text a link.
The template library was so far removed from anything googleable, there was no documentation, worse still no style guide or example style guide.
I am glad I don't do web GUIs, they are hard and nasty.
I used to work for a very large financial news company. They were all javascript. I though to my self: I learnt JS in 2008, this should be easy (it was 2014). Apparently very hard.
HTML isn't HTML anymore JS is forced everywhere, and everything has a different name.
Anyway, I disappeared back into HPC/VFX/machinelearning, to re-appear into webdev in 2020.
I was forced to do some front end work as part of a bootcamp. It took the best part of a week to make a bit of text a link.
The template library was so far removed from anything googleable, there was no documentation, worse still no style guide or example style guide.
I am glad I don't do web GUIs, they are hard and nasty.
It took the best part of a week to make a bit of text a link.
This highlights the problem with web dev quite nicely. Not doubt you weren't really just making some text in to a link. I know the sort of thing bootcamps do, so you were probably installing and learning React, displaying a set of components, implementing React Router, and figuring out how to update app state based on the browser's History API. All with a template library that was a terrible choice for learning (because it didn't have docs). You could summarize that as "making a bit of text a link", but that's glossing over the details.
What you were really doing was learning component orchestration, reactive state management, and a browser API. Why wouldn't that take a week?
So the problem with web dev is actually that it's really, really easy to trivialize. You can make hard things sound like they should be very straightforward because they look a lot like something simple from the past. And when people find that they're not really doing the simple thing any more they complain because the new, harder way of doing something (which is actually doing something completely different eg reactive state management and in-browser routing vs asking the browser to do a GET) is hard.
This highlights the problem with web dev quite nicely. Not doubt you weren't really just making some text in to a link. I know the sort of thing bootcamps do, so you were probably installing and learning React, displaying a set of components, implementing React Router, and figuring out how to update app state based on the browser's History API. All with a template library that was a terrible choice for learning (because it didn't have docs). You could summarize that as "making a bit of text a link", but that's glossing over the details.
What you were really doing was learning component orchestration, reactive state management, and a browser API. Why wouldn't that take a week?
So the problem with web dev is actually that it's really, really easy to trivialize. You can make hard things sound like they should be very straightforward because they look a lot like something simple from the past. And when people find that they're not really doing the simple thing any more they complain because the new, harder way of doing something (which is actually doing something completely different eg reactive state management and in-browser routing vs asking the browser to do a GET) is hard.
The actual task was this:
There is an Icon, and some text that reads: "you are on a trail find out more" The find out more is a link, make sure all the text _and_ the icon are included in the link.
So its not creating a new state or widget. its literally moving a href tag.
Well it would have been, if it wasn't for the template engine, and what ever the state engine was/is.
Now, if this was QT creator, I'm pretty sure I could have done this task in about 4 hours (I don't know C++ very well and I've used QT about 3 times before, each for less than a day.)
There is an Icon, and some text that reads: "you are on a trail find out more" The find out more is a link, make sure all the text _and_ the icon are included in the link.
So its not creating a new state or widget. its literally moving a href tag.
Well it would have been, if it wasn't for the template engine, and what ever the state engine was/is.
Now, if this was QT creator, I'm pretty sure I could have done this task in about 4 hours (I don't know C++ very well and I've used QT about 3 times before, each for less than a day.)
Which leads me onto the next gripe
As this is all templateable and composable. why in living fuck isnt there a GUI to do this sort of thing?
I mean, on a large website you need to make widgets, then you need to compose those widgets into screens. Those screens maintain state (basically flash, but with 100x worse performance)
the people that are styling the widgets are not always the same as those adding the logic, so why not make a tool where a visual artist can actually work visually?
So why on earth is it not drag and drop? Yes some things are auto generated based on context, but a lot of things are static screens where the widgets update the state.
As this is all templateable and composable. why in living fuck isnt there a GUI to do this sort of thing?
I mean, on a large website you need to make widgets, then you need to compose those widgets into screens. Those screens maintain state (basically flash, but with 100x worse performance)
the people that are styling the widgets are not always the same as those adding the logic, so why not make a tool where a visual artist can actually work visually?
So why on earth is it not drag and drop? Yes some things are auto generated based on context, but a lot of things are static screens where the widgets update the state.
the people that are styling the widgets are not always the same as those adding the logic, so why not make a tool where a visual artist can actually work visually?
Framer X. Sketch. Adobe Xd. etc
I mean, on a large website you need to make widgets, then you need to compose those widgets into screens.
Storybook (sort of).
So why on earth is it not drag and drop?
Dreamweaver.
The tools you want all exist. Those are just a few examples. There's hundreds more. You just didn't look very hard.
Framer X. Sketch. Adobe Xd. etc
I mean, on a large website you need to make widgets, then you need to compose those widgets into screens.
Storybook (sort of).
So why on earth is it not drag and drop?
Dreamweaver.
The tools you want all exist. Those are just a few examples. There's hundreds more. You just didn't look very hard.
And there's the problem...
"...component orchestration, reactive state management..."
Webdev tool makers are sadists, and many of the 'thought leaders' in the field have Stockholm syndrome.
Some aspects of web development certainly are hard (security especially).
Frontend development however, does not need to be as hard as it currently is. At the very least, I think there's an opportunity to simplify that layer of the stack, and give the control back to design professionals.
Dreamweaver to my mind, represented a more productive direction than React currently does.
"...component orchestration, reactive state management..."
Webdev tool makers are sadists, and many of the 'thought leaders' in the field have Stockholm syndrome.
Some aspects of web development certainly are hard (security especially).
Frontend development however, does not need to be as hard as it currently is. At the very least, I think there's an opportunity to simplify that layer of the stack, and give the control back to design professionals.
Dreamweaver to my mind, represented a more productive direction than React currently does.
On the contrary, front end isn't good enough yet, and getting to the point where it's ideal will require more complexity. Users want things to work fast. If you're including a roundtrip to a server your app will never be good enough for a user on a rural 1Mbps satellite connection with 500ms latency. That's the goal - your app has to work well for that guy. You can't get it to be fast if you're using the network a lot. We need concurrent rendering, async background processing, and as much work shifted on to worker threads as we can manage. All the things that native apps do to crank up the performance need to happen on the web too.
Also, don't forget that the web is fantastically backwards compatible. You can write web apps without the complexity if you want to. You can use Dreamweaver! You can server-side render every page. It works. If that's what you want to serve your users then you should do that. Just don't complain when they use the competitor's app that's faster, more responsive, and does fun things like working offline.
Also, don't forget that the web is fantastically backwards compatible. You can write web apps without the complexity if you want to. You can use Dreamweaver! You can server-side render every page. It works. If that's what you want to serve your users then you should do that. Just don't complain when they use the competitor's app that's faster, more responsive, and does fun things like working offline.
More than any framework, what most market leading applications need to provide a competitive service, is greater access to creator resource (developers / designers / content creators, whatever) and / or innovation in pre-production and design.
The problem with current development trends is in how the complexity found in the tools, makes the pool of available creators (at a want for a better term) a tiny, exclusive club. This has a detrimental effect on efforts to innovate at design / pre-production stage, because the designers are near enough held hostage by us developers.
It also has an overall effect on project velocity, because work that could possibly be covered by techically competent designers (namely HTML/CSS development), can only be addressed by expensive and exclusive 'framework specialists'.
I strongly believe this can change by rethinking where the complexity lies.
The culture built around over-complicated tooling is possibly analogous to placing the responsibility for the production of a feature film in the hands of the VFX and lighting teams, or perhaps only deeming a mechanic as having the aptitude to drive a car.
I'm not rigidly stuck to this idea, I'm just musing here.
The problem with current development trends is in how the complexity found in the tools, makes the pool of available creators (at a want for a better term) a tiny, exclusive club. This has a detrimental effect on efforts to innovate at design / pre-production stage, because the designers are near enough held hostage by us developers.
It also has an overall effect on project velocity, because work that could possibly be covered by techically competent designers (namely HTML/CSS development), can only be addressed by expensive and exclusive 'framework specialists'.
I strongly believe this can change by rethinking where the complexity lies.
The culture built around over-complicated tooling is possibly analogous to placing the responsibility for the production of a feature film in the hands of the VFX and lighting teams, or perhaps only deeming a mechanic as having the aptitude to drive a car.
I'm not rigidly stuck to this idea, I'm just musing here.
perhaps only deeming a mechanic as having the aptitude to drive a car
This analogy is a really good one, but you've got it slightly wrong. The website user is the person driving the car. The developer is the equivalent of the person building the car. Saying the development process is "too complicated" is like saying developers should only ever build compact family cars like a Ford, but some of us want to be building BMWs and Teslas and Porsches and F1 cars and monster trucks and semi-trucks, etc. Sure, it's more complex and difficult, but the end result suits the user better, fulfils their needs better, and often just makes them a lot happier.
This analogy is a really good one, but you've got it slightly wrong. The website user is the person driving the car. The developer is the equivalent of the person building the car. Saying the development process is "too complicated" is like saying developers should only ever build compact family cars like a Ford, but some of us want to be building BMWs and Teslas and Porsches and F1 cars and monster trucks and semi-trucks, etc. Sure, it's more complex and difficult, but the end result suits the user better, fulfils their needs better, and often just makes them a lot happier.
> If you're including a roundtrip to a server your app will never be good enough for a user on a rural 1Mbps satellite connection with 500ms latency.
That's a tiny minority of users. And that sort of user probably has an old, slow computer which is crushed under the weight of your vast javascript framework, anyway.
Whatever about a _better_ experience (some subjectivity there, though I don't really buy it) I'm pretty sure the modern web isn't a _faster_ experience for most users than it was a decade ago.
That's a tiny minority of users. And that sort of user probably has an old, slow computer which is crushed under the weight of your vast javascript framework, anyway.
Whatever about a _better_ experience (some subjectivity there, though I don't really buy it) I'm pretty sure the modern web isn't a _faster_ experience for most users than it was a decade ago.
That's a tiny minority of users. And that sort of user probably has an old, slow computer which is crushed under the weight of your vast javascript framework, anyway.
React isn't vast. React and ReactDOM together are about 40Kb over the wire of total download size if they're gzipped (https://bundlephobia.com/[email protected] && https://bundlephobia.com/[email protected]). For reference, the latest version of jQuery is 30Kb. Personally, I test all my production builds on a Chromebook that has 2GB of RAM, and they usually hit 60fps even on relatively complex pages. This is what the standard should be for building for the web - small, fast web apps that work well on slow computers and slow, high latency internet connections.
You can decide not to support everyone if you want, but that's a decision. The technology already works quite well if you put the effort in to building things properly. Build complexity is not a good excuse for serving slow apps.
React isn't vast. React and ReactDOM together are about 40Kb over the wire of total download size if they're gzipped (https://bundlephobia.com/[email protected] && https://bundlephobia.com/[email protected]). For reference, the latest version of jQuery is 30Kb. Personally, I test all my production builds on a Chromebook that has 2GB of RAM, and they usually hit 60fps even on relatively complex pages. This is what the standard should be for building for the web - small, fast web apps that work well on slow computers and slow, high latency internet connections.
You can decide not to support everyone if you want, but that's a decision. The technology already works quite well if you put the effort in to building things properly. Build complexity is not a good excuse for serving slow apps.
It's rarely just react, though.
It's possible in principle to build small fast SPAs, but they're vanishingly rare in real life. And even where code size is small, CPU usage and latency are often bizarrely high; even that 500ms latency user might find a roundtrip to the server faster than waiting on some SPAs.
I do a lot of my personal browsing on an elderly (six year old) iPad. It's totally fine for normal web pages, but every year the number of websites which are painful on it rises, due to (usually largely unnecessary) javascript.
It's possible in principle to build small fast SPAs, but they're vanishingly rare in real life. And even where code size is small, CPU usage and latency are often bizarrely high; even that 500ms latency user might find a roundtrip to the server faster than waiting on some SPAs.
I do a lot of my personal browsing on an elderly (six year old) iPad. It's totally fine for normal web pages, but every year the number of websites which are painful on it rises, due to (usually largely unnecessary) javascript.
Sure there's lots of reasons to not use React and many places don't. Bad management/decision making is more to blame in this case than anything.
If a simple link to another page requires component orchestration and reactive state management I'd say it is a bit over-engineered.
> JS is forced everywhere
I think this has a lot to do with it, interestingly. The web has become so ubiquitous, and the major players are so resourced, that the barrier to entry can seem really high. Maybe I'm wrong, but my perception is that potential customers wouldn't take a second glance at my sight if it were plain html with minimal CSS and didn't render perfectly on mobile.
> I am glad I don't do web GUIs
Same.
I think this has a lot to do with it, interestingly. The web has become so ubiquitous, and the major players are so resourced, that the barrier to entry can seem really high. Maybe I'm wrong, but my perception is that potential customers wouldn't take a second glance at my sight if it were plain html with minimal CSS and didn't render perfectly on mobile.
> I am glad I don't do web GUIs
Same.
You can make a very lovely site with just HTML and CSS! CSS is extremely powerful by itself. As long as you don’t need any dynamic content or dynamic UI, you can likely avoid JS entirely :)
You can, but if you’re doing any sort of non-static website then you still have complexities in the backend to deal with. (For static sites, I fully agree that HTML and CSS are more than enough most of the time; at least, they’re good enough for my website!)
> As long as you don’t need any dynamic content or dynamic UI
People seem to be ordering more of that, not less. Look at Reddit. They have all manner of stupid animations now.
People seem to be ordering more of that, not less. Look at Reddit. They have all manner of stupid animations now.
You can do all kinds of stupid animations with pure CSS.
Honestly, they're only hard and nasty if you make them hard and nasty. A text link is as easy as <a href="">...</a>.
You can of course use React with 2 different libraries to achieve the same goal, but why would you do that.
You can of course use React with 2 different libraries to achieve the same goal, but why would you do that.
React by itself isn't difficult either. Anyone who has an hour or so to spare can go through the tutorial https://reactjs.org/tutorial/tutorial.html.
They need to rewrite it to use hooks at some point, but it's still a pretty good intro.
They need to rewrite it to use hooks at some point, but it's still a pretty good intro.
> They need to rewrite it to use hooks at some point
What is the reason that not using hooks is so frowned upon? When I took my first steps with React, I didn't know hooks, so I write component classes and it immediately "clicked" with me because it behaves like most other GUI frameworks do, from Win32-based (1) to X-based to Android to iOS to macOS to Java, and tons of higher-level frameworks in between. It also feels obvious to write a component class because you are describing the behavior of a stateful thing, that is, exactly what classes are made for.
With hooks, OTOH, we have a stateful "function", but all state is implicit, which describes behavior other than its main behavior (rendering) but that is implicit too... I can see why developers outside the React world don't "get" hooks, because I can't even remember seeing this idiom outside the React world. (BTW, I've seen the claim that hooks are easier to reason about, for which I found the opposite to be true, and the claim that this makes the main function stateless, which is absurd since the state just becomes implicit).
(1) the raw system-level APIs are often not class-based, but nearly all frameworks above it are.
edit: the footnote broke formatting
What is the reason that not using hooks is so frowned upon? When I took my first steps with React, I didn't know hooks, so I write component classes and it immediately "clicked" with me because it behaves like most other GUI frameworks do, from Win32-based (1) to X-based to Android to iOS to macOS to Java, and tons of higher-level frameworks in between. It also feels obvious to write a component class because you are describing the behavior of a stateful thing, that is, exactly what classes are made for.
With hooks, OTOH, we have a stateful "function", but all state is implicit, which describes behavior other than its main behavior (rendering) but that is implicit too... I can see why developers outside the React world don't "get" hooks, because I can't even remember seeing this idiom outside the React world. (BTW, I've seen the claim that hooks are easier to reason about, for which I found the opposite to be true, and the claim that this makes the main function stateless, which is absurd since the state just becomes implicit).
(1) the raw system-level APIs are often not class-based, but nearly all frameworks above it are.
edit: the footnote broke formatting
Regardless of how you feel about hooks, they are, or are quickly becoming, the new standard for most React code, so a React tutorial obviously needs to familiarize people with them. The current tutorial is good at explaining React concepts though, so making the jump to hooks after reading it and doing a few simple playground projects should be trivial.
As for why people like them so much as opposed to classes, my guess would be that they let people more easily write reusable behaviors for common tasks (e.g. look at this one for integrating with RxJS Observables for projects using RxJS https://github.com/LeetCode-OpenSource/rxjs-hooks#examples - with classes you would have to put code in several lifecycle methods, sometimes duplicate code, for each individual component).
As for why people like them so much as opposed to classes, my guess would be that they let people more easily write reusable behaviors for common tasks (e.g. look at this one for integrating with RxJS Observables for projects using RxJS https://github.com/LeetCode-OpenSource/rxjs-hooks#examples - with classes you would have to put code in several lifecycle methods, sometimes duplicate code, for each individual component).
Classes look familiar, but that's misleading because they don't actually work like one would expect.
Hooks are actually much closer to what react does internally, and therefore behave much less surprisingly once you get into advanced things. Think of hooks like a missing primitive (similar to functions, classes and closures) that are critical to the react language, but don't actually exist in JavaScript and thus have to be approximated.
Hooks are actually much closer to what react does internally, and therefore behave much less surprisingly once you get into advanced things. Think of hooks like a missing primitive (similar to functions, classes and closures) that are critical to the react language, but don't actually exist in JavaScript and thus have to be approximated.
> Classes look familiar, but that's misleading because they don't actually work like one would expect.
Can you give an example for that? So far, they worked pretty much as I expected. (Coming from a mostly Java background, which is relevant for what I expect).
Can you give an example for that? So far, they worked pretty much as I expected. (Coming from a mostly Java background, which is relevant for what I expect).
Hooks are a fair bit easier to manage than classes in React.
It won't change what you do with React much, but from file to file, it's more pleasant to read a hook based component, when compared to a class based component, once you're used to it.
Hooks also simplify working with React context a great deal. If that's your option for state management.
Hooks also simplify working with React context a great deal. If that's your option for state management.
As a frontend developer with a great deal of React experience, I respectfully disagree.
React is unnecessarily hard, and this fact probably keeps the UI development out of the hands of the people who could potentially do the best work (graphic designers).
If you want, there are APIs and frameworks that will allow programmers to configure print layouts and typography by hand.
As you can imagine, they're not as well known or as popular as the Adobe tools used for this work.
I'm not saying that web based UI development isn't without its unique challenges (state management), but I will say that things could stand to change, and we should move towards simplifying stacks, for what will ultimately be the user's sake.
React is unnecessarily hard, and this fact probably keeps the UI development out of the hands of the people who could potentially do the best work (graphic designers).
If you want, there are APIs and frameworks that will allow programmers to configure print layouts and typography by hand.
As you can imagine, they're not as well known or as popular as the Adobe tools used for this work.
I'm not saying that web based UI development isn't without its unique challenges (state management), but I will say that things could stand to change, and we should move towards simplifying stacks, for what will ultimately be the user's sake.
I start getting my friends up to speed by giving them just enough JavaScript + HTML to write a Tetris clone. Of course - I guide them along the way, answer any questions they have, and push them towards the right direction if they get stuck. If they can get through it - then they can make it in the world of WebDev, and software engineering in general.
The main thing to remember is that you can only show the way - the person in question must actually step forwards and be motivated enough to go through the learning process. Otherwise - you need to think of another career. Programming is not for everyone IMHO.
The main thing to remember is that you can only show the way - the person in question must actually step forwards and be motivated enough to go through the learning process. Otherwise - you need to think of another career. Programming is not for everyone IMHO.
A little tangential, but learning to code through video games has always seemed like one of the best ways to do it, particularly with friends.
And it looks like you've just scratched the surface, you're not finished encountering walls yet.
Hang in there, there's a relative important chance that the fog will clear for you in some months/years.
The hard truth is, software engineering today is made of layers upon layers of abstractions. You have to dig threw those layers, understanding sufficiently the inherent and different complexity of each one before jumping to the next. And then finally it'll make sense, and new layers (there's a new layer every year btw) will be easier to grok.
If it helps, we've all had to go threw this learning process. All of us. However I don't think everyone is made for enjoying this process, that's another hard truth.
The real difficulty is sticking to the learning process while everything and everyone is screaming at you "Go! Build something! Ship Something! Show something you lazy b".
Hang in there, there's a relative important chance that the fog will clear for you in some months/years.
The hard truth is, software engineering today is made of layers upon layers of abstractions. You have to dig threw those layers, understanding sufficiently the inherent and different complexity of each one before jumping to the next. And then finally it'll make sense, and new layers (there's a new layer every year btw) will be easier to grok.
If it helps, we've all had to go threw this learning process. All of us. However I don't think everyone is made for enjoying this process, that's another hard truth.
The real difficulty is sticking to the learning process while everything and everyone is screaming at you "Go! Build something! Ship Something! Show something you lazy b".
Http is just clearly not made for the current state of the internet. It was obviously made for much simpler uses cases but has organically grown together with the internet. This has led to numerous hacks and workaroudns which often became the defacto standard to do things on the web. It's not necessarily an easy problem to solve, client server communication is hard. But http protocol seems like the least effective way to do it, but almost impossible to replace now.
Applies even more so to CSS and JS.
All the above are quick hacks that never really did the job properly in the first place. So they're propped up by a creaking pile of frameworks and other support systems that try to mitigate their awfulness, but mostly just add extra awfulness of their own.
Meanwhile backend systems - core server and db infrastructure, user authentication, computing resource allocation, payments, media and content management, privacy, and project build systems - are even more byzantine and non-standardised and more or less have to be retooled from scratch for every project - a huge job which duplicates an incredible amount of work across the entire industry.
All the above are quick hacks that never really did the job properly in the first place. So they're propped up by a creaking pile of frameworks and other support systems that try to mitigate their awfulness, but mostly just add extra awfulness of their own.
Meanwhile backend systems - core server and db infrastructure, user authentication, computing resource allocation, payments, media and content management, privacy, and project build systems - are even more byzantine and non-standardised and more or less have to be retooled from scratch for every project - a huge job which duplicates an incredible amount of work across the entire industry.
> Http is just clearly not made for the current state of the internet
As a corollary/reformulation of that sentence you get that what businesses are trying to do in the web is clearly something it wasn't made for. Yet businesses want to free-ride on the web's success, and are ruining it.
As a corollary/reformulation of that sentence you get that what businesses are trying to do in the web is clearly something it wasn't made for. Yet businesses want to free-ride on the web's success, and are ruining it.
> or the “right” framework to choose, changes from year to year
I've been using React in all of my projets since it came out in 2013. That's 7 years. React's (+ node for backend if needed) been one of the recommended for at least 4 years.
And lots of things have changed ! Using modules and building complex apps now with Webpack & co is so much easier than before. Not needing to support IE is also a blessing.
The piece doesn't say how mobile dev would be easier. Or how embedded dev would be easier.
It just says : dev is hard. Yes it is, it's a a full job.
Hopefully nocde solutions will reduce the gap for some use cases.
I've been using React in all of my projets since it came out in 2013. That's 7 years. React's (+ node for backend if needed) been one of the recommended for at least 4 years.
And lots of things have changed ! Using modules and building complex apps now with Webpack & co is so much easier than before. Not needing to support IE is also a blessing.
The piece doesn't say how mobile dev would be easier. Or how embedded dev would be easier.
It just says : dev is hard. Yes it is, it's a a full job.
Hopefully nocde solutions will reduce the gap for some use cases.
React - as a library, but especially as a UI paradigm - has clearly won the frontend war for now, taking the mantle from JQuery. I don't even use React directly, just its DOM renderer through a ClojureScript wrapper (reagent) and the high-level concepts are still basically the same.
Still, when I glance at the JS frontend world people are still looking for the next hot frontend framework all the time, even if many are doing something similar to React. Going by github stars, Vue is the biggest one now. There's also Angular, Svelte, Ember, and Aurelia.
Still, when I glance at the JS frontend world people are still looking for the next hot frontend framework all the time, even if many are doing something similar to React. Going by github stars, Vue is the biggest one now. There's also Angular, Svelte, Ember, and Aurelia.
You're right. React and Vue are hard to compare though. The fact that React has no router, while react-router has 40k stars makes it a very imprecise metric.
Yes, web development is hard and stupidly complicated and slapdash and we all hate it. It sucks, and the whole thing should be burned to the ground, the ashes sown with salt, and pissed on by donkeys.
Unfortunately we're stuck with this Rube Goldberg collection of excrement, and it pays the bills. So there's nothing much you can do except lean in and embrace the suck. Maybe hit yourself in the forehead a few times with a clawhammer or start carrying a hipflask to keep yourself at happy-drunk.
Unfortunately we're stuck with this Rube Goldberg collection of excrement, and it pays the bills. So there's nothing much you can do except lean in and embrace the suck. Maybe hit yourself in the forehead a few times with a clawhammer or start carrying a hipflask to keep yourself at happy-drunk.
Very thoughtful article. There is so much accidental complexity in software.
It's kind of ironic. While I've been thinking about the complexity of software/webdev for a while, I was feeling a little unsure about posting this particular link because I wrote it in a hasty, semi-embittered fit of annoyance. Regardless, thanks for the read. :)
hopefully this observation won't be taken too harshly...
I have also worked in programming for the sciences my entire career, multiple decades, would not want to do anything else. There is an awful lot of actual complexity out there especially in the life sciences which is inherently messy ... a system of exceptions ... Sometimes looking at the stuff that has to happen now, just to give information away for free it is staggering. The only conclusion I come to is complexity envy, Because I can't believe all this inevitablely became necessary. Yes there are bad actors and good security is very hard, and much more so if you are transacting money, but in other cases you can't really steal what is being given away but now there is the expectation that all this extra stuff is par for the course it you are putting on the web.
I have also worked in programming for the sciences my entire career, multiple decades, would not want to do anything else. There is an awful lot of actual complexity out there especially in the life sciences which is inherently messy ... a system of exceptions ... Sometimes looking at the stuff that has to happen now, just to give information away for free it is staggering. The only conclusion I come to is complexity envy, Because I can't believe all this inevitablely became necessary. Yes there are bad actors and good security is very hard, and much more so if you are transacting money, but in other cases you can't really steal what is being given away but now there is the expectation that all this extra stuff is par for the course it you are putting on the web.
I love these kinds of retrospectives as they give insight into the learning process. We tend to focus on the acquisition of specialized knowledge but there is often a body of shared knowledge, the context you might call it, that we pick up indirectly.
You have to read between the lines but what we have in this post is a Data Scientist skilled in the Python programming language struggling to acquire the shared knowledge required to build and deploy Web Apps. One toolchain (Python), one framework (Django), one backend architecture (12-Factor Apps) but the narrative surrounding URLs, HTTP, DNS, TLS, REST APIs, JSON, CORS, and www-forms is missing.
We seem to be missing a simple way to learn the shared knowledge needed to build apps in a world reshaped by Web, Mobile, and Cloud technologies.
You have to read between the lines but what we have in this post is a Data Scientist skilled in the Python programming language struggling to acquire the shared knowledge required to build and deploy Web Apps. One toolchain (Python), one framework (Django), one backend architecture (12-Factor Apps) but the narrative surrounding URLs, HTTP, DNS, TLS, REST APIs, JSON, CORS, and www-forms is missing.
We seem to be missing a simple way to learn the shared knowledge needed to build apps in a world reshaped by Web, Mobile, and Cloud technologies.
I’m not sure exactly what it is about this comment, but I really like it. “Context” is a great word for this idea. There are a bunch of small/medium complexity things that need to play well together to get things to work, and it can feel trickier than necessary.
Honestly I've always found UI development or anything with some aspect of UI in general has the potential to devolve into a tangled mess. I've avoided web facing development for the last 10 years of my career, and have transitioned to embedded work. Every time I traipse through job postings and look at the tech stack companies are using ("wanted: full stack developer") I just shake my head. I am sure most of the choices are justified, but it all seems so brutally overwrought from the outside looking in. But beyond a fetish for novelty in our industry, I think that's just endemic to user facing development.
Trying to make the opposite transition, my quibble with web development is that much of the work seems so pointless.
I hope that after this crisis we will care less about making people click ads than about learning how stuff works in biology.
I hope that after this crisis we will care less about making people click ads than about learning how stuff works in biology.
Trying to make the opposite transition, my quibble with web development is that much of the work seems so pointless.
That's the problem with web development today. Something that not only should be straightforward, but used to be straightforward, is now very complex. Partly because it involves connecting a large number of complex and constantly changing packages.
The last time I wanted to do something with an API, I coded up the API in Go and ran it from FCGI to get multiple copies. Easier than all those "frameworks". Probably faster, too, since Go is hard-compiled.
That's the problem with web development today. Something that not only should be straightforward, but used to be straightforward, is now very complex. Partly because it involves connecting a large number of complex and constantly changing packages.
The last time I wanted to do something with an API, I coded up the API in Go and ran it from FCGI to get multiple copies. Easier than all those "frameworks". Probably faster, too, since Go is hard-compiled.
>used to be straightforward
As someone who remembers having to do weird CSS tricks for really basic layouts (negative margins? hidden extra elements that don't make any sense, but cause other elements to move?), manually modifying the UI made up of jQuery plugins that require a very specific prescribed HTML structure to even work, or doing significant adjustments for 3 or 4 different browsers, I definitely dispute that claim.
Web development right now (at least the frontend for sure) feels like a breeze compared to about 10-15 years ago.
Steps 1 and 2 of the whole process may have gotten more difficult, but steps 5 through 100 are a lot easier these days.
As someone who remembers having to do weird CSS tricks for really basic layouts (negative margins? hidden extra elements that don't make any sense, but cause other elements to move?), manually modifying the UI made up of jQuery plugins that require a very specific prescribed HTML structure to even work, or doing significant adjustments for 3 or 4 different browsers, I definitely dispute that claim.
Web development right now (at least the frontend for sure) feels like a breeze compared to about 10-15 years ago.
Steps 1 and 2 of the whole process may have gotten more difficult, but steps 5 through 100 are a lot easier these days.
I heard there's a great new site that can help you learn about bio if you're interested... You know, if the site will let you login, that is. ;)
[Looks like you've read 0.1% of this page! Enter your email into this popup to receive emails you don't want about things you're not interested in. Also turn off your ad blocker, because we're not a charity, you freeloader.]
We still run into the problem of how to pay for content on the web. Do we really want everything to either be volunteer or sponsored by someone?
it's a public good that will likely be underfunded if we stick with volunteers and sponsored
except maybe if status is a sufficient payment?
mayyybe good if altruistic ideas are more likely to propagate under those incentives?
except maybe if status is a sufficient payment?
mayyybe good if altruistic ideas are more likely to propagate under those incentives?
Backwards compatibility.
It's possible to design coherent easy to use UI systems. Flutter and Qt are great and both much easier to develop for than the web IMO. But they don't have to worry about running in web browsers.
It's possible to design coherent easy to use UI systems. Flutter and Qt are great and both much easier to develop for than the web IMO. But they don't have to worry about running in web browsers.
I don't think the author is saying "why front end application development hard?" so much as "why is setting up a website hard?".
On one end of the spectrum, simply putting content on a website isn't so hard, right? On the other end, making a web application is understandably difficult.
Features like users, payment forms, and email confirmation (so e.g. the website can be used for a course) sounds like they ought to be as simple as the former, but are closer to the latter.
Is the value proposition from the website building websites like Wix that they take care of the hard stuff?
On one end of the spectrum, simply putting content on a website isn't so hard, right? On the other end, making a web application is understandably difficult.
Features like users, payment forms, and email confirmation (so e.g. the website can be used for a course) sounds like they ought to be as simple as the former, but are closer to the latter.
Is the value proposition from the website building websites like Wix that they take care of the hard stuff?
[deleted]
People who understand the mess are much more influential and they are moving the show forwards. Making things easy for newcomers is just not very high on the agenda (anymore)
Perhaps things have to get complicated enough for it to get priority, perhaps cool things will happen all of a sudden. I was highly amused by the spreadsheet programming tool a while back. It seems a giant leap in the other direction.
Perhaps things have to get complicated enough for it to get priority, perhaps cool things will happen all of a sudden. I was highly amused by the spreadsheet programming tool a while back. It seems a giant leap in the other direction.
This is common among academics, who expect full transparency and understanding in what they're doing, instead of a "get it working and go" attitude.
The problem is the OP didn't test anything: He launched and users couldn't log in.
Ideally I'd like to gain 100% visibility into everything related to my tasks in webdev, but it's impractical and counter productive.
The problem is the OP didn't test anything: He launched and users couldn't log in.
Ideally I'd like to gain 100% visibility into everything related to my tasks in webdev, but it's impractical and counter productive.
It is hard to get the hang of Django is because the polls tutorial isn't helpful to actually understand Django and no one wrote this book yet, but I think Nigel has done a great job:
https://djangobook.com/mastering-django-2-book/
[deleted]
Related: institutions and labs need to recognize that 'research software engineer' is an increasingly necessary role and attract/fund more of them.
Because instead of fixing genuine problems and making it easier people invent new languages and frameworks to do things in their own way.
It is no brainer for professionals, but to be frank, it's too hard for everyone. It's time for a new PHP.
looks like it got hugged
https://web.archive.org/web/20200507060013/https://jessimeki...
https://web.archive.org/web/20200507060013/https://jessimeki...
Oh no! It's github pages and totally static, so it should recover pretty easily, I hope. Thanks for the archive though!
Looks fine on my end. Good choice on using GitHub Pages by the way; had you not I’m sure half the comments would have been about how you would have it so much easier and moral it would be had you used that ;)
Must have been a blip on my end then.
It definitely got hugged, I couldn't access it either for a bit. Recovered quickly enough though
Is the author asking why putting up a paywall for Django hosted on Heroku with Stripe integration on what should be static content is difficult (for stories related to overcoming idiosyncrasies in using Python for bioinformatics, or all things)? I'd like to give a rebuttal, but the site (the Minimal Errors theme for Jekyll?) does click-jacking (blocks copy/paste) so I won't bother. I guess the answer is right there: webdev (and bioinformatics) is full of incidental complexity and zero-sum "solutions" advertised to you when the original purpose of the web is just publishing structured documents via HTML/SGML (even though I'd agree Python is great for learning, and Heroku has at least a reputation for being the easiest cloud hosting option out there).
The site that's giving me problems isn't the site you were on, but another site I'm building:
https://mycodestories.com/
It's not static; it's a platform for helping people (life scientists) learn to code.
In general, Heroku has been a great platform. The fact that you can just `git push`, and your dyno rebuilds and restarts and your changes are live is so cool.
https://mycodestories.com/
It's not static; it's a platform for helping people (life scientists) learn to code.
In general, Heroku has been a great platform. The fact that you can just `git push`, and your dyno rebuilds and restarts and your changes are live is so cool.
A great read!
> Why’s there so much data in the url? A common refrain I’ll use for the remainder of the post is, “I don’t know”. One of my biggest gripes with web programming is the number of rabbit holes to fall down. There’s just not enough time.
I mean, bioinformatics is _notorious_ for spending just enough time working on software to publish a result and no more. If we want to talk about rabbitholes I nominate BioPerl.
> here are a list of terms that you’d never come across as a bioinformatician: ["X-Forwarded-Proto", "proxying", "requests", "TLS connection", "HTTP", "loopback/localhost", "whitelisted proxy servers", "gunicorn", "nginx", "Apache/httpd", "HAProxy", "Django", "SECURE_SSL_REDIRECT", "SecurityMiddleware", "adapter", "URI"]
This list is more than a little disingenuous:
Django: This is like, the framework OP built their software on top of. I don't know how one gets the point of filing this bug without knowing the term Django
requests: literally, a user's web request.
HTTP/URI: this is the web developer equivalent of a biologist not knowing the term DNA. Or DOI for that matter.
whitelist: the opposite of a blacklist
Much of rest comes down to the simple fact that job of web developers is to build a network service. This is primarily done with layers; each layer only needs to know how to interface with the adjoining layers above and below, like a lego brick. This is actually a _solution_ to the complexity, allowing us to constrain the design of each layer to interact only with those above and below it. This layer abstraction lets us reason about each layer locally, independently of the system as a whole.
nginx, Apache/httpd, HAProxy: in this context, mentioned as possible proxies forming the outermost layer. in this context, accepts HTTP/HTTPS connections and makes HTTP connections to gunicorn
gunicorn: the thing that runs the Django app. Accepts HTTP connections, makes WSGI calls to the Django app
Django: Accepts WSGI function calls, routes them through middleware, then to the app as you define in urlpatterns
SecurityMiddleware: a class that potentially modifies inbound requests and outbound responses
Finally, OP is in the unhappy situation of bridging two network protocols: HTTP and email. The normal trick of using a protocol relative URI will not work here because you've gone outside the browser request-response loop. This puts them in the unhappy state of the lowest layer needing to know the details of the highest layer, hence the discussion of absolute uris. This is a direct violation of the local reasoning constraint, and what makes all of this hard.
X-Forwarded-Proto: An HTTP header, added to incoming HTTP requests designed to indicate to layers below which URI scheme was originally used, HTTP or HTTPS.
SECURE_SSL_REDIRECT: a setting for SecurityMiddleware that will stop processing the request and send a response to the browser telling it to resend the query over https (an HTTP 302 redirect), if the incoming connection is seen as coming over not http.
tl;dr: email is the thing that sucks.
I mean, bioinformatics is _notorious_ for spending just enough time working on software to publish a result and no more. If we want to talk about rabbitholes I nominate BioPerl.
> here are a list of terms that you’d never come across as a bioinformatician: ["X-Forwarded-Proto", "proxying", "requests", "TLS connection", "HTTP", "loopback/localhost", "whitelisted proxy servers", "gunicorn", "nginx", "Apache/httpd", "HAProxy", "Django", "SECURE_SSL_REDIRECT", "SecurityMiddleware", "adapter", "URI"]
This list is more than a little disingenuous:
Django: This is like, the framework OP built their software on top of. I don't know how one gets the point of filing this bug without knowing the term Django
requests: literally, a user's web request.
HTTP/URI: this is the web developer equivalent of a biologist not knowing the term DNA. Or DOI for that matter.
whitelist: the opposite of a blacklist
Much of rest comes down to the simple fact that job of web developers is to build a network service. This is primarily done with layers; each layer only needs to know how to interface with the adjoining layers above and below, like a lego brick. This is actually a _solution_ to the complexity, allowing us to constrain the design of each layer to interact only with those above and below it. This layer abstraction lets us reason about each layer locally, independently of the system as a whole.
nginx, Apache/httpd, HAProxy: in this context, mentioned as possible proxies forming the outermost layer. in this context, accepts HTTP/HTTPS connections and makes HTTP connections to gunicorn
gunicorn: the thing that runs the Django app. Accepts HTTP connections, makes WSGI calls to the Django app
Django: Accepts WSGI function calls, routes them through middleware, then to the app as you define in urlpatterns
SecurityMiddleware: a class that potentially modifies inbound requests and outbound responses
Finally, OP is in the unhappy situation of bridging two network protocols: HTTP and email. The normal trick of using a protocol relative URI will not work here because you've gone outside the browser request-response loop. This puts them in the unhappy state of the lowest layer needing to know the details of the highest layer, hence the discussion of absolute uris. This is a direct violation of the local reasoning constraint, and what makes all of this hard.
X-Forwarded-Proto: An HTTP header, added to incoming HTTP requests designed to indicate to layers below which URI scheme was originally used, HTTP or HTTPS.
SECURE_SSL_REDIRECT: a setting for SecurityMiddleware that will stop processing the request and send a response to the browser telling it to resend the query over https (an HTTP 302 redirect), if the incoming connection is seen as coming over not http.
tl;dr: email is the thing that sucks.
Because it's artificially made hard by people who enjoy complexity for its own sake. The article is about implementing a shopping cart, as far as I can make out. Well people used to write those kinds of websites, that perfectly well enabled users to browse products, fill a basket, place the order and make a payment using nothing more than notepad.exe.
Some software is legitimately hard to write, even some websites. But this isn't an example of that. The author has been sent on a wild goose chase.
Some software is legitimately hard to write, even some websites. But this isn't an example of that. The author has been sent on a wild goose chase.
I'm curious where you got the idea that the site is a shopping cart?
CodeStories is a site to help life scientists learn to program by giving them problems, then letting them submit scripts and then auto-graded their scripts for correctness.
CodeStories is a site to help life scientists learn to program by giving them problems, then letting them submit scripts and then auto-graded their scripts for correctness.
CodeStories is a site to help life scientists learn to program by giving them problems, then letting them submit scripts and then auto-graded their scripts for correctness.
Because it’s a pretty generic e-commerce application that could have been built on the web 25 years ago. It should not have been complicated to do.
Because it’s a pretty generic e-commerce application that could have been built on the web 25 years ago. It should not have been complicated to do.
"I am a professional in Field X but I am doing work in Field Y despite having minimal skills or experience and I ran into loads of problems and issues that I don't know how to fix! Field Y is total bullshit and people need to make it easier so people with no skills or experience can do it! Wah wah!"
... or to be less facetious, modern web/frontend development is a genuine skill set that benefits from experience and knowledge.
Should we expect anyone to just jump in cold with minimal experience and not find it difficult? E.g. in the same way would we expect people to start bioinformatics cold and be productive immediately?
I'd doubt it.
On the past web dev has been looked down upon as somehow "lesser" than "real" engineering, with sneering criticism from people more used to traditional development approaches (particularly criticism of JavaScript). It is just some web pages right? HTML and CSS? How hard can it be?
Well guess what: it is a genuine skillset and people should not expect to just be able to jump into something without knowledge training or experience and be proficient at it, just like we'd not for brain surgery, car mechanics, concert piano, structural engineering, or indeed bioinformatics.
</rant>