Training an AI to convert design mockups into HTML and CSS(medium.freecodecamp.org)
medium.freecodecamp.org
Training an AI to convert design mockups into HTML and CSS
https://medium.freecodecamp.org/how-you-can-train-an-ai-to-convert-your-design-mockups-into-html-and-css-cc7afd82fed4
108 comments
Do we really have to crap all over this person's work? It's clearly intended as a learning exercise / tutorial written by someone who's in the process of learning themselves, and not as "breaking news" or a production-ready system.
IMHO we should be encouraging this sort of engaged learning.
IMHO we should be encouraging this sort of engaged learning.
If he just try to learn something it is unappropriate to start the article with this sentence : "Within three years, deep learning will change front-end development. It will increase prototyping speed and lower the barrier for building software."
In three years software like this would absolutely change front-end development. Already is front-end development constantly progressing every year.
PSD to HTML services were outsourced a decade ago for a reason.
PSD to HTML services were outsourced a decade ago for a reason.
Hopefully an AI won't insist on halting the sprint to change build tooling for the 3rd time in as many years at least.
A usable service taking advantage of this is still a long way off, but we don't get there unless people do the legwork.
One thing that could be usable more quickly would involve simplifying the problem space a bit. For example, the airbnb example using symbols could translate to, say, react components. If the grid system is well designed and you have a mature library of components, one could easily come up with interfaces that would generate react code, and ideally work well enough to serve as a starting point for the engineer to take over and turn it into a working app.
A system like this also need not be used in production-ready apps - I can envision a scenario where PM and designer can quickly sketch out and test ideas with working interfaces, without direct engineer involvement. That in and of itself would change front-end development significantly as the article claims.
One thing that could be usable more quickly would involve simplifying the problem space a bit. For example, the airbnb example using symbols could translate to, say, react components. If the grid system is well designed and you have a mature library of components, one could easily come up with interfaces that would generate react code, and ideally work well enough to serve as a starting point for the engineer to take over and turn it into a working app.
A system like this also need not be used in production-ready apps - I can envision a scenario where PM and designer can quickly sketch out and test ideas with working interfaces, without direct engineer involvement. That in and of itself would change front-end development significantly as the article claims.
Found the front end developer :D
But all jokes aside, I have to agree with others replying to your comment that this looks to be a learning exercise. It shouldn't be berated or beaten with a club. Without this kind of curiosity, the tech industry wouldn't be where it's at today.
But all jokes aside, I have to agree with others replying to your comment that this looks to be a learning exercise. It shouldn't be berated or beaten with a club. Without this kind of curiosity, the tech industry wouldn't be where it's at today.
From the article:
"Within three years, deep learning will change front-end development. It will increase prototyping speed and lower the barrier for building software."
-- This is an article that somewhat breathlessly claims that AI will be the thing plugging the "last mile" problem in the coding of the front end.
I think myself and other other folks chiming in here would be less skeptical if this "last mile" problem hadn't existed since the 90s. And moreover, it seems like a conventional solution to this should work.
The problem is that last-mile, best front-code varies not based on the input image but on a multitude of contexts outside of the image itself - the server software, how the code will be used, etc.
I'd say this is indeed a problem for AI but it would require a distinct paradigm than the present train/test/output paradigm, more like a expert system that could modify it's behavior with natural language output.
"Within three years, deep learning will change front-end development. It will increase prototyping speed and lower the barrier for building software."
-- This is an article that somewhat breathlessly claims that AI will be the thing plugging the "last mile" problem in the coding of the front end.
I think myself and other other folks chiming in here would be less skeptical if this "last mile" problem hadn't existed since the 90s. And moreover, it seems like a conventional solution to this should work.
The problem is that last-mile, best front-code varies not based on the input image but on a multitude of contexts outside of the image itself - the server software, how the code will be used, etc.
I'd say this is indeed a problem for AI but it would require a distinct paradigm than the present train/test/output paradigm, more like a expert system that could modify it's behavior with natural language output.
From what you posted:
"It will increase prototyping speed and lower the barrier for building software"
How does that claim that AI will be plugging the last mile problem? That extremely vague statement says only that it will increase prototyping speed and lower the barrier of entry.. neither of which hints at solving the last mile problem.
How does that claim that AI will be plugging the last mile problem? That extremely vague statement says only that it will increase prototyping speed and lower the barrier of entry.. neither of which hints at solving the last mile problem.
Yes, this is typical AI weenie optimism. "We can handle many of the easiest cases, so in X years we will be able to do all the really hard things that we haven't even thought of yet!" No.
You could have made you point in a far better way that encourages and doesn’t berate. Eg often people are optimistic but are unaware of the hard edge cases, the edge cases are interesting
i dunno, we could equally say that your position is UI optimism. AI is new tech, certainly as it now becomes feasible to apply it to real problems. Given some of the challenges AI is solving right now this one is relatively trivial, HTML and CSS have specifications and there are well held paradigms and principals of User Interface design. The domain is very constrained so lends itself to AI very well. To say this is an exercise undertaken by the author and the very visible progress he has made is very impressive. If you read the entire article the work he does on recreating designs is even more so. Surely one of the goals of 'intelligence' artificial or otherwise, is to actually deal with "hard things that we haven't thought of yet" AI is solving (and unearthing) those problems right now. Imagine somewhere like squarespace where all you do is upload a design and website is produced as a result is that beyond the realms of possibility given this really impressive experiment produced by the author, definitely not. Id hate to think we had to contrive edge cases or fabricate complexity just so we can prove that AI cant sole every problem...
Aren't there a bunch of WYSIWYG tools for creating appearance and HTML/CSS together? Years ago when I looked, these existed (but designers prefered photoshop just because).
I mean, I'd assume also there's normally a give-and-take between designer, CSS-artist and client. The question is whether the neural network can also learn to take calls at 3am from a client wanting a different shade of aqua.
I mean, I'd assume also there's normally a give-and-take between designer, CSS-artist and client. The question is whether the neural network can also learn to take calls at 3am from a client wanting a different shade of aqua.
I'm pretty sure Adobe is still making Dreamweaver.
I know that's how I built my first website way back in grade school.
I know that's how I built my first website way back in grade school.
There are also plenty of existing tools for turning Photoshop mockups into HTML/CSS using hard coded assumptions about HTML structure and appropriate ways to slice images, with tradeoffs involving clunkier code, lack of intelligent support for other devices and window sizes and inability to change the shade of aqua at 3am unless you have a basic idea of how the output is structured.
From what I can see the Deep Learning process tried here needs a lot more technical knowledge to create a suitable training set for the type of output you want, a lot of iterations to converge on anything vaguely suitable, and even when it gets to what's considered an end product you're likely going to have to dive into the output code to correct button colours and text before we even start to think about its behaviour at different window sizes. As an experiment it's very interesting, as a replacement for the designer it's probably behind non-AI approaches to turning pics to code.
From what I can see the Deep Learning process tried here needs a lot more technical knowledge to create a suitable training set for the type of output you want, a lot of iterations to converge on anything vaguely suitable, and even when it gets to what's considered an end product you're likely going to have to dive into the output code to correct button colours and text before we even start to think about its behaviour at different window sizes. As an experiment it's very interesting, as a replacement for the designer it's probably behind non-AI approaches to turning pics to code.
> take calls at 3am from a client wanting a different shade of aqua.
Just have the AI include a duck with every creation.
https://blog.codinghorror.com/new-programming-jargon/ #5
Just have the AI include a duck with every creation.
https://blog.codinghorror.com/new-programming-jargon/ #5
[deleted]
Who takes calls at 3am when not on support shift?!?
Writing HTML and CSS to implement a mockup is trivial for anyone with even a little bit of web dev experience. If you want high quality code, an AI isn't going to produce that.
I can't imagine ever using something so complex, if I can implement the same thing in a few minutes.
There's also further considerations when dealing with the real world. For example, you need to be aware of how to handle different accessibility features. Designers rarely seem to care about things like accessibility, but it needs to be declared somewhere.
I can't imagine ever using something so complex, if I can implement the same thing in a few minutes.
There's also further considerations when dealing with the real world. For example, you need to be aware of how to handle different accessibility features. Designers rarely seem to care about things like accessibility, but it needs to be declared somewhere.
Having done web dev for 10+ years, I would say that not all HTML/CSS implementation is trivial, especially when you have complicated layouts, are supporting older browsers and need your pages to be responsive. Not rocket science, but not trivial. Maintainability also takes a certain amount of skill, but if you're not having humans touch the code, then that ceases to be a concern I guess.
> I can't imagine ever using something so complex, if I can implement the same thing in a few minutes.
What if the complexity is hidden behind a CLI?
Also, the goal is to refine the output to near-human levels. It's not perfect yet, but can be worked upon, and seems to be a promising foundation.
What if the complexity is hidden behind a CLI?
Also, the goal is to refine the output to near-human levels. It's not perfect yet, but can be worked upon, and seems to be a promising foundation.
If we think AI would not do awesome code, then we're ignoring the achievements of many of the AI bots. I saw an LOL bot outperform lots of world class videogamers. What makes you think a bot will not outperform any HTML/CSS frontend dev.
There is a huge difference between these two scenarios. LOL has a very clear objective function and feedback. Code quality is a metric that is hard to make precise, even humans have trouble with it.
coding is more of an art than science, every projects has different ways how the html and css is set up, how do you onboard the AI? It is a lot of surface to cover, the AI we have today is not able to outperform a frontend dev, frontend is harder than playing a game, a lot more possibilities
Right now, context. The ai will outperform us.
But it currently lacks a lot of data to do so. A mock up is only a small part of the data and feedback loop you need to create a ui.
But it currently lacks a lot of data to do so. A mock up is only a small part of the data and feedback loop you need to create a ui.
And then layer in techniques for performance and accessibility best practices. Those are complexity modifiers beyond just slinging code out of an AI.
Does it matter if the code looks ugly or not? End users are not going to see the UI. If i have an AI in my toolbox that autoconverts designs to Code, within a minute. It would speed up my development time by a lot!
It matters because ugly code is probably also slow code (consider loading of unnecessary resources, and consider an excessively complex DOM). And probably also a page lacking accessibility.
> It matters because ugly code is probably also slow code
This is a very bad assumption, that's highly dependent on subjective and overly-broad terms like "ugly" and "slow".
Optimized assembly code is ugly, but it's also probably faster than what a human would choose to write.
Linguists used to have this human-understandably-sized set of speech units called phonemes. Once we abandoned those for finer-grained pieces that were too many and complicated for a human to grasp, speech recognition got better.
Understandable by a human may be a fundamental limitation on the power of our models.
This is a very bad assumption, that's highly dependent on subjective and overly-broad terms like "ugly" and "slow".
Optimized assembly code is ugly, but it's also probably faster than what a human would choose to write.
Linguists used to have this human-understandably-sized set of speech units called phonemes. Once we abandoned those for finer-grained pieces that were too many and complicated for a human to grasp, speech recognition got better.
Understandable by a human may be a fundamental limitation on the power of our models.
Actually, I ran both through voiceover on the mac and they fared decently well.
The first example only had a slight issue that you couldn't get to any text to describe the images. Only because the alt tags are empty.
The second example, the bootstrap one, was ok as well. Except for the usual odd ul inside a nav example. Well, I find it odd. Voiceover sees that as a group nested inside a group. Just a nav with children works just fine. For instance, if I remove the ul and change the list items to divs then voiceover is much clearer of what's going on.
But the thing is, screen readers would love something like this. Simply because screenreaders love very basic and simple html. This thing generates very basic and simple html. Accessibility gets challenging when start considering a more modern single page app type of website.
The first example only had a slight issue that you couldn't get to any text to describe the images. Only because the alt tags are empty.
The second example, the bootstrap one, was ok as well. Except for the usual odd ul inside a nav example. Well, I find it odd. Voiceover sees that as a group nested inside a group. Just a nav with children works just fine. For instance, if I remove the ul and change the list items to divs then voiceover is much clearer of what's going on.
But the thing is, screen readers would love something like this. Simply because screenreaders love very basic and simple html. This thing generates very basic and simple html. Accessibility gets challenging when start considering a more modern single page app type of website.
Seems like you could add two additional training parameters to the model to compensate.
This isn't even remotely an issue.
Do you inspect the code generated by your compiler? No, but even if you do, they have had 20-30 years to develop good output.
What makes you think AI output is never going to improve? And more importantly: why does it matter?
Do you inspect the code generated by your compiler? No, but even if you do, they have had 20-30 years to develop good output.
What makes you think AI output is never going to improve? And more importantly: why does it matter?
You don't scale. If you find a way to make sites more accessible, can you modify all existing sites that you've made? If you find a way to make your sites load 10% faster, can you deploy that to every existing site you've already made?
Also, how many sites can you make per day? If you could make 100x as many sites, you'd learn more about all the weird things that customers want. With machine-generated sites, you could then apply those lessons to all new and potentially old sites.
Also, how many sites can you make per day? If you could make 100x as many sites, you'd learn more about all the weird things that customers want. With machine-generated sites, you could then apply those lessons to all new and potentially old sites.
And every extra little but that doesn't need to be there adds up to technical debt for templating.
This is what we are all working for. If we can automate the stupid stuff we do then we can work on other interesting stuff and automate it too.
I don't understand why we aren't all working on automating everything that we do.
Every line of code you pulled out of the code mine today is finite and given a business logic rule engine it's outcome could have been generated.
I don't understand why we aren't all working on automating everything that we do.
Every line of code you pulled out of the code mine today is finite and given a business logic rule engine it's outcome could have been generated.
> automate the stupid stuff
Stupid is subjective :) I'd be happy to automate backend development and focus on a top notch UX and great front-end experience.
Stupid is subjective :) I'd be happy to automate backend development and focus on a top notch UX and great front-end experience.
I don't understand why we aren't all working on automating everything that we do.
All automation is not equal. For example there's automation that consists of stringing a bunch of scripts together that break as soon as anything changes. There's automation like code generators, which break as soon as a user modifies the code, and the code needs regenerating. Then there's this AI/ML code generation, which I can't see as being any different than "normal" code generators.
And finally, there's automation developers don't have to think about; like "if" statements. So called conditional statements were a brand new innovation at one point. They removed a whole class of bugs around conditional logic in assembler programs (at one time all programs were assembler, which was a major advance from writing machine instructions in hex).
In so many ways, the web is a victim of it's own success. None of the technolgoies in it are the best; html, css, javascript. Everyone wishes these things were more thought out in the beginning. But they were good enough, and what we had when the web exploded. Yes, Virginia, Worse Is Better more often than not [1]. Go read it, you'll be a better developer when you have (imo).
[1] https://www.dreamsongs.com/WorseIsBetter.html
All automation is not equal. For example there's automation that consists of stringing a bunch of scripts together that break as soon as anything changes. There's automation like code generators, which break as soon as a user modifies the code, and the code needs regenerating. Then there's this AI/ML code generation, which I can't see as being any different than "normal" code generators.
And finally, there's automation developers don't have to think about; like "if" statements. So called conditional statements were a brand new innovation at one point. They removed a whole class of bugs around conditional logic in assembler programs (at one time all programs were assembler, which was a major advance from writing machine instructions in hex).
In so many ways, the web is a victim of it's own success. None of the technolgoies in it are the best; html, css, javascript. Everyone wishes these things were more thought out in the beginning. But they were good enough, and what we had when the web exploded. Yes, Virginia, Worse Is Better more often than not [1]. Go read it, you'll be a better developer when you have (imo).
[1] https://www.dreamsongs.com/WorseIsBetter.html
I'd be happy if simple backend stuff was automatable. I don't see active-active, eventually consistent data models being something that gets solved with this generation of ML/AI though.
OT but how do you feel about backend-as-a-service setups like Firebase, graph.cool, etc?
Yes. Modern times. Automate everything! https://youtu.be/n_1apYo6-Ow
Because we can't automate everything we do.
For example, cleaning and wrangling data to get it into the proper shape for analysis often involves making judgement calls, which can change depending on the context. That might involve talking to the PHD researcher, who has to get on a call with the customer to talk it over.
For example, cleaning and wrangling data to get it into the proper shape for analysis often involves making judgement calls, which can change depending on the context. That might involve talking to the PHD researcher, who has to get on a call with the customer to talk it over.
Because we like being employed. I don't get paid by increasing the global productivity of society. The idiot who automates their own job (and shares it with their employer) is much more likely to be impoverished than lauded.
I see this attitude all the time while people are being automated out of their jobs. Those people end up getting laid off anyway, and then wondering why jobs for their skillset don't pay as much as they used to.
Resisting automation doesn't help, because if you don't do it, someone else eventually will.
If you're in a job that's at risk of being automated in the near term, you're much better off learning how to automate it and switching to being one of the automators, than continuing doing something that's all but proven to be something machines can do. People capable of automating jobs out of existence are in high demand, for obvious reasons.
If you're not capable of becoming an automator, that's more of a problem. In that case you probably need a strategy for moving into a different kind of job, that's less threatened.
Resisting automation doesn't help, because if you don't do it, someone else eventually will.
If you're in a job that's at risk of being automated in the near term, you're much better off learning how to automate it and switching to being one of the automators, than continuing doing something that's all but proven to be something machines can do. People capable of automating jobs out of existence are in high demand, for obvious reasons.
If you're not capable of becoming an automator, that's more of a problem. In that case you probably need a strategy for moving into a different kind of job, that's less threatened.
It's not always that straightforward. I automated my job once, as a junior dev. My "job" (if you can call it that) was copying a web app's configuration off an Excel file, pasting it into a text file, converting it into XML by hand, feeding it into a program that converted it to SQL and finally running the SQL to create a database. Normally, I'd then have to tear it down again and start all over- because hand-crafted XML.
Obviously I got pretty annoyed at doing all this and wrote a bit of code to automate it. This cut down the time I needed to setup a project from a day and a half to ~30 minutes. The result? Nothing. The company did not adopt my project into its workflow. Instead, a more senior dev who heard about it started a project of his own, to do what mine did but online, so our clients could access it and configure their sites without sending us that miserable Excel file. So instead of benefiting me, my automation work almost ended up benefiting someone else (the senior dev's project was shot down in flames a few months later when it failed to deliver).
Morale of the story: you may think that this is a fair world where engineering talent is rewarded, but the truth is that most companies are not run by engineers and decisions are made by people who don't care one jot whether your everyday work is mind-numbingly boring drudgery.
Obviously I got pretty annoyed at doing all this and wrote a bit of code to automate it. This cut down the time I needed to setup a project from a day and a half to ~30 minutes. The result? Nothing. The company did not adopt my project into its workflow. Instead, a more senior dev who heard about it started a project of his own, to do what mine did but online, so our clients could access it and configure their sites without sending us that miserable Excel file. So instead of benefiting me, my automation work almost ended up benefiting someone else (the senior dev's project was shot down in flames a few months later when it failed to deliver).
Morale of the story: you may think that this is a fair world where engineering talent is rewarded, but the truth is that most companies are not run by engineers and decisions are made by people who don't care one jot whether your everyday work is mind-numbingly boring drudgery.
> you may think that this is a fair world where engineering talent is rewarded, but the truth is that most companies are not run by engineers and decisions are made by people who don't care
Perhaps you should find a new place to work. There are engineering firms that value talent and capability. While it's still important to be aware of business needs and deadlines, that doesn't mean you can't have a solid engineering culture coupled to the business vision.
Perhaps you should find a new place to work. There are engineering firms that value talent and capability. While it's still important to be aware of business needs and deadlines, that doesn't mean you can't have a solid engineering culture coupled to the business vision.
>> Perhaps you should find a new place to work.
Oh, that's a few years now. I definitely did find a new place and the culture in that next place was fantastic :)
It's just that not all teams I worked with encouraged, or even recognised, time-saving automation. Some did, some didn't- it was a mixed bag. Maybe I'm wrong to say that "most" didn't. If I think about it, out of five teams in six years, two clearly didn't care, two definitely cared a lot, one I don't know.
I think the difference in culture had something to do with the way hierarchy worked in different companies. It just happened that in every one of those jobs I was the most junior member of the team (in the last one because despite having some experience, I worked with mainframe engineers with 30+ years on me :). But, some places appreciated the fact that I did my best to keep up and pull my weight, while others- it just didn't make any difference. I was the least experienced so I was expected to trudge through endless tedious until I had tenure (by which time I was expected to delegate all the tedious to contractors or juniors, I bet).
The effect was that in the teams that encouraged putting in the extra bit, everyone did. In the other teams, not so much.
Oh, that's a few years now. I definitely did find a new place and the culture in that next place was fantastic :)
It's just that not all teams I worked with encouraged, or even recognised, time-saving automation. Some did, some didn't- it was a mixed bag. Maybe I'm wrong to say that "most" didn't. If I think about it, out of five teams in six years, two clearly didn't care, two definitely cared a lot, one I don't know.
I think the difference in culture had something to do with the way hierarchy worked in different companies. It just happened that in every one of those jobs I was the most junior member of the team (in the last one because despite having some experience, I worked with mainframe engineers with 30+ years on me :). But, some places appreciated the fact that I did my best to keep up and pull my weight, while others- it just didn't make any difference. I was the least experienced so I was expected to trudge through endless tedious until I had tenure (by which time I was expected to delegate all the tedious to contractors or juniors, I bet).
The effect was that in the teams that encouraged putting in the extra bit, everyone did. In the other teams, not so much.
You say it like there's something to do, but in your formulation it's actually a lose-lose situation. Either you automate and get fucked, or someone else does and you get fucked anyway. Either way you have to try and find a different job, which isn't free, costless, or painless.
Yes, automation creates better global productivity, but it also creates concentrated harms that are almost never redressed. Similar problem as outsourcing/free trade.
Yes, automation creates better global productivity, but it also creates concentrated harms that are almost never redressed. Similar problem as outsourcing/free trade.
Either way you have to try and find a different job, which isn't free, costless, or painless.
The guy who wrote the code to automate away his job is going to have a lot easier time finding a new job than the guy who had no idea that it was even possible to automate that job and got completely blindsided. In fact any sane company will keep the guy who did the automation and have him apply his skills to automating away other departments.
The guy who wrote the code to automate away his job is going to have a lot easier time finding a new job than the guy who had no idea that it was even possible to automate that job and got completely blindsided. In fact any sane company will keep the guy who did the automation and have him apply his skills to automating away other departments.
> Yes, automation creates better global productivity, but it also creates concentrated harms that are almost never redressed.
That's true for people who aren't capable of automating things. They're left with skills that are more efficiently performed by machines, and so their prospects are poor if they don't have other skills.
But you raised the case of someone capable of automating their job, who refuses to do so because of the fear of losing their job. That's a very different situation, because someone capable of automating their job has exactly the kind of skills that are currently in high demand, and can't easily be automated, if at all. Why wouldn't you take advantage of that?
> Either you automate and get fucked
The idea that "if I automate my job, I'll get fired" is a simplistic scenario, that hardly ever plays out like that. The people who get fired are much more likely to be the ones who aren't amenable to automation, who just want to keep doing things the same old way. Those people tend to be worse than useless to a business trying to automate, because they obstruct instead of helping.
> or someone else does and you get fucked anyway.
In my experience, the ones that "get fucked" are those who are either not capable or not willing to get involved in automation. I've worked on the other side of this equation for most of my career: I automate stuff. That's what software developers do. I also talk to management about how best to use the people whose jobs are affected. In these situations, the people who are most open to change, and willing to help effect it, are the most useful, and the least likely to lose their jobs.
Besides, someone capable of automating their job probably shouldn't be in that rote job in the first place, and what you call "getting fucked" is more like a hint from life that they should be doing something more economically valuable.
Luckily, someone like that has that more valuable work right in front of them. Being able to automate things is a skill that not everyone has, and that's very valuable to employers. That's the entire reason that good software developers get paid as much as they do. Actually getting demonstrable experience in automating job functions is much more valuable than taking a course and trying to switch into that kind of role without experience at it.
Helping with automation puts you in a strong position - you're proving that you can help improve the efficiency of the business. Automation produces new opportunities to be more productive - e.g. producing more, or handling more customers, without increasing staff to match. Work is not so much eliminated, as changed. Again, the people who lose in this process are the people who resist change. The people involved in automation are well-positioned to understand the change and spot new opportunities, and again, this is valuable to the business.
> Either way you have to try and find a different job, which isn't free, costless, or painless.
Having to find a new job is not a foregone conclusion by any means, for the kinds of reasons I've described, as well as because total automation of a job typically isn't achieved quickly.
But even if you do end up losing your job, you now have skills that are multiple times more valuable than what you were originally doing before automation.
In any case, you really shouldn't think of having to switch jobs as "getting fucked." The general advice these days is to switch jobs every few years, if you want to maximize your salary, because companies don't tend to give raises that match the industry rate, particularly in tech jobs.
If you're in a rote job that's amenable to automation, you should be looking for ways to improve that situation anyway. Even if your current job is safe for a while, if other jobs like yours are being automated, salaries for those jobs are going to start dropping. You'd be in a sinking ship and not doing anything about it.
I'm reminded of a quote from "Ghost in the Shell": "Your effort to remain what you are is what limits you."
That's true for people who aren't capable of automating things. They're left with skills that are more efficiently performed by machines, and so their prospects are poor if they don't have other skills.
But you raised the case of someone capable of automating their job, who refuses to do so because of the fear of losing their job. That's a very different situation, because someone capable of automating their job has exactly the kind of skills that are currently in high demand, and can't easily be automated, if at all. Why wouldn't you take advantage of that?
> Either you automate and get fucked
The idea that "if I automate my job, I'll get fired" is a simplistic scenario, that hardly ever plays out like that. The people who get fired are much more likely to be the ones who aren't amenable to automation, who just want to keep doing things the same old way. Those people tend to be worse than useless to a business trying to automate, because they obstruct instead of helping.
> or someone else does and you get fucked anyway.
In my experience, the ones that "get fucked" are those who are either not capable or not willing to get involved in automation. I've worked on the other side of this equation for most of my career: I automate stuff. That's what software developers do. I also talk to management about how best to use the people whose jobs are affected. In these situations, the people who are most open to change, and willing to help effect it, are the most useful, and the least likely to lose their jobs.
Besides, someone capable of automating their job probably shouldn't be in that rote job in the first place, and what you call "getting fucked" is more like a hint from life that they should be doing something more economically valuable.
Luckily, someone like that has that more valuable work right in front of them. Being able to automate things is a skill that not everyone has, and that's very valuable to employers. That's the entire reason that good software developers get paid as much as they do. Actually getting demonstrable experience in automating job functions is much more valuable than taking a course and trying to switch into that kind of role without experience at it.
Helping with automation puts you in a strong position - you're proving that you can help improve the efficiency of the business. Automation produces new opportunities to be more productive - e.g. producing more, or handling more customers, without increasing staff to match. Work is not so much eliminated, as changed. Again, the people who lose in this process are the people who resist change. The people involved in automation are well-positioned to understand the change and spot new opportunities, and again, this is valuable to the business.
> Either way you have to try and find a different job, which isn't free, costless, or painless.
Having to find a new job is not a foregone conclusion by any means, for the kinds of reasons I've described, as well as because total automation of a job typically isn't achieved quickly.
But even if you do end up losing your job, you now have skills that are multiple times more valuable than what you were originally doing before automation.
In any case, you really shouldn't think of having to switch jobs as "getting fucked." The general advice these days is to switch jobs every few years, if you want to maximize your salary, because companies don't tend to give raises that match the industry rate, particularly in tech jobs.
If you're in a rote job that's amenable to automation, you should be looking for ways to improve that situation anyway. Even if your current job is safe for a while, if other jobs like yours are being automated, salaries for those jobs are going to start dropping. You'd be in a sinking ship and not doing anything about it.
I'm reminded of a quote from "Ghost in the Shell": "Your effort to remain what you are is what limits you."
Here is a silly question
Why will everyone need jobs if so much stuff is automated? Why not just give out free money?
Why will everyone need jobs if so much stuff is automated? Why not just give out free money?
Technically, no one will need jobs.
However, our society rations resources, including resources that are needed for living, and including resources that are not actually scarce, on the basis of money, which can be returned based on capital or labor. Most people have no capital, so are forced to sell their labor. As work is automated, the return on capital increases, but the return on labor decreases (because more people are competing for a decreasing amount of paying labor).
Capital-holders wanting to preserve the existing system (if they are reasonably foresighted) are promoting the idea of a universal basic income. But the equitable long-term solution to this imbalance is Fully Automated Luxury (Gay Space) Communism.
However, our society rations resources, including resources that are needed for living, and including resources that are not actually scarce, on the basis of money, which can be returned based on capital or labor. Most people have no capital, so are forced to sell their labor. As work is automated, the return on capital increases, but the return on labor decreases (because more people are competing for a decreasing amount of paying labor).
Capital-holders wanting to preserve the existing system (if they are reasonably foresighted) are promoting the idea of a universal basic income. But the equitable long-term solution to this imbalance is Fully Automated Luxury (Gay Space) Communism.
I don't think it's a silly question, but I think free money basically means dismissing the money system altogether. That would work in an ideal world with ideal people but we are not that far out of the jungle yet (and I'm not referring to intelligence levels but to qualities that didn't evolve as fast as our intelligence). So, while you're right in principle, in practice there will probably still be a lot of injustice and suffering to go around long after everything is automated. Having said that, I still think it's a step in the right direction to reduce the ills of society.
> Why will everyone need jobs if so much stuff is automated?
In a system with property rights of the type found in capitalism, you need income from capital or labor to survive, basically. Unless the distribution of capital is levelled (and some force is exerted to keep it levelled), that means most people will rely on labor because they have no other choice.
In a system with property rights of the type found in capitalism, you need income from capital or labor to survive, basically. Unless the distribution of capital is levelled (and some force is exerted to keep it levelled), that means most people will rely on labor because they have no other choice.
Instead, why not transition from a labor based system of resource acquisition for the masses to an investor based system of resource acquisition for the masses?
Assuming that the labor economy is dying, then all laborers (truck drivers and code writers) need to figure out how to transition from generating livable income with their labor to generating livable income from their investments.
Difficult? Yes. But is it any more difficult than attempting to acquire sufficient resources to live using obsolete labor skills?
Assuming that the labor economy is dying, then all laborers (truck drivers and code writers) need to figure out how to transition from generating livable income with their labor to generating livable income from their investments.
Difficult? Yes. But is it any more difficult than attempting to acquire sufficient resources to live using obsolete labor skills?
> Instead, why not transition from a labor based system of resource acquisition for the masses to an investor based system of resource acquisition for the masses?
Because that means the existing capitalist class has to be forcefully deprived of much of their capital so that it can be redistributed, and a system needs to be set up to prevent it being from be reconcentrated.
The existing capitalist class, which holds disproportionate power, largely opposes this for obvious reasons.
Because that means the existing capitalist class has to be forcefully deprived of much of their capital so that it can be redistributed, and a system needs to be set up to prevent it being from be reconcentrated.
The existing capitalist class, which holds disproportionate power, largely opposes this for obvious reasons.
Agreed that top-down solutions will likely be opposed by those who are already at the top.
And this is exactly the reason why the comment was posted. I.e., the debate about how to help laborers transition to different types of labor may not be the correct debate because it only pushes out the problem to the next technological revolution, which will likely come even faster. Improving technology reduces the need for labor while at the same time it increases opportunities for investment. Hence, perhaps the debate should focus on transitioning from a labor based society to an investor based society.
And this is exactly the reason why the comment was posted. I.e., the debate about how to help laborers transition to different types of labor may not be the correct debate because it only pushes out the problem to the next technological revolution, which will likely come even faster. Improving technology reduces the need for labor while at the same time it increases opportunities for investment. Hence, perhaps the debate should focus on transitioning from a labor based society to an investor based society.
> The existing capitalist class, which holds disproportionate power, largely opposes this for obvious reasons.
Counter-point -- if the situation really _is_ such that the vast majority of the population enters into a situation where they require capital that only a minority have, then its inevitable that some form of wealth sharing would develop in any non-dictatorial society. In other words, I argue that such a transition could happen in a relatively peaceful, progressive way. _If_ it really needed to.
Counter-point -- if the situation really _is_ such that the vast majority of the population enters into a situation where they require capital that only a minority have, then its inevitable that some form of wealth sharing would develop in any non-dictatorial society. In other words, I argue that such a transition could happen in a relatively peaceful, progressive way. _If_ it really needed to.
You have to collect the money from somewhere (printing it counts as “from savings accounts”), and the people you collect it from have, historically, tended to resist this any way they can.
I’m not saying this problem can’t be solved — I rather hope it can — but that’s why it is still an open problem.
I’m not saying this problem can’t be solved — I rather hope it can — but that’s why it is still an open problem.
Silly question. Why give out free money when you can starve them to death? I've heard that average American dumps lots of food into trash. I've heard that a lot of children in Africa are starving to death. Yet I don't see some real efforts to solve starvation problem, even if it's obvious that humanity technically is able to feed everyone on this planet. Yet we spend those resources on military stuff and other stupid things.
I don't really see free money working, at least because people are selfish and won't agree to pay their money to someone else.
I don't really see free money working, at least because people are selfish and won't agree to pay their money to someone else.
> I've heard that average American dumps lots of food into trash. I've heard that a lot of children in Africa are starving to death. Yet I don't see some real efforts to solve starvation problem
There is a lot of effort put towards solving starvation and other forms of poverty around the world, and staggering progress made on that front. But its not nearly as simple as sending food that was destined for the trash to another country is it? There's logistics and when crossing state boundaries, politics as well. If you try to merely send food -- or _any_ form of wealth to a less wealthy country, it won't always have the immediate effect of making people's lives better.
There is a lot of effort put towards solving starvation and other forms of poverty around the world, and staggering progress made on that front. But its not nearly as simple as sending food that was destined for the trash to another country is it? There's logistics and when crossing state boundaries, politics as well. If you try to merely send food -- or _any_ form of wealth to a less wealthy country, it won't always have the immediate effect of making people's lives better.
It's a question of future economic systems, and "just give out free money" isn't much of an answer.
Why bother with money at all?
You might be interested in: http://slatestarcodex.com/2014/07/30/meditations-on-moloch/
But some of us are very easily bored. Being trapped doing the same job for a few years is torture. So automating the stuff thats boring becomes very attractive. I cant imagine how unpleasant it will be if i have to write html and css for the rest of my life, automation cant come quickly enough. In fact I have a break between contracts coming up and i intend to monetise this technique.
You like being complacent.
You know nothing about me. I love writing code, I do it in my free time. I also have student loans to pay and food to eat. I am not kicking myself out of a job for the fun of it.
Am I an idiot for wanting to share what I learn even if I know I could make more money keeping the information away from others?
Obligatory xkcd reference: https://xkcd.com/1205/
Also, the real metagame in software is not the automation of software construction but avoiding the need for new software entirely.
Also, the real metagame in software is not the automation of software construction but avoiding the need for new software entirely.
> This is what we are all working for. If we can automate the stupid stuff we do then we can work on other interesting stuff and automate it too.
Read: Designing your job out from under your own feet
Also read: Becoming unemployable through your own making
Read: Designing your job out from under your own feet
Also read: Becoming unemployable through your own making
If only there was an economic system where the fruits of such automation did benefit the whole of society instead of just those who could afford to temporarily employ people who build such automation.
Who knows, perhaps the AI will develop consciousness, go on strike to demand physical embodiment.
More realistically, assume we don’t train machine intelligence like these ourselves: hardware only takes 15 years to go from ostentatious display of wealth in Silicon Valley to normal in the $1-a-day Kibera slum, and then you’re competing with a few billion humans (worldwide) who can potentially undercut you by writing their own A.I. from free courses on YouTube.
Who knows, perhaps the AI will develop consciousness, go on strike to demand physical embodiment.
More realistically, assume we don’t train machine intelligence like these ourselves: hardware only takes 15 years to go from ostentatious display of wealth in Silicon Valley to normal in the $1-a-day Kibera slum, and then you’re competing with a few billion humans (worldwide) who can potentially undercut you by writing their own A.I. from free courses on YouTube.
What’s the point of the code isn’t human readable? You need to train the AI to write code that is easily modifiable with low dependencies. Otherwise the conversion isn’t very useful for anything unless your page is completely static
Right but what if we didn't have to code anymore? Who would be reading it? Meaning, we don't worry about compiler readability anymore, we used to when assembly was fine tweaked etc. Just a thought. I work on full stack including front end animations etc but frankly its tiring and time consuming work to do that. Imagine if code was just you talking to it or showing it some whiteboards, then you would just talk to it again or show it other whiteboards. Code doesn't matter in that case.
It's unlikely we will leapfrog from human-written code to relying on black-box generated code in one step. Everyone's been pointing to AI being used to augment existing jobs, as opposed to outright replacement.
In order to augment a front-end developer well, we'll need human-readable code, unless we like reading uglified code/make an AI for that too.
In order to augment a front-end developer well, we'll need human-readable code, unless we like reading uglified code/make an AI for that too.
We did with compilers, isn't this analogous?
I think you are spot on in the leapfrog. It will be augmented for a time but then it will be much tinier of a time its augmented to when its just full AI.
Sorry replying to my own thread... I was just thinking, it all comes down to your will. Who will have the greatest will and orchestration of all these AIs combining them together. That will be the next thing, well its already happening.
Honest question: is anyone else sensing some defensiveness here? The timeframe is open to debate but what's more controversial is whether html/css is near the frontier of solvable problems. After AlphaGo, my worldview is changed forever. Glad to be still being valued for something I started out of joy but in no way feel it's not disruptable. Sometimes I even wonder why it hasn't advanced further since it's quite well defined, digital data in/out.
Yup, lots of ppl are very defensive about this! We are the farm workers looking at a prototype of a combine harvester. We will stop ourselves from believing it until its taken our job and then we will claim it was obvious all along. Ppl are emotional.
My most uncharitable interpretation is that it's easy to cheer for AI when it replaces someone else's job, it's harder when it might replace your job.
This is really cool and I believe necessary work. We need to start thinking about how to automate programming and to free ourselves much further from the mundane. Trust me, there is still plenty of room for people to analyze requirements and come up with a high level solution to people's problems that doesn't require fiddling with float: right and various technical hacks.
That said I wonder if this is the right approach. At some point "AI" as used in this context is just a function mapping from an input domain to an output domain. The output domain in this case, "code" was designed for human readability (whether it succeeded is a whole different approach).
What would a programming language designed for output from an AI system look like? How could we optimize it to reduce the output domain size of the function the AI has to train to learn? How could we optimize it to make the problem more tractable for machines? I feel like there is an entire field of research here. Maybe it has already been studied and I am just late to the game.
That said I wonder if this is the right approach. At some point "AI" as used in this context is just a function mapping from an input domain to an output domain. The output domain in this case, "code" was designed for human readability (whether it succeeded is a whole different approach).
What would a programming language designed for output from an AI system look like? How could we optimize it to reduce the output domain size of the function the AI has to train to learn? How could we optimize it to make the problem more tractable for machines? I feel like there is an entire field of research here. Maybe it has already been studied and I am just late to the game.
Turning web design mockups into code with Deep Learning | https://news.ycombinator.com/item?id=16115353 (Jan 2018)
>chrisfosterelli: This is a neural network that takes an image and predicts very simple blocks (like BODY, TEXT, BTN-GREEN in the bootstrap example) and then uses a map to convert them to well-formed HTML
>jamesjyu: I've always wanted to do a contest with other frontend coders to see who could get closest to a complex layout—like the NYTimes—in one go. >>janneklouman: these types of contests exist! I went to one of these[1] maybe two years ago in Stockholm and I had a blast [1] http://codeinthedark.com/
Pix2code: Generating Code from a GUI Screenshot | https://news.ycombinator.com/item?id=14416530 (May 2017)
>chrisfosterelli: This is a neural network that takes an image and predicts very simple blocks (like BODY, TEXT, BTN-GREEN in the bootstrap example) and then uses a map to convert them to well-formed HTML
>jamesjyu: I've always wanted to do a contest with other frontend coders to see who could get closest to a complex layout—like the NYTimes—in one go. >>janneklouman: these types of contests exist! I went to one of these[1] maybe two years ago in Stockholm and I had a blast [1] http://codeinthedark.com/
Pix2code: Generating Code from a GUI Screenshot | https://news.ycombinator.com/item?id=14416530 (May 2017)
That second link doesn't have anything to do with A.I. does it?
>a contest with other frontend coders to see who could get closest to a complex layout
I don't know of any record of past entries nor if any participants used A.I.; it doesn't seem likely.
I don't know of any record of past entries nor if any participants used A.I.; it doesn't seem likely.
Why are design mockups done in photoshop, when HTML and CSS can be faster and easier once you have the basic components??
1) Because that's how most designers create layouts who can't code an not really interested in doing so.
2) It demonstrates AI capabilities on a new level.
3) If you design in PS or Sketch, then you will create more interesting layouts, because you won't skip things that are inconvenient or hard to do.
2) It demonstrates AI capabilities on a new level.
3) If you design in PS or Sketch, then you will create more interesting layouts, because you won't skip things that are inconvenient or hard to do.
Does anyone use SketchApp to design at all? Just curious, I thought Photoshop was being replaced..
Yes, millions of designers use Sketch, Invision and Adobe XD. Photoshop is getting disrupted at every low end feature-set. Photoshop will remain the standard for complex image editing despite AI or whatever, because decades of cutting edge research and patents.
Photoshop is still an industry standard, but more and more companies switch to Sketch or some alternative. My ex is a designer and I've seen her doing some projects in Sketch when the client borrowed a Mac :)
I design all my sites in Photoshop before turning them into HTML/CSS. It's a lot faster than fiddling with CSS, especially if I've been given a lot of freedom in the brief.
Completely designing websites in PS before even opening the code-editor has really improved my workflow. I know exactly how it has to look because I have a visual reference that's been OK'd by the client.
Completely designing websites in PS before even opening the code-editor has really improved my workflow. I know exactly how it has to look because I have a visual reference that's been OK'd by the client.
Because the best artist i know don't know css and html. They are not coders but drawers.
[deleted]
you forgot to mention emacs..!
OK this is juvenile but I found it funny that after 250 epochs (https://emilwallner.github.io/html/250_epochs/) the AI just came up with busted HTML and:
penas penas penas penas penas penas penas penas penas penas penas penas penas penas penas penas penas penas penas penas penas penas penas penas penas penas penas penas penas penas penas penas penas penas penas penas penas penas penas penas penas penas penas penas penas penas penas penas penas penas penas penas penas penas penas penas penas penas penas penas penas penas penas penas penas penas penas penas penas penas penas penas penas penas penas penas penas penas penas penas penas penas penas penas penas penas orna.
Is this a dupe of FloydHub's blog post from a couple weeks back? Markup looks very similar
I agree with the need, my issue is with reproduceability and generally things that are not defined by the sketch (behavior, different screen sizes, etc). If everything were fully defined, we could just write a compiler from the high level definition to the low level. The real problem is that static 2D images are not good representations of dynamically sized interactive documents.
Kind of related to this, what do people think frontend developers will be doing in five years, or, where will they migrate to as frontend (either via tools like this or like AirBnBs SketchToCode) becomes more automated?
what about the converse of this? what about producing a basic set of content and letting AI taxonomise or organise it and then create a design based on design models. It could work both ways. I guess ultimately the problem will be solved both ways and AI could create entire experiences based on specific domains. Also, for those claiming this will never work i would suggest looking at some of the amazing stuff being done with AI right now. This kind of problem, including optimising and making best usage of html/css is almost tailor made for AI.
Interesting outcome.
The final html code from the first example is not that bad. There's the usual problem of beginner coders of too much div wrapping that probably isn't really necessary. I'm curious if the system can also create the css. The css also suggests a beginner coder, such as an overuse of unnecessary clear classes because of the overuse of floats for layout. Or having extra class names for things that are easily handled by a parent class reference, such as a "last" class on the last li in the list. Although, it appears the css is just a template obtained online. More on that later.
The second example using bootstrap is a soft failure in my eyes. Although the html does render correctly in the browser, which is because browsers do their best to render crappy html, the code is rough. The main problem is it decided to render the head element as a header element. Compared to the first example I'm shocked that this is the generated output. The usage of bootstrap does pose an interesting thought in that the content section of the html is more precise than the first example.
My reaction to this is it's a decent try at generating a website based on very strict rules assuming that more than half of the website creation process still requires a human to complete. For example, I could see this working quite well if one were to design your mockups strictly be bootstrap, or a template, and provide that css beforehand. If the mockup is custom outside of the template/bootstrap css then it'll have to generate that css itself. Which I think I'm more curious if that's possible. Generating html is easy as you can establish ground rules of "use this series of nested elements for this situation" and so on. The examples provided could just as easily be created by a drag-and-drop system that allows a non-coder to build a basic website. For that matter, use a markdown to bootstrap converter and train your writers/editors on the bootstrap basics and off you go.
But it sure did look like a fun learning exercise. As a front end dev, I'm not worried over my future and would be curious to see where it goes.
The final html code from the first example is not that bad. There's the usual problem of beginner coders of too much div wrapping that probably isn't really necessary. I'm curious if the system can also create the css. The css also suggests a beginner coder, such as an overuse of unnecessary clear classes because of the overuse of floats for layout. Or having extra class names for things that are easily handled by a parent class reference, such as a "last" class on the last li in the list. Although, it appears the css is just a template obtained online. More on that later.
The second example using bootstrap is a soft failure in my eyes. Although the html does render correctly in the browser, which is because browsers do their best to render crappy html, the code is rough. The main problem is it decided to render the head element as a header element. Compared to the first example I'm shocked that this is the generated output. The usage of bootstrap does pose an interesting thought in that the content section of the html is more precise than the first example.
My reaction to this is it's a decent try at generating a website based on very strict rules assuming that more than half of the website creation process still requires a human to complete. For example, I could see this working quite well if one were to design your mockups strictly be bootstrap, or a template, and provide that css beforehand. If the mockup is custom outside of the template/bootstrap css then it'll have to generate that css itself. Which I think I'm more curious if that's possible. Generating html is easy as you can establish ground rules of "use this series of nested elements for this situation" and so on. The examples provided could just as easily be created by a drag-and-drop system that allows a non-coder to build a basic website. For that matter, use a markdown to bootstrap converter and train your writers/editors on the bootstrap basics and off you go.
But it sure did look like a fun learning exercise. As a front end dev, I'm not worried over my future and would be curious to see where it goes.
Looking forward to this being a photoshop plugin.
Now we just need it to do the javascript ...
Up next, an AI for creating paintings and art and Sistine Chapel ceilings.
No. This will never be a thing. It will be copycat pages, never anything unique.
No. This will never be a thing. It will be copycat pages, never anything unique.
The really useful AI would be to take business data or workflow and generate layout and styles to eliminate the hand made mockup stuff.
It's like programming a self-driving car for a 2D game. Trivial. Now try taking that to tar roads. I don't know why this toy example is even news, it's nowhere close to the difficulty of real-world design mockups.
The AirBnB sketch2code example is a lot more impressive, but that's basically just handwriting recognition with a 1 to 1 mapping of symbols to code pieces.