Cool work! I'm very interested in this topic. Just wondering, how good does it generalize your training data other than just remembering strict input-output mapping?
Mapping screenshots to code is not hard. By having the model simply memorize the screenshots to code mappings of the training data can give you almost 100% accuracy (for some demo). What is hard is if given a new screenshot, how would this model generalize. To have something work for mobiles is a much easier task than having something work for other more complex UI though. Looking forward to seeing more updates on this!
Great article. Covers a lot of aspects on how ML can help designers be more creative and productive!
At Huula, I'm a firm believer that ML can automate various parts of web designs. We just released a new experiment CSSToucan[1] to auto color texts on web pages with Recurrrent Neural Networks. It learns to color texts on web pages without a single line of color theories in the code. All learned from the data. Hope to see more and more ML powered design tools emerging!
As a DIY drone builder for 5 years, here are my two pennies.
Arduino lets you program it. Apm and multiwii are essentially arduino. Video is a bit tricky, but you can always transmit it back to your laptop and process however you like
Good work! Training on vector pictures instead of rasterised images seems such a good way to go. With some related data, I imagine this can also be colored.
Good point, just blindly select several "harmonious" colors are not hard at all. But as some folks pointed out, if we want to also consider the context (where the colors will be used, is it a button? or background? or text?), deep learning is definitely a good fit.
Yeah, statistical models just give you the most probable result (in terms of the data we train it on), so if you are a creative person, you probably won't find it useful.
Looks cool! I recently posted HuulaTypesetter (https://huu.la/ai/typesetter) which infers font sizes for web pages based on DOM context and CSSRooster which infers CSS class names based on the context. I'm really happy to see that there are more intelligence happening in the design world! The dream of replacing Web UI design with AI will one day come true!
All (deep) learning models suffers the issue of 'garbage in garbage out', so one way that could make the palettes more related to real world web designs is to learn the colors used in web designs directly instead of from photographs and movies since those data will has much more noise than good web designs (with video and images stripped out of course).
Looks cool! Sometimes I think incremental initialization will be the ultimate performance optimization strategy, and this is definitely a great way to achieve that!
For the debate on ng2, I have been using ng2 to build Huula for about a year, which has some amount of ng2 code. So here are my two pennies for folks who are considering using it. I don't like to be constrained by a framework, so angular's routing system always stands on my way (including ng1), so I ditched angular routing entirely. I never used angular's inline styles either, for me sass works better. Ng2's change detection can be stupid sometimes, especially for stuff similar to drag drop, but there are solutions to those cases, although a bit clumsy and convoluted, so be careful .
Transforming .a-really-long-class-name-or-id into something shorter, like .x would save a lot more bytes.
Another thing is splicing unecessary properties, which can even save more bytes, but given web apps becoming becoming so dynamic right now, that would be hard or really awkward for developer to work with too.
Hmm, the article mentioned nothing about the weight. For a medium sized quadcopter, 3 times more energy (given the same weight) means as long as 3 hours of flight time which would make all these drone delivery dream come true.