How the Scripto App Became a Fixture of Late-Night Comedy News(newyorker.com)
newyorker.com
How the Scripto App Became a Fixture of Late-Night Comedy News
https://www.newyorker.com/tech/elements/how-scripto-the-app-that-stephen-colbert-helped-build-became-a-fixture-of-late-night-comedy-news
45 comments
This is the site that shows some of what the program looks like: http://scripto.computer/
Wow, this looks really nice, actually. Hard to believe the creator is simply an "amateur coder".
Hi! CTO here!
Scripto, itself, is based on etherpad-lite, with a ton of other code added to support various other features necessary to support the shows and was written by some contract employees.
Scripto, itself, is based on etherpad-lite, with a ton of other code added to support various other features necessary to support the shows and was written by some contract employees.
scripto.computer's Git repo is exposed http://scripto.computer/.git/HEAD
Depending on what is in your git repo, please don’t be the next Uber - if you had any secrets in your repository (even historical), definitely roll all your API keys and check secrets/credentials etc.
Thankfully that was just our marketing site! Our team has fixed it (and re-issued any api key that could have been potentially compromised).
Our product isn't a monorepo (and has a much more rigorous release process than this site), so even if the main repo that does the build is ever exposed, the secrets aren't in there :)
Our product isn't a monorepo (and has a much more rigorous release process than this site), so even if the main repo that does the build is ever exposed, the secrets aren't in there :)
Uh yup. You can traverse the complete repo of the website.
I was wondering if it was based on etherpad- super cool!
The new version we are working on (and briefly mentioned) is based on choo[0] with the editor provided code-mirror[1] and ot-delta[2] from quill with some custom (and soon to be open sourced) glue code to make them work together.
1. https://github.com/choojs 2. http://codemirror.net 3. https://quilljs.com
1. https://github.com/choojs 2. http://codemirror.net 3. https://quilljs.com
When I was in news television in ~2005, we dabbled with replacing iNEWS with something Web-friendly and nice, but the Web landscape just wasn't there. It seems like a million years ago, in terms of the Web. Our frustrations were the same, since iNEWS and ENPS are just slightly different flavors of shit sandwich, and while a typical ENG rundown can plod along with one-active-writer-per-slug the editing locks were starting to really get in the way of some other things we wanted to do. Reading this was just like being back in those meetings.
Nice to see someone have the exact same idea and execute well. If you wanted to take on Avid, I bet you could firm up some ENG-type integrations (wire integration would be a showstopper) and get some traction in newsrooms, all of whom are having their thin budgets bled by AP and Avid. MediaCentral is a hot pile of garbage, I hear, so Avid might be handing you the opportunity as we speak.
Showrunner has a big challenge in front of it taking on the Final Draft religion, but man, if you guys crack that nut...
Nice to see someone have the exact same idea and execute well. If you wanted to take on Avid, I bet you could firm up some ENG-type integrations (wire integration would be a showstopper) and get some traction in newsrooms, all of whom are having their thin budgets bled by AP and Avid. MediaCentral is a hot pile of garbage, I hear, so Avid might be handing you the opportunity as we speak.
Showrunner has a big challenge in front of it taking on the Final Draft religion, but man, if you guys crack that nut...
The creator isn’t simply an amateur coder, it’s a team of real coders working on a program originally conceived by Colbert and an amateur coder.
Oh, I see, thanks for the clarification.
Much of the core functionality was created by the Etherpad and Etherpad Lite team. The integration, rundown editor (originally), full-text search, and other features were created largely by myself (in addition to Mindy). The use of the term "amateur coder" here is slander. But I understand why Rob and them started describing it that way -- the system fell down a few times in high-profile situations of course they totally blame me rather than taking credit for project management issues, so they used the fact that I don't have a degree against me and found all of the bad code and system restarts and pointed at that (I also wrote a lot of _very good_ code and solved some very tough technical challenges).
Really the last time I was lead that we had an issue with the server going down was a third-party component that had a bug, and it came down to me deploying it in a hurry because we didn't have a good release process and I was tired from my other job. So I believe there were project management issue which I should have been more assertive over. But if I had money then I would consider suing regarding the use of terms like "amateur coder". I started learning software engineering when I was 8 years old (32 years ago), taught myself object-oriented programming in middle school. I have studied software engineering curriculum for many years online and also developed my own philosophy and practices around software engineering focused on feedback loops of different types and levels. I am a very good software engineer, not an "amateur coder".
Really the last time I was lead that we had an issue with the server going down was a third-party component that had a bug, and it came down to me deploying it in a hurry because we didn't have a good release process and I was tired from my other job. So I believe there were project management issue which I should have been more assertive over. But if I had money then I would consider suing regarding the use of terms like "amateur coder". I started learning software engineering when I was 8 years old (32 years ago), taught myself object-oriented programming in middle school. I have studied software engineering curriculum for many years online and also developed my own philosophy and practices around software engineering focused on feedback loops of different types and levels. I am a very good software engineer, not an "amateur coder".
The way your responses read, you would think this article was some kind of hit piece on your programming skills. Not only is (to my eyes) no slandering or insults to anyone going on in this article, but the only one they call an amateur coder is Rob Dubbin. The name Jason Livesay doesn’t appear anywhere in the article.
Ummm.. Rob Dubbin was mentioned as the amateur coder. Slow down there :)
I see the problem now. By describing him that way, it implies that he was involved in the coding. In the years I worked on the project, he wrote zero lines of code for it. I was confused because that implication led me to believe that they were talking about a coder on the project, such as myself.
You shouldn't be suing them, you should be thanking them.
Whatever your perceived status was before having a New Yorker article about the success of a commercial project that you helped create, after the article is published, it is definitely that of 'professional', even if the text of the article describes you as an 'amateur'.
Unless you manage to act exceedingly unprofessionally about the whole thing, say for instance, threatening to sue them after essentially being given free advertising.
Whatever your perceived status was before having a New Yorker article about the success of a commercial project that you helped create, after the article is published, it is definitely that of 'professional', even if the text of the article describes you as an 'amateur'.
Unless you manage to act exceedingly unprofessionally about the whole thing, say for instance, threatening to sue them after essentially being given free advertising.
First of all, this in no way provides advertisement for me, nor do I need any. I have projects, I don't need more. And if I go looking for another project, I just need a spec or idea -- I can prototype virtually any type of technology in a relatively short time.
There is a lot more to this story than you know about. My comments were actually extremely restrained.
You should talk about things that you actually know about.
They took advantage of me, then threw me under the bus and discounted my work.
The one thing that will absolutely enrage me is someone throwing around the word 'unprofessional'.
He did discount my work in that article. I hope that actually the 'amateur coder' comment was just implying falsely that Rob coded it and not referring to me. If he wants to refer to me as an amateur coder though, then yes, I will hope that I can save up and sue him some day for that slander, no matter how famous or connected a lawyer his dad is.
There is a lot more to this story than you know about. My comments were actually extremely restrained.
You should talk about things that you actually know about.
They took advantage of me, then threw me under the bus and discounted my work.
The one thing that will absolutely enrage me is someone throwing around the word 'unprofessional'.
He did discount my work in that article. I hope that actually the 'amateur coder' comment was just implying falsely that Rob coded it and not referring to me. If he wants to refer to me as an amateur coder though, then yes, I will hope that I can save up and sue him some day for that slander, no matter how famous or connected a lawyer his dad is.
It clearly does provide some advertisement for you, even if you do not want or require it. The general public are being informed about an item on your CV.
I am sure that there is a lot more to this story than I know. I only just encountered it and newspapers are notoriously bad at accurately portraying anything remotely technical.
I refuse to talk about only things I know. Nobody else does, and it would make finding out about new things difficult.
I am sorry to hear that you had a bad experience here, I have had numerous shitty clients over the years, however if your client does have a case to answer, the wise thing to do is go down the legal route and do not air any dirty laundry until after the dust settles.
You are not going to like this, but if you get enraged by someone throwing around the word unprofessional, then you are unprofessional. You might be the best coder on earth, but the professional response to being called unprofessional is to ask for written examples and a followup meeting, not rage.
As far as the plan to save up in order to go off and sue people whose fathers are famous and well connected lawyers, I do not know the details, however on the surface it doesn't sound all that good a route to happiness, but at least you will find it easier to accomplish now that you have had your work promoted in the national press.
I am sure that there is a lot more to this story than I know. I only just encountered it and newspapers are notoriously bad at accurately portraying anything remotely technical.
I refuse to talk about only things I know. Nobody else does, and it would make finding out about new things difficult.
I am sorry to hear that you had a bad experience here, I have had numerous shitty clients over the years, however if your client does have a case to answer, the wise thing to do is go down the legal route and do not air any dirty laundry until after the dust settles.
You are not going to like this, but if you get enraged by someone throwing around the word unprofessional, then you are unprofessional. You might be the best coder on earth, but the professional response to being called unprofessional is to ask for written examples and a followup meeting, not rage.
As far as the plan to save up in order to go off and sue people whose fathers are famous and well connected lawyers, I do not know the details, however on the surface it doesn't sound all that good a route to happiness, but at least you will find it easier to accomplish now that you have had your work promoted in the national press.
I always thought "amateur" had a very objective definition: what you do for a living. If you make your core income from programming, you're a professional, not an amateur. It says nothing about the quality of your work, although there's obviously an implied correlation.
I was Rob Dubbin's teaching fellow for a computer science course he took in college. He was pretty sharp!
Rob Dubbin was on the Track Changes podcast with Paul Ford a couple of months ago and it was a pretty great episode: https://trackchanges.postlight.com/rob-dubbin-goes-off-scrip...
Nice. Collaborative version control for Script writing/show running. Can the "amateur coder" and Stephen Colbert make something like this for programming? That would be very nice/useful. A layer on git? Is GitHub/GitLab/Bitbucket already it? Real-time and ease of use don't seem to be there.
Are there editor plugins like this for paired / multi programming? Is it just that the problem of "programming" is a lot more complex / open ended than show script development / execution? It seems like they have a lot of similarities.
Getting multiple people to agree on the same tool may be a big part of the battle, but surely there are some comparable programming tools out there.
Are there editor plugins like this for paired / multi programming? Is it just that the problem of "programming" is a lot more complex / open ended than show script development / execution? It seems like they have a lot of similarities.
Getting multiple people to agree on the same tool may be a big part of the battle, but surely there are some comparable programming tools out there.
https://code.visualstudio.com/blogs/2017/11/15/live-share
VS and VS Code are going to support this. Should be neat when it comes out.
VS and VS Code are going to support this. Should be neat when it comes out.
My name is Jason Livesay (https://github.com/runvnc). I was the original lead developer of this software (with Mindy Tchieu also doing a lot of the front end, and Sasha Stewart working hard on QA along with her other normal Colbert Report job, don't know why they weren't mentioned in the article). Rob Dubbin didn't code anything on the project by the way, although he does deserve at least as much credit as me for keeping the project going. My current income is below the federal poverty level, but at least this time I have partial ownership of the product/company I am building (its automated statistical arbitrage on cryptocurrencies).
To answer your question, yes, there are multiple good collaborative code editors out there. The software was originally based on Etherpad, and after a year or two I switched it over to Etherpad Lite. I don't know if they now are based on something else at this point, I doubt it -- maybe their new "Showrunner" has a different core. But for something just like that for code, there is this https://www.npmjs.com/package/ep_codepad. Probably the best option though would be something like Cloud 9 https://github.com/c9
In case anyone is interested in my own take in general on this software, I will write a few comments here also. We started working on it in 2011. I consider this to be a cautionary tale for 'freelance' developers working for startups everywhere (more on this towards the end).
I solved most of the key technical challenges that enabled the main Colbert Report writing group to start using the system well before 2015 (I forget when the main group started using it mostly full time, maybe 2014). We had a lot of problems, but certainly nothing catastrophic every few weeks the way Rob suggested. One note about this, the first request I made (and I repeated multiple times) to Rob was to hire a professional quality assurance engineer to help with the release process, and this was never done.
Some of the key technical challenges that I solved:
1) how to make a system for TV writers that works like Etherpad and has a few other features. This is how the problem was originally described, and I took the most direct approach -- I built a prototype based on Etherpad, which was a Rhino (JS in Java) and Scala program. (The other sort of 'runner-up' for developer selection had presented a prototype built from scratch with zero actual collaborative functionality.)
2) reverse-engineering the partially text/partially binary-based document database for ENPS and creating JavaScript code to read it.
3) full-text document search for the script document database (more than 1GB of data) -- there were multiple approaches, one thing that mostly worked was based if I recall on redis and https://www.npmjs.com/package/natural, then later I rebuilt it on Elasticsearch.
4) a Google Spreadsheets-like collaborative realtime rundown editor
5) outputting script and rundown data as PDFs to match their old print-out format
6) how to deploy the software. I built a single all-inclusive Docker container starting back in January 2014 (they later took it apart, which at the time the component infrastructure for Docker didn't make that a good trade off, now is a different situation).
There are a few lessons I took from this project:
-- if you find yourself working full-time as a 'freelance' developer for a project for more than a year or two, you are probably going to get screwed. For the last few years I have been careful to explain up-front that if I work full-time on a project for a significant amount of time then I will need equity.
-- be assertive in terms of project management. Sometimes this is hard to do because freelance developers may not always have stable income and that may make it hard to put your foot down. There were some core problems with the server administration, lack of resources, and release process that I would have been able to correct if I had more leverage (in terms of savings?) or balls. The server admin, back-end coding, tech support, and support for front-end was all on me, even after I had taken on another project in order to make ends meet. I should have insisted on more resources.
To answer your question, yes, there are multiple good collaborative code editors out there. The software was originally based on Etherpad, and after a year or two I switched it over to Etherpad Lite. I don't know if they now are based on something else at this point, I doubt it -- maybe their new "Showrunner" has a different core. But for something just like that for code, there is this https://www.npmjs.com/package/ep_codepad. Probably the best option though would be something like Cloud 9 https://github.com/c9
In case anyone is interested in my own take in general on this software, I will write a few comments here also. We started working on it in 2011. I consider this to be a cautionary tale for 'freelance' developers working for startups everywhere (more on this towards the end).
I solved most of the key technical challenges that enabled the main Colbert Report writing group to start using the system well before 2015 (I forget when the main group started using it mostly full time, maybe 2014). We had a lot of problems, but certainly nothing catastrophic every few weeks the way Rob suggested. One note about this, the first request I made (and I repeated multiple times) to Rob was to hire a professional quality assurance engineer to help with the release process, and this was never done.
Some of the key technical challenges that I solved:
1) how to make a system for TV writers that works like Etherpad and has a few other features. This is how the problem was originally described, and I took the most direct approach -- I built a prototype based on Etherpad, which was a Rhino (JS in Java) and Scala program. (The other sort of 'runner-up' for developer selection had presented a prototype built from scratch with zero actual collaborative functionality.)
2) reverse-engineering the partially text/partially binary-based document database for ENPS and creating JavaScript code to read it.
3) full-text document search for the script document database (more than 1GB of data) -- there were multiple approaches, one thing that mostly worked was based if I recall on redis and https://www.npmjs.com/package/natural, then later I rebuilt it on Elasticsearch.
4) a Google Spreadsheets-like collaborative realtime rundown editor
5) outputting script and rundown data as PDFs to match their old print-out format
6) how to deploy the software. I built a single all-inclusive Docker container starting back in January 2014 (they later took it apart, which at the time the component infrastructure for Docker didn't make that a good trade off, now is a different situation).
There are a few lessons I took from this project:
-- if you find yourself working full-time as a 'freelance' developer for a project for more than a year or two, you are probably going to get screwed. For the last few years I have been careful to explain up-front that if I work full-time on a project for a significant amount of time then I will need equity.
-- be assertive in terms of project management. Sometimes this is hard to do because freelance developers may not always have stable income and that may make it hard to put your foot down. There were some core problems with the server administration, lack of resources, and release process that I would have been able to correct if I had more leverage (in terms of savings?) or balls. The server admin, back-end coding, tech support, and support for front-end was all on me, even after I had taken on another project in order to make ends meet. I should have insisted on more resources.
Not only is this comment packed with good information, but the OP also has a fantastic last name.
What hourly/daily/weekly rate were they paying you? It sounds like it must have been low if you weren’t able to save money.
I've been around the block a few times, so maybe I can give you some counter advice which I hope you're find useful.
- Projects can be technical successes without them being satisfying/fulfilling in one way or another for all parties. Developers are usually skilled at development and not so skilled at navigating political problems. Good managers are worth their weight in gold because they take care of those problems (IMHO, managers who avoid dealing with those problems are probably not worth working with). It took me a long time to realise this. As I've gotten more experienced, I've learned how to deal with political issues and I can do that role if I need to -- but I'd rather not. So basically I tend to demand: give me a great manager or a lot more money ;-)
- Equity is fine, but it's often a big risk. Even more, asking for equity means placing you back into the role of having to deal with political issues -- because there are a lot of ways to screw people over when it comes to equity. Entrepreneurs often do that role because they like competing for money. Good entrepreneurs (the ones that will have successful companies) are good at making sure that they end up with the money -- just like you are good at writing code. You probably don't want to place yourself in competition with them. My philosophy: cash on the barrel head. I don't work for promises (though I have a bad habit of giving outrageous discounts to work on projects that I want to work on ;-) ).
- Project management is hard. A person who doesn't understand this is likely to be a really bad project manager. Most projects have lamentable project management. After decades in the industry I've learned innumerable ways to screw up a project. I have learned precious few ways to succeed dramatically -- and those ways can easily be pushed over by well meaning, but under experienced management. (And come to think about it, they can be pushed over by well meaning, but under experienced me). Any idiot can succeed when the situation is set up perfectly. The real trick is succeeding when everything is against you. Learn to play in hard mode -- it will help you in the long run.
- It's hard, but important to control your ego. In addition to not over-estimating your worth/ability I also mean not under-estimating it. All projects have risk. If, in a headstrong fashion, you demand to control everything that you think introduces risk, you assume that risk. Others then depend on you. If you fail, then you may destroy their dreams/livelihood/fortune. It is possible to put yourself into a situation where you have assumed all the risk for the project, but almost none of the consequences. People who will suffer the consequences, but control none of the risk will naturally be put into conflict with you. It pays to be careful. On the other hand, judiciously putting risk into the hands of those who are most able to mitigate that risk is wise. That person may be you. Before getting too deep into the project, make sure that you see eye to eye about this with your partners. Failing to do so can cause the failure of the project all by itself.
You seem like a passionate, genuine developer so I wish you good luck in your endeavours.
- Projects can be technical successes without them being satisfying/fulfilling in one way or another for all parties. Developers are usually skilled at development and not so skilled at navigating political problems. Good managers are worth their weight in gold because they take care of those problems (IMHO, managers who avoid dealing with those problems are probably not worth working with). It took me a long time to realise this. As I've gotten more experienced, I've learned how to deal with political issues and I can do that role if I need to -- but I'd rather not. So basically I tend to demand: give me a great manager or a lot more money ;-)
- Equity is fine, but it's often a big risk. Even more, asking for equity means placing you back into the role of having to deal with political issues -- because there are a lot of ways to screw people over when it comes to equity. Entrepreneurs often do that role because they like competing for money. Good entrepreneurs (the ones that will have successful companies) are good at making sure that they end up with the money -- just like you are good at writing code. You probably don't want to place yourself in competition with them. My philosophy: cash on the barrel head. I don't work for promises (though I have a bad habit of giving outrageous discounts to work on projects that I want to work on ;-) ).
- Project management is hard. A person who doesn't understand this is likely to be a really bad project manager. Most projects have lamentable project management. After decades in the industry I've learned innumerable ways to screw up a project. I have learned precious few ways to succeed dramatically -- and those ways can easily be pushed over by well meaning, but under experienced management. (And come to think about it, they can be pushed over by well meaning, but under experienced me). Any idiot can succeed when the situation is set up perfectly. The real trick is succeeding when everything is against you. Learn to play in hard mode -- it will help you in the long run.
- It's hard, but important to control your ego. In addition to not over-estimating your worth/ability I also mean not under-estimating it. All projects have risk. If, in a headstrong fashion, you demand to control everything that you think introduces risk, you assume that risk. Others then depend on you. If you fail, then you may destroy their dreams/livelihood/fortune. It is possible to put yourself into a situation where you have assumed all the risk for the project, but almost none of the consequences. People who will suffer the consequences, but control none of the risk will naturally be put into conflict with you. It pays to be careful. On the other hand, judiciously putting risk into the hands of those who are most able to mitigate that risk is wise. That person may be you. Before getting too deep into the project, make sure that you see eye to eye about this with your partners. Failing to do so can cause the failure of the project all by itself.
You seem like a passionate, genuine developer so I wish you good luck in your endeavours.
Could you give more insight on "precious few ways to succeed dramatically". I'd love to hear them.
My best projects have been full on XP projects. However, XP is a subtle beast. It's tempting to look at the 12 practices of XP and say, "We're doing XP. We will be awesome", but it doesn't really work that way (in my experience). The 12 practices in XP are "generative practices". What that means is that they are meant to lead you to a process that will be "hyper productive" (I think Ron Jeffries coined that term). The 12 practices are not the process itself. It's a bit like the difference between pointing at something and the thing itself. The practices only point to the process -- which will be different for each team.
And in case you are wondering what I mean by "hyper productive" (and I apologise if this is not the original meaning, but I think it is...): it means being able to go faster as time progresses. On normal projects it is easiest to work on the project when it is still greenfield -- you have no constraints to work around. As you work on the project, the decisions you make create constraints and make it more difficult to change what it does.
With a hyper productive team it works the other way around. Imagine working on an embedded system where you may have a compiler, but you have no standard libraries, or frameworks or third party code to help you. It's very slow because you have to build all of the tools that you need to go forward. You have to consider all of the abstractions and build designs that make it easy to follow those abstractions. On modern systems we have lots of tools that allow our "green field" projects to be boot strapped quickly. Now imagine that you are constantly keeping your code in a state where every future change is like that. As the code base gains capability, it actually makes it easier to make the next thing. This, of course, is the holy grail of programming :-) You can make a lot of money selling maps to the holy grail, so you shouldn't believe me!
With that disclaimer, I will rashly state that I have been on "hyper productive" projects before. But it's a pretty ridiculous tightrope. For anything I could say, there will be someone who will reply, "Yes, that sounds great in theory, but what if X happens? It will all come crashing down". Those people are correct. That's why XP is not the 12 practices. They are just something that will hopefully make you aware of what you need to do when X happens.
I don't think XP is the only way to get to "hyper productive", but it is they way I have used to get there. I don't know any other way, personally, but that doesn't mean that it doesn't exist. Usually when people ask me the question you have asked, I direct them to XP and suggest that they study it thoroughly (including running projects in full-on XP mode). The problem is the "Agile" attitude that people usually bring with them. Because they think, "Well, that doesn't make sense", or "I can't do that because Y" and they use their experience to insert something else. In the end they create a kind of "We tried baseball and it didn't work" situation (to borrow from Ron Jeffries again). It's a catch 22. Blindly applying the XP practices won't give you success, but you probably can't get the experience to understand where the practices are pointing without doing just that.
Even for me, where I've had success with those techniques, I often find myself in situations where I can't actually get there from here. So I try things I haven't tried before (usually without success :-) ). Or I wait until the situation changes and I can get there from here. It's really hard.
I'd love to talk more about politics and the difficulty of just getting people to walk in the same direction. For example, I often tell the story about 5 people all walking in different directions. One of the people is walking in the right direction, but every one of them thinks they are walking in the right direction, so they can't agree. Let's take the position of the person walking in the right direction. If that person just pushes on, then they will arrive safely, but the 4 other team members will fail -- and hence the project will fail. So it makes more sense for everyone to walk together even if it is in the wrong direction, because you have a chance that you can change course and succeed. If everyone goes in different directions, then you will fail, almost by definition -- even though one of them might be right.
The problem about talking about things like the above is that I don't fully understand what to do. Kent Beck's famous C3 project was a technical success (though, I'm not sure it was every fully implemented) because all of the team members were fed up with the previous process and were willing to do pretty much anything else. My most successful project was actually similar -- the previous manager had been overbearing and couldn't work with the team members. He was removed from the project and I was inserted "to make everyone happy". So I did -- they wanted to try "this XP thing" and my job was to make them happy. (Side note: you usually don't go far wrong if you make your workers happy).
But there is far more to it than that. I always say that the most dangerous actors in a project are senior management. Virtually every failed project I have been on (and I've been on a lot) has failed because it was cancelled by senior management. Now, I say that partially tongue in cheek because projects don't fail until they stop and they don't stop until they are cancelled :-). But it's actually true in a very real fashion. Most poor choices actually stem from trying to appease stake holders or managers who are worried about something. Again, I will say that XP can really help you there, but I will forgive you if you look at the 12 practices and say, "I don't see how".
I didn't really intend to write a long response to this question, but... you can see what happened. It's really a difficult question to answer and I'm not sure that my answer sheds any light. However, just keep pressing on. The holy grail is there. I've seen it. Your path will be different than mine, but you can get there if you keep believing.
And in case you are wondering what I mean by "hyper productive" (and I apologise if this is not the original meaning, but I think it is...): it means being able to go faster as time progresses. On normal projects it is easiest to work on the project when it is still greenfield -- you have no constraints to work around. As you work on the project, the decisions you make create constraints and make it more difficult to change what it does.
With a hyper productive team it works the other way around. Imagine working on an embedded system where you may have a compiler, but you have no standard libraries, or frameworks or third party code to help you. It's very slow because you have to build all of the tools that you need to go forward. You have to consider all of the abstractions and build designs that make it easy to follow those abstractions. On modern systems we have lots of tools that allow our "green field" projects to be boot strapped quickly. Now imagine that you are constantly keeping your code in a state where every future change is like that. As the code base gains capability, it actually makes it easier to make the next thing. This, of course, is the holy grail of programming :-) You can make a lot of money selling maps to the holy grail, so you shouldn't believe me!
With that disclaimer, I will rashly state that I have been on "hyper productive" projects before. But it's a pretty ridiculous tightrope. For anything I could say, there will be someone who will reply, "Yes, that sounds great in theory, but what if X happens? It will all come crashing down". Those people are correct. That's why XP is not the 12 practices. They are just something that will hopefully make you aware of what you need to do when X happens.
I don't think XP is the only way to get to "hyper productive", but it is they way I have used to get there. I don't know any other way, personally, but that doesn't mean that it doesn't exist. Usually when people ask me the question you have asked, I direct them to XP and suggest that they study it thoroughly (including running projects in full-on XP mode). The problem is the "Agile" attitude that people usually bring with them. Because they think, "Well, that doesn't make sense", or "I can't do that because Y" and they use their experience to insert something else. In the end they create a kind of "We tried baseball and it didn't work" situation (to borrow from Ron Jeffries again). It's a catch 22. Blindly applying the XP practices won't give you success, but you probably can't get the experience to understand where the practices are pointing without doing just that.
Even for me, where I've had success with those techniques, I often find myself in situations where I can't actually get there from here. So I try things I haven't tried before (usually without success :-) ). Or I wait until the situation changes and I can get there from here. It's really hard.
I'd love to talk more about politics and the difficulty of just getting people to walk in the same direction. For example, I often tell the story about 5 people all walking in different directions. One of the people is walking in the right direction, but every one of them thinks they are walking in the right direction, so they can't agree. Let's take the position of the person walking in the right direction. If that person just pushes on, then they will arrive safely, but the 4 other team members will fail -- and hence the project will fail. So it makes more sense for everyone to walk together even if it is in the wrong direction, because you have a chance that you can change course and succeed. If everyone goes in different directions, then you will fail, almost by definition -- even though one of them might be right.
The problem about talking about things like the above is that I don't fully understand what to do. Kent Beck's famous C3 project was a technical success (though, I'm not sure it was every fully implemented) because all of the team members were fed up with the previous process and were willing to do pretty much anything else. My most successful project was actually similar -- the previous manager had been overbearing and couldn't work with the team members. He was removed from the project and I was inserted "to make everyone happy". So I did -- they wanted to try "this XP thing" and my job was to make them happy. (Side note: you usually don't go far wrong if you make your workers happy).
But there is far more to it than that. I always say that the most dangerous actors in a project are senior management. Virtually every failed project I have been on (and I've been on a lot) has failed because it was cancelled by senior management. Now, I say that partially tongue in cheek because projects don't fail until they stop and they don't stop until they are cancelled :-). But it's actually true in a very real fashion. Most poor choices actually stem from trying to appease stake holders or managers who are worried about something. Again, I will say that XP can really help you there, but I will forgive you if you look at the 12 practices and say, "I don't see how".
I didn't really intend to write a long response to this question, but... you can see what happened. It's really a difficult question to answer and I'm not sure that my answer sheds any light. However, just keep pressing on. The holy grail is there. I've seen it. Your path will be different than mine, but you can get there if you keep believing.
Overall, I appreciate and respect your thoughts about the relationship between the aspirations of XP, your experiences with XP, and XP-as-a-written-document (c.f. the pointing hand vs the things pointed at). Thank you for sharing them. And so I apologize if the following distracts or detracts from that...
My understanding is that C3 was a failure in every respect (except in terms of the results on the careers of certain evangelists). So I'm curious what you mean when you say "a technical success".
My understanding is that C3 was a failure in every respect (except in terms of the results on the careers of certain evangelists). So I'm curious what you mean when you say "a technical success".
First, I have to say that I think you are right to call me out on that one. Especially since I was venturing into the territory of "projects getting cancelled because senior management is getting worried". I've been lucky enough to have passing conversations with some of the team members on mailing lists and met one or two more at conferences. I'll be honest in that I don't really know what went on there. My impression (and I'm happy to be corrected) is that the team members felt that the way the work progressed was a success. That's what I meant from the kind of caveat of "technical success". I don't think the project was a success from the perspective of delivering a product that the stakeholders were happy with (again, not trying to step on toes and I'm happy to be corrected). My main point in bringing up that project was more to highlight the conditions that might be necessary to get a group of people working together in that way -- namely that people were unhappy with the previous process.
It's hard to talk about this kind of stuff for reasons like you bring up. As you say, XP and the mythos surrounding it brought many people to the public eye that might not have been so noticeable. Although having lived through that time, Kent Beck and Ward Cunningham were pretty big names in the OO community even before XP. In fact, I doubt that anybody would have looked at XP (given that it is so strange) if someone lesser known had been behind it.
I struggle with this quite a bit. I've been lucky enough to meet and even work with some of the big names in the London tech scene. I mean, people are people and they sometimes have some pretty stupid ideas. They sometimes do some stupid things that don't work well. They sometimes don't realise which of their ideas are stupid :-) But without exception, they've been really awesome to talk to, work with and learn from.
I think C3 suffers a bit from the perspective of being "the example", even though it's clear that it wasn't all positive. I tend to look at it as the point at which the people involved realised, "Hey, this could work." When you decide to write a book about this kind of stuff, on one side you are being a good self-promoter, but on the other side you are really putting yourself out there. Is there some defensiveness there? I wouldn't doubt it.
My view has been to try to ignore the people behind the ideas. A lot of time you have to think to yourself, "How do I evaluate this idea? There are so many ideas. Which ones should I consider and which ones should I let go?". Unlike Pokemon, I think the secret is that you don't have to catch them all. Although there are many paths out there, my personal feeling is that there are probably not that many destinations. We'll all converge eventually. So follow the path that looks interesting to you.
XP is the path that took me the farthest so far. It's hard to give people advice because I can say, "I know XP can take you here, because I did it that way". I don't know any other way so far, so I can't give good advice on other good paths (although I can give a lot of advice about bad paths -- as I'm sure all experienced developers can relate to :-) ).
Hope that answers the question!
It's hard to talk about this kind of stuff for reasons like you bring up. As you say, XP and the mythos surrounding it brought many people to the public eye that might not have been so noticeable. Although having lived through that time, Kent Beck and Ward Cunningham were pretty big names in the OO community even before XP. In fact, I doubt that anybody would have looked at XP (given that it is so strange) if someone lesser known had been behind it.
I struggle with this quite a bit. I've been lucky enough to meet and even work with some of the big names in the London tech scene. I mean, people are people and they sometimes have some pretty stupid ideas. They sometimes do some stupid things that don't work well. They sometimes don't realise which of their ideas are stupid :-) But without exception, they've been really awesome to talk to, work with and learn from.
I think C3 suffers a bit from the perspective of being "the example", even though it's clear that it wasn't all positive. I tend to look at it as the point at which the people involved realised, "Hey, this could work." When you decide to write a book about this kind of stuff, on one side you are being a good self-promoter, but on the other side you are really putting yourself out there. Is there some defensiveness there? I wouldn't doubt it.
My view has been to try to ignore the people behind the ideas. A lot of time you have to think to yourself, "How do I evaluate this idea? There are so many ideas. Which ones should I consider and which ones should I let go?". Unlike Pokemon, I think the secret is that you don't have to catch them all. Although there are many paths out there, my personal feeling is that there are probably not that many destinations. We'll all converge eventually. So follow the path that looks interesting to you.
XP is the path that took me the farthest so far. It's hard to give people advice because I can say, "I know XP can take you here, because I did it that way". I don't know any other way so far, so I can't give good advice on other good paths (although I can give a lot of advice about bad paths -- as I'm sure all experienced developers can relate to :-) ).
Hope that answers the question!
Check out: SubEthaEdit [1]. It's been around a long time, and it's a beautiful product. Caveats: it costs money (worth it IMO), and it's Mac only.
[1] https://www.codingmonkeys.de/subethaedit/
[1] https://www.codingmonkeys.de/subethaedit/
It was an innovation, but is it maintained?
If you like <sarcasm>editors with long startup times </sarcasm> you can check out Teletype from the Atom team: https://teletype.atom.io/
Have you checked out https://glitch.com? It has multi-user features in it.
Wouldn't there already be a hundred things like this?
Yes.
I’m in the midst of writing a script with a few other people, and we’ve been using Writer Duet: https://writerduet.com
A lot of people don’t know this but there’s a one person startup called Rundown Creator that a lot of big companies use to plan shows.
Wow. I use Etherpad every day. Really enjoy Colbert. Colbert is really innovative in his use of tech. He uses Adobe to make the cartoon Trump and gives regular updates to Adobe to improve the product(the name escapes me).