This is true but keep in mind that you have to set up a pipeline for collecting the data into HDFS (Storm? batch loading?) & you have to pay for the machines.
So while your analysis is valid, there are more "costs" at play like developer time, cluster maintenance, hardware. I like to play with Spark's ML libraries but am wary about designing projects specifically around them because of this overhead, especially when trying to distribute some API/tech that you'd like others to use.
Not trying to be a downer, I actually wish the choice to go distributed was more of a no-brainer, hah. Would love for some APIs to emerge that could be used locally/distributed transparently without actually having to run a dummy cluster & data migration to run locally.
yeah i do Scala now but same deal. Where I think the OP comment really goes wrong is when he says something like "you're gonna need all that stuff and you're gonna end up re-implementing it yourself". I really don't find that to be the case... when you're developing with libs you are cognizant of your app endpoints and just add whats needed. It's better to consciously add a lib or turn a feature on than to just accept a big ball of yarn that some cowboy framework devs shat out. In the Rails app at my job it takes a ton of debugging just to figure out how many layers of decorators are firing off around every model.
I might even go so far as to say that a lot of where JVM has strength is that contexts are pretty measurable... Spring containers, Akka systems. You instantiate them and hold them basically in the palm of your hand, you understand your scopes. Any technology that allows application scope to slip out of your fingers and intermingle with everything else under the sun is something I'd be wary of.
There is a huge legal difference between getting caught doing drugs vs facilitating the trafficking of mass quantities of controlled substances. I don't think casual marijuana smokers should get all up in arms like "hey man they are just going after everybody now!", do they offer the same support to heroin cartels?
Even if you're against the war on drugs I don't think you should really take it as a personal slight when someone operating on this scale gets arrested. Unless you really 100% believe that distribution of heavy narcotics doesn't damage society.
I feel like this is already indirectly possible with cache systems... yes there is an initial load from DB or whatever, but with Ehcache for example I think(?) all those objects just sit in the JVM, and therefore should be in RAM. If you wrote some app startup batch process to stick every object possible into the cache proactively, I think you'd have essentially what you're asking for.
I'd like to at least offer some support to this -- I'm not sure I agree with all of your conclusions but since switching to Scrum mentality I definitely notice that the goals of our team have become much less ambitious... Rather than insulating for development cycles that may not yield much for a few of months but in the end equals proper enterprise architecture, the business/devs have found it much easier to focus on small iterative tasks that are much more superficial in the grand scheme of things.
I think engineering processes benefit when they are insulated from direct scrutiny of timelines. Resorting to tackling smaller, neater issues in Agile seems to be a sign that there is a lack of technical leadership/vision. This is not necessarily the fault of Agile (I don't mind Agile, I think projects can succeed with it if approached properly), I simply agree that increasing process rigor seems to be a symptom of an engineering/mgmt team that is floundering.
In short, if your team's problem is that things are getting "messy" and "hectic" because of too much crappy copy/paste code, I have seen how Agile can just deepen that wound while masking it in a veil of "tangible progress" seen in burndown charts. Then, again, if you don't have the right people you don't have the right people. What else can you say?
my company is "totally agile" (they wish) and it gives the management layer something to do cuz they are analyzing JIRA all day coming up with horse-shit charts, so of course they keep pushing all of it like its solving high-level organizational issues in ways we will experience tangibly sometime by 2018...
if you got rid of all that process infrastructure and just had a few programmers discussing proper design patterns now & then our company would be lightyears ahead of where it is now. i think everything becomes cargo cult when you refocus technical people onto processes that discourage technicality to make management feel fuzzy about their stats. people aren't more productive now, they are just gaming the agile system
process planning has its place i just feel like its becoming the focal point of discussion rather than a means to it. bureaucracies will continue to grow, what can you do other than look for a job with a more technical group that is focused on code?
i'm a former academic, i was astronomically more productive in that setting and there were no real process restrictions imposed at all. people need to be cultured into code & self-education, not process. optimize process once there is a good technical core in place, not in lieu of one
(i've also noticed that fake agile just promotes a lack of requirements docs from business... in that case i'd be better off doing damn waterfall with a half-decent reqs doc anyway...)
A doctor would probably classify this as some level of abuse, since medically accepted drinking levels are low (few drinks a week, I think?). Even if its just "1 per day", chemical dependency in order to function is usually seen as an issue. Sure, small doses of coffee & alcohol are well... small, but there are tons of people out there who rely on neither at all, where as the absence of these "small" doses would greatly impact your life. EDIT: I should mention though that I am not really against small-dose self-medication, as the alternative of getting medicated through the "proper system" is probably even more frightening. I do wish I could kick all small-dose habits though, rather than dealing with relapses (boring work without coffee is haaaard) & other compensatory habits.
As for your 10+ beers followed by welcome hangover comment, I believe Adorno addresses this in Minima Moralia and some of his other texts. His basic belief is that people are so alienated from themselves that they treat the weekend (their only time to really be themselves completely) as a respite that will ready them for the onslaught of the next week's work. Basically, the work week and attitudes that have arisen to rationalize it are a system of total self-alienation, with all traces of your "in an ideal world" personality smothered out of you.
Anyway, sorry if this came off sounding very dramatic. I really am not like deeply worried about you, hah, I'm just saying from a theoretical standpoint what you're describing does not prove your point at all. It's more like you're just using black humor to perpetuate bread&circus mentality.
heh thanks for the words of encouragement. i'm going to try now, yes. but basically.... to be fair i'm getting paid a pretty good wage so i really just need some experience to avoid looking like a dabbler. I'm looking at senior level positions now & they seem to want to see that you're really committed to a certain type of code.
I went from research Matlab / dabbling in C# at a dayjob, to RoR, to javascript, to now pretty hardcore backend Java (EE, Spring, high concurrency for last 2 yrs), trying to move into Clojure a bit more. Most well-funded companies don't seem to like polyglots from what I can tell... But yeah I will be looking for the ones that do, plus I am studying C++ to make a bit of a more lateral move into audio digital signal processing / data-mining / machine learning.
I don't know... companies tell me I'm too expensive for the level of experience I have, yet all the high-paid seniors I work with are awful so I don't want to lower my salary expectations unless I can find a company thats actually worth sacrificing for.
yep there are also a lot of situations in enterprise where you don't exactly know how your code will be used in near future (in <5 years it could grow to exponentially more customers using it compared to soft releases to select test customers).
Because of that I'd say.... it's not like I see a lot of people get raked over the coals for it, but there is definitely some organizational disappointment when a silly scaling bug crops up in production[1]. I think as a mature dev it is proper to internalize as many performance-related techniques as possible so that you can write more performant code with each subsequent project. Sometimes it doesn't even affect the amount of initial coding hours, it is just one of the freebie gains that comes with experience.
[1] Many more such bugs would probably occur if the company didn't occasionally get around to load-testing, so that should be part of CI or dev cycle if possible.
Honestly, I'd rather they make it a paid service than a dystopian clusterfuck of advertising espionage.
I'm more likely to quit the wonnnnderful "free" service you describe and defend because it "costs millions of dollars a month to run". Those millions of dollars are just R&D on how to get advertisers more intimate data. If they didn't pay for all that gross research, the infrastructure wouldn't cost nearly as much.
This cost-center point is key. I work for one of the big international electronics corps as an enterprise web dev, but they have no interest in the quality of our work. They don't understand it, they have no belief in codebase maintainability (everything is new development or bugfix, basically as soon as an app is written it becomes frozen legacy code that will sit for 5+ years until a rewrite is finally budgeted).
A lot of the superiors they hire really affect the architectural policies of the company. For example, we can use Spring & jQuery but little else. Now they are trying to write apps with heavier front-end & getting into trouble cuz its a mess of jQuery callbacks with no real js framework. Why? The main architect has never heard of React, Angular, backbone, etc. (not to mention the fact that I'm one of like 2 devs who can actually write js, and unfortunately the other does not possess the equally rare skill: "uses proper design patterns"). You'd think I was joking until you met a bunch of corporate consultant programmers.
Basically, these companies try to boil web development into an old-school "IT" process but the level of intellect is astonishingly low. I've been lookin for a new gig at a tech company for quite a while but it was tough to make a change cuz this is my first gig out of grad school, had to prove myself a bit & build resume.
The only relief is that the corp "cost-center" IT kindof job I have is such a mess that its a good place to study & relax, pace of work is veeeeery slow.
hah i am jealous of you both... i don't get the kickbacks, i just get one more team of consultants that i have to work with on every project who relish the tiny bit of authority they have now been granted (Message Broker team thinks they're hot shit cuz they are a bottleneck on every project & we can't revise our service interfaces without their permission).
Funny thing is, all the consultants who specialize in these products are awful pseudo-programmers & I end up bypassing their techs (WMB, BPM, etc.) with pure Java solutions whenever possible since the overhead of speaking to them creates like a 10x productivity loss. If the nosy busybodies (awful below average pseudo-programmers within our own group) didn't tattle to mgmt to help them reign us back in, the whole company would be none-the-wiser & we'd get work done at breakneck speed... ALL MOST OF US NEED IS A GODDAMN SPRING OR EE CONTAINER + THE JVM, hah.
this may be true. I use a lot of IBM server/development techs & pretty much all of them are missing the magic they claim, boiling down to "confusing forced development paradigm that bleeds millions from your company in infrastructure and niche developer costs" (Web Experience Factory, Coremetrics, Portal etc.). None of the out-of-the-box-everything-integrates-seamlessly aspect is there, despite that being the whole basis for these monstrous sales prices. In short, many of their products have laughably little value and even non-technical corporate management is realizing this.
(You can tell a lot of the software is sold with IBM thinking "We'll tell them its their fault for a couple of years unless they hire a few of our consultants, then if they realize the product actually sucks hopefully we will have an upgrade ready by then.")
However, whenever I see articles from that Cringely dude (the one referenced repeatedly in this summary) I have to think "eh, maybe.... I'll believe it when I see it". He is pretty much an anti-IBM conspiracy theorist from what I can tell, though of course they are a strange enough organization that a lot of what he says turns out to be true.
I think a big issue is that a lot of these rules need to be revised as language features evolve. A lot of productive paradigms are no longer "pure" code.
I see a lot of comments kindof poo-pooing annotations, but one of the better Java devs I know is convinced that annotations are the solution to code readability/overengineering in Java -- in essence, custom annotation processors can replace both the need for interfaces and abstract classes.
One of the biggest problems I see is simply that the schism-ing of modern design paradigms means that debugging tools have to play catchup and therefore make code seem a bit less linear. But the reality is, through IoC/AOP/annotations, developers are often reducing the number or interfaces to traverse and making the code more readable, while at the same time actually making it more generic (your class doesn't have to conform to so many standards if you can tack the annotations on whatever fields/methods you want). Should someone be introducing a proxy layer for every class just in case they need to fit it into a more advanced design in the future? Or would it be easier to just code more literally while language/container designers work on a more seamless replacement method?
In a way, it does just seem like some of these new techniques are a hacky way of forcing FP into OOP. Lots of different design paradigms playing nicely within the same VMs, ecosystems is a nice problem to have though. :)
i did this with an AWS system because we never configured security keys properly -- basically got tired of whitelisting 1 IP at a time for development tasks, especially since I had issues accessing from behind corp firewalls.
client told me they got a big IO bill from AWS & i was deathly afraid it was from bots just probing the SQL server ports. thankfully it wasn't (it was from s3, they are a media heavy site with many users) but it reminded me not to just leave things accessible via internet, cuz even though it seems irrational to think someone would just probe you over & over, the fact is you have to pay an AWS bill, & attackers are just operating from their laptop.... if the amplification is right they can cost you looooots of $$ with no cost to them.
amen. there are downsides to the chaos of consulting that i don't particularly enjoy but the only time i ever spoke to HR was when they handed me my badge, and there is no such thing as a "performance review", reprimands, etc. (lots of annoying memos about corporate policies at this particular client tho). In fact, I talk to my real "boss" about once every few months, in between it is just scrums with PMs
I pretty much just go to work, chill out, and hope I can find a better gig sometime in the near future (I'm on basically an ever-renewing contract but the vendors in our system are horrible about rate increase).
i kindof believe all languages should have some ability to run with a browser-based GUI... why bother with all the hair-brained implementations (Swing, Qt, etc.) when we could just have a "desktop mode" (altered security) version of Chrome pop up to provide GUI. it wouldnt even have to follow same rules as browser really (navigation could be via buttons only, no back/refresh), just be based on html & perhaps js to allow a bit of advanced nav logic (though hopefully not enough to start polluting app logic)
i believe chrome apps may have some steps toward this, though not sure... at 1 point i considered trying to make a browser-based GUI that does all local RPC communication to a c++ app, but i feel like you know.... there should be a nice standard way of doing this out there
amen... i struggle with this a lot & though its cute to think "oh but i get to write code at work & be creative!!" all of the monstrous bureaucracy is very stifling, & my creativity is definitely in spite of my job.
is this "work is play" mentality something that the new generation of office workers actually believes? Given that 9-5 (or 6) is an outdated force-fed corporate notion, people seem to just be paid in a way that is almost directly correlated with how much annoyance they will face daily. The difficulty of the work i face at a corp is laughable.... after 6 months adapting, these jobs are mostly about keeping a straight face while beefing up one of the millions of coding projects that are basically child's play for anyone competent, yet need to be done to keep corp bureaucracy running or keep people clicking for some service.
Basically, my mind reels thinking about the great things i could achieve if i were not in this chair. If you are really "into" society (a yuppie, basically) then you might think that living in the big city, decorating your room just right & going out to see an indie film after work means you're inspired & hitting creative peaks (Brooklyn is filled with people living vicariously like this). The creative geniuses of previous generations would die of laughter, given that most of them truly devoted their lives to excellence in a given field in ways that were basically pathological.
There are rare counter-cases but I appreciate you (peterwwillis) pointing out that many of these examples are a complete crock. "Did you know ______ was a banker when he/she wrote his/her first symphony?" This can mean that ______ was either extremely well-adjusted and working in off hours, or was extremely irate and doing it in spite of his/her job. The anecdote alone has an implied simultaneity that is usually grossly distorted to serve as a success story to inspire continued enslavement of bright, delusional minds.
It's not that I believe EVERYTHING in the world is a corporate conspiracy, but basically you know whether or not you have a shit job that is destroying your creative mojo, and a few flaky anecdotes cant change that... time theft, maybe. :) But even that is tough, since many companies shun remote work ("oh no the developers might complete tasks efficiently & then enjoy a sandwich instead of working"). They wouldnt keep you strapped in the (mental) electroshock chair if they didn't want to come by & press the button.
So while your analysis is valid, there are more "costs" at play like developer time, cluster maintenance, hardware. I like to play with Spark's ML libraries but am wary about designing projects specifically around them because of this overhead, especially when trying to distribute some API/tech that you'd like others to use.
Not trying to be a downer, I actually wish the choice to go distributed was more of a no-brainer, hah. Would love for some APIs to emerge that could be used locally/distributed transparently without actually having to run a dummy cluster & data migration to run locally.