Confessions of an Unreal Engine 4 engineering firefighter(allarsblog.com)
allarsblog.com
Confessions of an Unreal Engine 4 engineering firefighter
https://allarsblog.com/2018/03/16/confessions-of-an-unreal-engine-4-engineering-firefighter/
116 comments
> The simplified executive line of reasoning, why to rather trust an external entity over their own people, goes usually along these lines: "We do not take advice from our employees. We hired them for grunt work and we only expect them to follow orders. What could they possibly know that could benefit our high-level decision making. Afterall we pay them so little".
It's not always that leadership ignores line-engineers just because they consider them stupid. Even obvious recommendations get tangled into the web of politics and personal favors. Sometimes the 'obvious solution' is also a play to get more power or influence for someone you're opposed to. That factors into how the advice is taken.
Part of the fiction management sells itself is consultants can cut across those organizational boundaries and find the 'right solution'. Sometimes having an 'impartial' voice echoing what others have already said is enough to convince you it's not just someone trying to poach your turf, it's actually good advice.
Further, our tendencies towards tribalism influence this process. We tend not to think of other employees as part of the same group - they are devs, or ops, or product. That builds an 'us vs. them' mentality that's hard to break. Bringing in an outsider tends to unify the group into us (the company/team) vs. the outsider (the consultant). That psychology helps us to justify bitter medicine like our team getting more work (which will help the company).
I tend to have a negative opinion of consultants, but someone pointed this aspect of group psychology to me in the past and it resonated.
It's not always that leadership ignores line-engineers just because they consider them stupid. Even obvious recommendations get tangled into the web of politics and personal favors. Sometimes the 'obvious solution' is also a play to get more power or influence for someone you're opposed to. That factors into how the advice is taken.
Part of the fiction management sells itself is consultants can cut across those organizational boundaries and find the 'right solution'. Sometimes having an 'impartial' voice echoing what others have already said is enough to convince you it's not just someone trying to poach your turf, it's actually good advice.
Further, our tendencies towards tribalism influence this process. We tend not to think of other employees as part of the same group - they are devs, or ops, or product. That builds an 'us vs. them' mentality that's hard to break. Bringing in an outsider tends to unify the group into us (the company/team) vs. the outsider (the consultant). That psychology helps us to justify bitter medicine like our team getting more work (which will help the company).
I tend to have a negative opinion of consultants, but someone pointed this aspect of group psychology to me in the past and it resonated.
> Part of the fiction management sells itself is consultants can cut across those organizational boundaries and find the 'right solution'.
Being a consultant, and knowing multiple others, I feel that not being part of the company "tribes", really allows you to more easily cut across those organizational boundaries you mention.
How many times I've heard of 2 technical divisions complaining about each-other where the main issue is the lack of communication and simply not knowing who to talk to in the other "tribe". Consultants however are part of their own little 'consultant tribe' across company tribe boundaries, who more easily bond together and talk about stuff like that. This means this group is more diverse and has more contacts spread throughout the organization, making them ideal to cut trough the bullshit. You also have the fact that when a company was happy about a consultant, chances are that when they need someone somewhere else, they'll hire the same consultant again, dumping them in a completely different group of people.
Yes this is 100% a social problem, but it's there, and a group of outsiders will address this more efficiently.
Being a consultant, and knowing multiple others, I feel that not being part of the company "tribes", really allows you to more easily cut across those organizational boundaries you mention.
How many times I've heard of 2 technical divisions complaining about each-other where the main issue is the lack of communication and simply not knowing who to talk to in the other "tribe". Consultants however are part of their own little 'consultant tribe' across company tribe boundaries, who more easily bond together and talk about stuff like that. This means this group is more diverse and has more contacts spread throughout the organization, making them ideal to cut trough the bullshit. You also have the fact that when a company was happy about a consultant, chances are that when they need someone somewhere else, they'll hire the same consultant again, dumping them in a completely different group of people.
Yes this is 100% a social problem, but it's there, and a group of outsiders will address this more efficiently.
I think you hit the nail here with "poaching someone's turf" and the "tribalism" inside a company. I didn't mean to say that leadership considers engineers stupid. They hire them and trust them with solving complex technical problems. It just bothers me, they would often rather take a vague, generic management advice from a third party, who could not care less, over well intentioned, well reasoned, concrete advice, backed by concrete evidence from an engineer (or just do not act at all). This is just my colorful imagination, but I could almost 'hear' some managers thinking while dismissing an idea without giving reason: "I am supposed to manage you, no way you are going to tell me how to do my job". I've seen such behavior and I have no other explanation for it other than ego, office politics and 'cover my ass strategy'. This creates toxic environments and not just for the people working there. Productivity, quality and I'm 100% convinced that ultimately revenue and the ability to compete suffers from this as well.
In addition to what direfungasaur said about helping cut through political issues in a company, bringing in outside analysis can help for a couple of additional reasons. One is that individuals employees can be wrong. A change that engineers have been asking for might seem ok, but in actuality would destroy margins or otherwise be detrimental to the company as a whole if you look at broader information that might not be available to the engineers. Another reason is that if a company gets to the point where it's even considering bringing in outside help, it likely has a large number of issues that need to be prioritized. The one that a few employees have been making noise about might already be well-acknowledged, but is way down the list of Big Problems.
That's the "good case" anyway. In actuality, management consultants (the kind you're referring to) are usually brought in so someone can cover their ass and say "look, the smart, expensive people said we should do X!" regardless of whether or not X was a difficult solution to figure out.
That's the "good case" anyway. In actuality, management consultants (the kind you're referring to) are usually brought in so someone can cover their ass and say "look, the smart, expensive people said we should do X!" regardless of whether or not X was a difficult solution to figure out.
> A change that engineers have been asking for might seem ok, but in actuality would destroy margins or otherwise be detrimental to the company as a whole if you look at broader information that might not be available to the engineers
As someone who has been on both sides of this, the best advice I have for a manager in that situation is "if you say no, tell them why." As someone in a senior engineering role, to me, it's my responsibility to understand the forces at play with the product I'm developing, so that I can do my job. I accepted a long time ago that engineering is not just about being technically excellent, but also understanding how the product fits into the business. If the answer to engineering recommendations is just a flat "no", then that probably means that there's information that is keeping me from doing my job to my full capacity; I don't just throw out ideas without thinking about them, I take all of the facts that I have and come up with cost-effective solutions.
As someone who has been on both sides of this, the best advice I have for a manager in that situation is "if you say no, tell them why." As someone in a senior engineering role, to me, it's my responsibility to understand the forces at play with the product I'm developing, so that I can do my job. I accepted a long time ago that engineering is not just about being technically excellent, but also understanding how the product fits into the business. If the answer to engineering recommendations is just a flat "no", then that probably means that there's information that is keeping me from doing my job to my full capacity; I don't just throw out ideas without thinking about them, I take all of the facts that I have and come up with cost-effective solutions.
As a management consultant myself, another thing that clients tell me is that the ability to bring the message across in a way management understands and can act upon is typically valued.
I’ve of course heard the “you’re saying the same as what the engineers have been saying for years” at various points, and management might not always listen well enough to engineers, but I think people underestimate the importance of synthesizing their messages to get management to act
I’ve of course heard the “you’re saying the same as what the engineers have been saying for years” at various points, and management might not always listen well enough to engineers, but I think people underestimate the importance of synthesizing their messages to get management to act
The amazing thing to me is that now people can make games without a bunch of great engineers. Back in the day when we made Battlezone we had at least four really amazing engineers (one was a Masters in CS from MIT, one wrote early 3d workstation code, UCLA, Cal Tech CS degrees). In those days you needed people like that to just 3D to work in Windows. Direct X helped a lot. Unity and Unreal have made things we couldn't have dreamed of possible.
> The amazing thing to me is that now people can make games without a bunch of great engineers.
I remember reading an interview with a well known person in the 3D and gaming industry in a gaming magazine well over a decade ago. I can barley recall the details but one thing that stuck out was the interviewer asked why couldn't the game engine be the API giving the developers higher level tools instead of OpenGL or direct 3d, as well as audio, input and others. The answer was that game engines were too focused and not flexible enough to be used as a kitted API.
Fast forward to today and that is exactly what Unreal and Unity have achieved. They built game engines that are flexible enough to be complete tool kits to build other games. Interesting to see how far we have progressed.
I remember reading an interview with a well known person in the 3D and gaming industry in a gaming magazine well over a decade ago. I can barley recall the details but one thing that stuck out was the interviewer asked why couldn't the game engine be the API giving the developers higher level tools instead of OpenGL or direct 3d, as well as audio, input and others. The answer was that game engines were too focused and not flexible enough to be used as a kitted API.
Fast forward to today and that is exactly what Unreal and Unity have achieved. They built game engines that are flexible enough to be complete tool kits to build other games. Interesting to see how far we have progressed.
This is at least in part because hardware has advanced to the point that for many games it makes sense to optimize for development time over runtime performance. Unity and Unreal are still not as performant as a well optimized specialized engine could be in many situations but it would probably take too long and cost too much to build that specialized engine.
There are also trade offs in flexibility that come with using Unity or Unreal. They make it easy to make games that fit within their paradigms but it's relatively harder to make a game that does something radical or experimental with rendering or physics.
I say this as someone who lives in Unity. It's absolutely the right choice in many situations but it's not without tradeoffs and compromises.
There are also trade offs in flexibility that come with using Unity or Unreal. They make it easy to make games that fit within their paradigms but it's relatively harder to make a game that does something radical or experimental with rendering or physics.
I say this as someone who lives in Unity. It's absolutely the right choice in many situations but it's not without tradeoffs and compromises.
I love the productivity Unity brings, but sometimes I feel like making a game in the engine requires extra polish as I need to make sure the game has character and not “a Unity game”.
I've found that the polish step is necessary no matter what engine you use, or even if you roll your own.
It is a cliche of game development that a game that is 75% there still has 90% of the work left to complete it to high quality.
It is a cliche of all software development, and very true in my experience.
Is it a cliche if it's true? The difference between some slapped together product and one that feels decent is layer after layer of detail and polish. Long after you feel that you've put forth a reasonable effort, you still have half of the development left to go.
I 100% agree, although I feel like I get better value for my polish effort using a different toolset.
Another reason is change of generations.
Many old style AAA developers see using off the shelf middleware as heresy.
Many old style AAA developers see using off the shelf middleware as heresy.
This has not been my experience.
Game developers I've worked with have generally been pragmatic and willing to use the best tool for the job.
Also triple AAA isn't that old! There's not a generation retiring quite yet :) People who want to still be in the industry are. I've seen the industry get older on average as I've worked in it.
Forms of middleware have been there since the early days of AAA style games - sound, video encoding etc. such as RAD Game Tools
Also triple AAA isn't that old! There's not a generation retiring quite yet :) People who want to still be in the industry are. I've seen the industry get older on average as I've worked in it.
Forms of middleware have been there since the early days of AAA style games - sound, video encoding etc. such as RAD Game Tools
Just speaking of the regular reasons I used to see on Gamasutra or GDC talks about creating the own engine in-house instead of commercial middleware
If I recall correctly, Tribes was one of the first ones to have some moderate adoption.
If I recall correctly, Tribes was one of the first ones to have some moderate adoption.
Can you give some examples? I had worked on many games and a licensed engine had been always evaluated. Some went with one, some did not for technical and economical reasons. Never seen people just outright declining it because it's "heresy".
Things were already moving in that direction though. Many games used Doom / Quake / etc engines in the 90s
I think the turning point came when the work to write a new engine became greater than the work to use an off the shelf and make it your own. And that only came about as people's expectations for physics, lighting, etc got pushed ever more.
I think the turning point came when the work to write a new engine became greater than the work to use an off the shelf and make it your own. And that only came about as people's expectations for physics, lighting, etc got pushed ever more.
We've never met, but I believe you owe me about $17,000 in quarters, with interest. :)
Now if I can only hunt down those bastards who made Tempest, Qix, and Galaga...
Now if I can only hunt down those bastards who made Tempest, Qix, and Galaga...
I don't, because it is not that Battlezone. Sorry.
I understand you're joking, but why would you be owed quarters? Did you not enjoy the games!
Jeff Minter posts regularly on Taitter.
Qix... I need to play that again now...
Was this the Pandemic Studios version of Battlezone (i.e. Battlezone II) or the first version of Battlezone for PC? If you had a hand in either, then I owe you a beer (or whatever you prefer) a bunch of times over because I loved both of those games so much. Endless hours of entertainment!
I worked on both. Thanks. If we ever run into each other I hope we can have a drink.
Oh man thanks for making those. I spent so much time as a kid playing a copy of Battlezone I checked out from the local library. Love me some biometal
Thank you! The physical model of the 'hovercrafts' in Battlezone 2 was so satisfying, I just enjoyed piloting over the landscape so much and no game has come this close to that feeling!
Hover physics was literally the very first thing I worked on for Battlezone, back when we were working in Interstate '76 and the game was called "Badlands". Within a week or so I had Groove Champion's orange Packard Piranha hovering around. There were many changes and additions later, but the core of hover physics started there.
I'm getting little pangs of joy just remembering battlezone. Such a wonderful game, and it still holds up.
I worked on both of them too, along with the unofficial patches and the remastered versions Battlezone 98 Redux and Battlezone Combat Commander. 20 years of Battlezone!
fancy seeing you here. I loved battlezone as well, though I just played the arcade version (at my local thrift store)
Thank you for making such a great game! Battlezone was one of my favorite games during my childhood. I loved the hybrid RTS/FPS aspect.
Yes. I'm not sure that is a good thing for the consumer though, and has led to the market being flooded by rubbish. It is one thing to produce a game. Another, to produce a good game.
Both ends of the market have been driven wildly off course. The major publishers have become so risk averse that the only games they greenlight are iterative skinner box perpetual loot box cash grab wastes of talent.
And on the low end you have the Unity asset store run amok.
I'm not sure how you do it, but there is a happy middle of reasonable risk with reasonable competency missing in the market. There are few games made there every year, but they are a paltry few next to the regurgitated annual CoD game or the thousand random shovelwares put on Steam every month.
And on the low end you have the Unity asset store run amok.
I'm not sure how you do it, but there is a happy middle of reasonable risk with reasonable competency missing in the market. There are few games made there every year, but they are a paltry few next to the regurgitated annual CoD game or the thousand random shovelwares put on Steam every month.
[deleted]
It's still good to have engineers on a Unity or Unreal Engine 4 project (especially the latter). Designers can accomplish a lot with visual scripting but for a more complex game you'll often need custom components written by engineers.
So the question would be what type of game could be made with that type of team today?
There are certainly teams of people of that technical caliber today at Blizzard, Riot, Sony Santa Monica (and lots of other places farther from me) making amazing, cutting edge game tech. But there are also teams like the ones the author described that don't have amazing tech skills that still can make games we couldn't dream of thanks to game engines. I think it is exciting.
What's Blizzard done that's technically impressive as of late? They scrapped Titan, and Overwatch isn't particularly groundbreaking from what I can tell.
Blizzard has never made anything truly groundbreaking, it's mostly re using existing ideas, with a high level of polish.
This is a sad comment that reeks of the gaming community where nobody creates anything so they've no grasp on the feats that these companies have been delivering over and over again for decades.
It's a genuine question; their most recent games have been an FPS, a MOBA, and a card game. I can't tell what "cutting edge game tech" that they happen to be creating with them.
Good games? Depending on your outlook, yeah. But there's nothing really...different? About them?
Their anticheat, maybe? It's a bit impressive.
Good games? Depending on your outlook, yeah. But there's nothing really...different? About them?
Their anticheat, maybe? It's a bit impressive.
There are still great people working in the games industry, but it's larger and segmented. These days you can be an engine programmer working on e.g. Frostbite or id Tech for EA and ZeniMax, respectively. These have resulted in games like Dragon Age: Inquisition and Doom (2016). Dragon Age: Inquisition required a lot of engine work because Frostbite was originally designed for first-person shooters like Battlefield, and Doom is a bit of a technological marvel, and it's kind of a showcase piece for id Tech 6.
This article is mind expanding if you want to understand what happens on just the graphics side of things in a modern game: http://www.adriancourreges.com/blog/2016/09/09/doom-2016-gra...
Bear in mind everything that's not included in this analysis: AI, mesh deformation, level logic, player mechanics, enemy mechanics, all the lower level stuff and so many things I haven't even thought of.
Bear in mind everything that's not included in this analysis: AI, mesh deformation, level logic, player mechanics, enemy mechanics, all the lower level stuff and so many things I haven't even thought of.
This is one reason why the simple mindset of just using OpenGL or Vulkan would solve all Linux gamming problems doesn't work.
Battlezone and Battlezone 2 were the first two games I worked on professionally so they'll always be special to me. That's part of why I ended up supporting them unofficially for almost 15 years after release. :D
> Unity and Unreal have made things we couldn't have dreamed of possible.
And that is why nearly 95% FPS games are practically clone of each other.
And that is why nearly 95% FPS games are practically clone of each other.
Oh hi co-worker of my old friend Ken :)
OMG: Ken is one of the most amazing people I have ever worked with. You are lucky to count him as a friend.
I haven't seen him in person in years, what with me living in Seattle these days and him being in Austin. But yeah, he's a pretty good guy!
Golem reporting.
> Unity and Unreal have made things we couldn't have dreamed of possible.
Such as mountains of bloatware/shovelware template games clogging up every single online store and platform.
Such as mountains of bloatware/shovelware template games clogging up every single online store and platform.
Every single person I've ever heard talk about Unity or Unreal in such an aggressive way was completely clueless about software development. They all have high hopes of making the best game/engine the world has ever seen yet are stuck in the planning phase. Either that or they're gullible gamers who watched too many youtubers.
The reality is, every single engineer who built the very optimized engines you my have in mind (that power a few specific games at most) have great respect for Unity and Unreal.
Game development is a small world, people know each other across companies, share tech stuff on Twitter and go to the same conferences.
The reality is, every single engineer who built the very optimized engines you my have in mind (that power a few specific games at most) have great respect for Unity and Unreal.
Game development is a small world, people know each other across companies, share tech stuff on Twitter and go to the same conferences.
Shall we go back to the old days, then? Because there were no crap games then.
The solution is better curation and discoverability. Not harsher gate-keepers and horribly difficult development.
So many incredible games wouldn't exist without modern development tools. Just because you were mildly inconvenienced the last you time browsed an app store, don't wish it all away.
The solution is better curation and discoverability. Not harsher gate-keepers and horribly difficult development.
So many incredible games wouldn't exist without modern development tools. Just because you were mildly inconvenienced the last you time browsed an app store, don't wish it all away.
There has always been mountains of books and huge libraries of music being created and we have always managed. Find people you trust to recommend things to you. I recommend Tom Chick at Quarter to Three.
I've never understood why there's so much rage about crappy games on steam. I certainly wouldn't have the gall to charge money for a unity tutorial game, but I absolutely love some of the insane games people like zachtronics make. If we want to go back to publishers controlling access, who in their right mind would approve Schenzen IO? There's a lot of crappy products on EBay as well, but that doesn't mean the platform is useless.
>I've never understood why there's so much rage about crappy games on steam.
It's because there are so many crappy games on Steam that it makes it all but impossible for indie developers with decent games to be discovered. The perception that Steam is a cesspool of con-artists and incompetents depresses the perceived value of any game on the platform not released by a known publisher, meaning any independent game is more likely to fail merely by being associated with Steam to begin with. Also because the last time the video game market was drowning in mediocrity, it crashed.
I mean... when I was in high school I used to doodle Mario levels on my schoolbook jackets because I was just that kind of nerd, and now I live in the glorious future of Unity and Godot and Game Maker and C++ and SDL2 if I want to be a masochist. It's fantastic that anyone can more or less make something that can at least theoretically be called a "game." But that doesn't mean all of it deserves to be on the open market.
>If we want to go back to publishers controlling access, who in their right mind would approve Schenzen IO?
You're right, if it were up to publishers no one would be making anything but MOBAs and first person shooters and maybe Candy Crush clones. Independent game development allows niche and non-profitable genres to thrive. However, again, it's still a problem that the sheer bulk of garbage on these platforms makes it difficult to find that good content without some kind of gatekeeping.
It's because there are so many crappy games on Steam that it makes it all but impossible for indie developers with decent games to be discovered. The perception that Steam is a cesspool of con-artists and incompetents depresses the perceived value of any game on the platform not released by a known publisher, meaning any independent game is more likely to fail merely by being associated with Steam to begin with. Also because the last time the video game market was drowning in mediocrity, it crashed.
I mean... when I was in high school I used to doodle Mario levels on my schoolbook jackets because I was just that kind of nerd, and now I live in the glorious future of Unity and Godot and Game Maker and C++ and SDL2 if I want to be a masochist. It's fantastic that anyone can more or less make something that can at least theoretically be called a "game." But that doesn't mean all of it deserves to be on the open market.
>If we want to go back to publishers controlling access, who in their right mind would approve Schenzen IO?
You're right, if it were up to publishers no one would be making anything but MOBAs and first person shooters and maybe Candy Crush clones. Independent game development allows niche and non-profitable genres to thrive. However, again, it's still a problem that the sheer bulk of garbage on these platforms makes it difficult to find that good content without some kind of gatekeeping.
Similarly, it is not because one can play a guitar that they are entitled to turn into musician, or writing something doesn't turn one into a writer.
There is crap in all kinds of artistic work, for some reason devs seem to think they should be special and get everything for free.
There is crap in all kinds of artistic work, for some reason devs seem to think they should be special and get everything for free.
> Similarly, it is not because one can play a guitar that they are entitled to turn into musician, or writing something doesn't turn one into a writer.
I disagree.. playing a guitar does make one a musician, and writing does make one a writer. What either is not entitled to is success in the marketplace, but then, no one is, regardless of skill. Once you put your work into the marketplace, the market decides its value, and your value by proxy. But for that to work successfully, the marketplace has to efficiently sort for quality, and one way to enforce quality is having the barriers to entry be sufficiently high. In practice, that doesn't often work because entrants don't want to be sorted efficiently, they want to be sorted to the top to maximize exposure and revenue. It's why SEO exists - to optimize the Googlebot for you and you alone by any means necessary. Unfortunately, Steam also wants to maximize the size of its library and revenue, so it has an incentive to engineer a marketplace that favors quantity over quality.
But even amateurs put in more effort than the developers publishing much of the garbage onto Steam - using the term "developers" loosely. These are incoherent rage games or weekend projects, barely functional tech demos and tutorial games, asset flips, etc. Games that don't even run. If the bar for quality on Steam was merely "amateur" then I would be happy, but it isn't. There seems to be no bar, and that makes it bad for everyone, except maybe Valve.
I disagree.. playing a guitar does make one a musician, and writing does make one a writer. What either is not entitled to is success in the marketplace, but then, no one is, regardless of skill. Once you put your work into the marketplace, the market decides its value, and your value by proxy. But for that to work successfully, the marketplace has to efficiently sort for quality, and one way to enforce quality is having the barriers to entry be sufficiently high. In practice, that doesn't often work because entrants don't want to be sorted efficiently, they want to be sorted to the top to maximize exposure and revenue. It's why SEO exists - to optimize the Googlebot for you and you alone by any means necessary. Unfortunately, Steam also wants to maximize the size of its library and revenue, so it has an incentive to engineer a marketplace that favors quantity over quality.
But even amateurs put in more effort than the developers publishing much of the garbage onto Steam - using the term "developers" loosely. These are incoherent rage games or weekend projects, barely functional tech demos and tutorial games, asset flips, etc. Games that don't even run. If the bar for quality on Steam was merely "amateur" then I would be happy, but it isn't. There seems to be no bar, and that makes it bad for everyone, except maybe Valve.
If you're going to "get discovered", in any industry, you're doing to be very disappointed. Marketing, just like engineering, shouldn't be an afterthought.
Tom's alright; his forums are a mess, though. I find Three Moves Ahead to encapsulate a similar viewpoint with a lot less gross around it.
Unity and Unreal did not exist when 1983 crash happened.
I had tons of floppies and CDs with shovelware for ZX Spectrum, MS-DOS, Amiga and Windows 3.x, 9x, Linux,....
All before Unreal or Unity ever happened.
I had tons of floppies and CDs with shovelware for ZX Spectrum, MS-DOS, Amiga and Windows 3.x, 9x, Linux,....
All before Unreal or Unity ever happened.
I grew up in the 90s, and still have a big mountain of CDs full of shovelware my well meaning mom picked up.
FTA, a take on entry/mid/senior level distinctions I haven't heard before:
"If you are the guy asking for help from people below, to your side, and above you, you’re probably an entry-level engineer. If you frequently need support from those above you (excluding C-level and executive support, that my friends are a whole new level of hell), there’s a good chance you are a mid-level engineer. If you find yourself supporting and being supported by those at your side, you are likely a senior or lead level engineer."
"If you are the guy asking for help from people below, to your side, and above you, you’re probably an entry-level engineer. If you frequently need support from those above you (excluding C-level and executive support, that my friends are a whole new level of hell), there’s a good chance you are a mid-level engineer. If you find yourself supporting and being supported by those at your side, you are likely a senior or lead level engineer."
In fact it's very accurate. This is what distinguishes a true senior/10x engineer from a "ninja" or a "rockstar"; they are force mulitpliers and they make everybody around them 10x more productive.
This also suggests what C-levels are supposed to be doing: providing the vision and culture that actually fosters this kind of mutual web of support.
This also suggests what C-levels are supposed to be doing: providing the vision and culture that actually fosters this kind of mutual web of support.
I've heard a similar thing but phrased as what frustrates you in you work most. It's a bit more cynical.
Basically: If you're frustrated by your work... probably junior. If you're frustrated by your colleagues... probably mid-level. If you're frustrated by too much management... probably senior. If you're frustrated by too little management... probably lead.
Basically: If you're frustrated by your work... probably junior. If you're frustrated by your colleagues... probably mid-level. If you're frustrated by too much management... probably senior. If you're frustrated by too little management... probably lead.
I think about it in relation to how much responsibility each person in the team is comfortable taking.
If you're responsible to solve tickets, you're probably junior. If you're responsible for implementing features, mid. If you're responsible for making sure the whole project will work in time for delivery, senior. And if you're responsible for making sure your team can deliver the project on time, lead.
If you're responsible to solve tickets, you're probably junior. If you're responsible for implementing features, mid. If you're responsible for making sure the whole project will work in time for delivery, senior. And if you're responsible for making sure your team can deliver the project on time, lead.
"Usually an outstanding problem or issue is already known by your staff, and often they have tried to make you aware of this problem, but usually it gets prioritized into nothingness, or filtered into white noise by your chain of command where no action gets done."
This is the same as being a consultant. You come on site, you listen to everyone and make notes of it, and then you make a presentation and plan for direction.
They pay you way more than their own people, to tell you what your own people are telling you. It's quite crazy.
This is the same as being a consultant. You come on site, you listen to everyone and make notes of it, and then you make a presentation and plan for direction.
They pay you way more than their own people, to tell you what your own people are telling you. It's quite crazy.
As somebody who writes code for games for a living, I found this article fascinating. Something that was coming up in my mind again and again as I was reading through it is the question of who has the power to affect the necessary changes. That seems to be the crux of a lot of the issues; is the studio leadership wrongminded, or have they put the wrong people at the gate?
The parable of the producer ignoring engineering specialities when assigning tickets seemed especially gregarious. Why was that person unable to learn, even after the engineers raised the issue? Why were they still there? Were they friendly with the higher-ups, or what? Why was there no will to improve the process until external help had to be brought on?
It also reminds me of stories I had heard from inside Microsoft in the pre-Nadella era. Tales of power brokers and entrenched lifers who could make things easy or impossible on a whim. What is the nature of the organisations that allow people like these to take root?
It's funny how small scale engineering problems are rooted in code and large scale engineering problems are rooted in people.
The parable of the producer ignoring engineering specialities when assigning tickets seemed especially gregarious. Why was that person unable to learn, even after the engineers raised the issue? Why were they still there? Were they friendly with the higher-ups, or what? Why was there no will to improve the process until external help had to be brought on?
It also reminds me of stories I had heard from inside Microsoft in the pre-Nadella era. Tales of power brokers and entrenched lifers who could make things easy or impossible on a whim. What is the nature of the organisations that allow people like these to take root?
It's funny how small scale engineering problems are rooted in code and large scale engineering problems are rooted in people.
It also reminds me of stories I had heard from inside Microsoft in the pre-Nadella era. Tales of power brokers and entrenched lifers who could make things easy or impossible on a whim. What is the nature of the organisations that allow people like these to take root?
At some point when a company gets large enough the actual product it produces stops being something sold outside the company and instead the product becomes one’s increased status within the organization and that of their immediate manager. Status seekers who contribute less technically but spend all their time producing “good optics” for the people above and selling/exaggerating their achievements get promoted a lot more, even if the good optics were achieved by doing something that makes the product less sellable. Eventually status seekers get promoted into management and then promote upward like people who are also status seekers.
The people who spend all their time doing the heaviest engineering aren’t often spending much time selling their achievements to busy executives who don’t have time to look at their accomplishments at let their work speak for itself. Which is why they tend to stay in the lower levels of the company.
At some point when a company gets large enough the actual product it produces stops being something sold outside the company and instead the product becomes one’s increased status within the organization and that of their immediate manager. Status seekers who contribute less technically but spend all their time producing “good optics” for the people above and selling/exaggerating their achievements get promoted a lot more, even if the good optics were achieved by doing something that makes the product less sellable. Eventually status seekers get promoted into management and then promote upward like people who are also status seekers.
The people who spend all their time doing the heaviest engineering aren’t often spending much time selling their achievements to busy executives who don’t have time to look at their accomplishments at let their work speak for itself. Which is why they tend to stay in the lower levels of the company.
tangentially, the "blueprint spaghetti" link is pretty great:
https://blueprintsfromhell.tumblr.com/
at a former job i knew a team of devs who built and maintained a framework to help teams of researchers put together pipelines of simulation (often fluid simulation), animation, rendering, etc. This framework had its own blueprinting functionality.
One of the cute features they had was nested blueprints -- e.g. you could define some blueprint, and abstract it behind an interface as its own component, then re-use the component in another blueprint.
So i guess in a "blueprints from hell" scenario that lets you have "fractal blueprints from hell".
edit:
i'm also reminded of Chris Domas' defcon talk "Psychological Warfare in Reverse Engineering", where he sets out to construct binaries that cause despair when someone tries to reverse engineer them -- in particular he asks the question: "could i make a binary that draws a particular picture when someone views the binary's control flow graph in IDA"
https://www.youtube.com/watch?v=HlUe0TUHOIc
https://blueprintsfromhell.tumblr.com/
at a former job i knew a team of devs who built and maintained a framework to help teams of researchers put together pipelines of simulation (often fluid simulation), animation, rendering, etc. This framework had its own blueprinting functionality.
One of the cute features they had was nested blueprints -- e.g. you could define some blueprint, and abstract it behind an interface as its own component, then re-use the component in another blueprint.
So i guess in a "blueprints from hell" scenario that lets you have "fractal blueprints from hell".
edit:
i'm also reminded of Chris Domas' defcon talk "Psychological Warfare in Reverse Engineering", where he sets out to construct binaries that cause despair when someone tries to reverse engineer them -- in particular he asks the question: "could i make a binary that draws a particular picture when someone views the binary's control flow graph in IDA"
https://www.youtube.com/watch?v=HlUe0TUHOIc
Good article, but I'm not sure who it is targeted at. Companies/teams that do bad things the author recommends against (or don't do good things he recommends) are usually ran by people who either don't care or are pathologically clueless. They're not going to read this. If they do, they are not going to change the right thing. They will instead go to some business conference, listen to overhyped process consultants and force-implement more "agile" techniques that have nothing to do with what actually hurts their productivity.
Maybe, strip out most of the game dev stuff and write a small book? That might help. I don't know why, but books work on managers much better than articles. At least I can give or recommend someone a book without making it seem like an insult. "Hey, I've read this book by an engineer who worked on many large game projects. You should check it out."
Maybe, strip out most of the game dev stuff and write a small book? That might help. I don't know why, but books work on managers much better than articles. At least I can give or recommend someone a book without making it seem like an insult. "Hey, I've read this book by an engineer who worked on many large game projects. You should check it out."
You're right. I wrote this article knowing this, which is what I tried opening with.
I'm not qualified to write a book however because this article itself is proof my writing skills need work.
Otherwise I'd be able to be more clear.
I'm not qualified to write a book however because this article itself is proof my writing skills need work.
Otherwise I'd be able to be more clear.
All writ{ers,ing} {are,is} emotionally soluble. And sometimes you need to unload before you can think clearly and articulate with perfect moderation. It's fine.
The article and writing was very good. The rant sections could have devolved several orders of magnitude further, that wasn't the end of the world IMO.
I got some major general-worldview TILs about engineering from the parts that talked about experience levels.
The bits you said could be their own articles, and/or which you said you could go on about... I hope to read about those in followup posts. :D
The article and writing was very good. The rant sections could have devolved several orders of magnitude further, that wasn't the end of the world IMO.
I got some major general-worldview TILs about engineering from the parts that talked about experience levels.
The bits you said could be their own articles, and/or which you said you could go on about... I hope to read about those in followup posts. :D
Are you kidding? It was a great article.
Agreed - great article. That said, I think most authors of books like the one we're all imagining have support (in the form of friends and colleagues) in addition to an editor. I don't want to pressure you to write a book, but please don't let your own criticism of your writing abilities get in the way of a book.
I’d help in whatever way you’d like. My dream is to make a company engineers love to work in. The chance to learn from whatever you share would be amazing.
> Overtaxing your engineering effort is a surefire way to end up with hard-to-maintain work. Unless you are developing a throwaway prototype, hard-to-maintain work is one of the fastest ways to cause engineering fatigue
Sadly this was still all par-for-the-course back when I was in the industry and all my old contacts still affirm that this is the case today across most studios. Easier to burn people, ship something and then re-hire/train a new batch.
It's a shame the incentives of development aren't aligned with reasonable employment, the industry would be a lot better for it.
Sadly this was still all par-for-the-course back when I was in the industry and all my old contacts still affirm that this is the case today across most studios. Easier to burn people, ship something and then re-hire/train a new batch.
It's a shame the incentives of development aren't aligned with reasonable employment, the industry would be a lot better for it.
> They immediately asked who was giving up this information and wanted to crack down on this perceived insubordination, rather than trying to address the issue. I was then dismissed for not specifying names, and my services were no longer required.
Wow! That sounds terrible.
Wow! That sounds terrible.
This is horribly routine in all hierarchical organisations across the world; those that manage to escape from it usually become runaway successes.
Fixing the blame is easier then fixing the problem.
If that company is still around, it's worth naming and shaming.
Great article that reminded me of John Cleese’s wonderful talk about creativity.
> So, here's how to stamp out creativity in the rest of the organization and get a bit of respect going.
> One: Allow subordinates no humour, it threatens your self-importance and especially your omniscience. Treat all humour as frivolous or subversive.
> Because subversive is, of course, what humour will be in your setup, as it's the only way that people can express their opposition, since (if they express it openly) you're down on them like a ton of bricks.
> So let's get this clear: blame humour for the resistance that your way of working creates. Then you don't have to blame your way of working. This is important. And I mean that solemnly. Your dignity is no laughing matter.
> Second: keeping ourselves feeling irreplaceable involves cutting everybody else down to size, so don't miss an opportunity to undermine your employees' confidence.
> A perfect opportunity comes when you're reviewing work that they've done. Use your authority to zero in immediately on all the things you can find wrong. Never never balance the negatives with positives, only criticize, just as your school teachers did.
> Always remember: praise makes people uppity.
> Third: Demand that people should always be actively doing things. If you catch anyone pondering, accuse them of laziness and/or indecision. This is to starve employees of thinking time because that leads to creativity and insurrection. So demand urgency at all times, use lots of fighting talk and war analogies, and establish a permanent atmosphere of stress, of breathless anxiety, and crisis.
> In a phrase: keep that mode closed.
> In this way we no-nonsense types can be sure that the tiny, tiny, microscopic quantity of creativity in our organization will all be ours!
> But! Let your vigilance slip for one moment, and you could find yourself surrounded by happy, enthusiastic, and creative people whom you might never be able to completely control ever again!
> So be careful.
https://www.youtube.com/watch?v=Pb5oIIPO62g
https://www.brainpickings.org/2012/04/12/john-cleese-on-crea...
> So, here's how to stamp out creativity in the rest of the organization and get a bit of respect going.
> One: Allow subordinates no humour, it threatens your self-importance and especially your omniscience. Treat all humour as frivolous or subversive.
> Because subversive is, of course, what humour will be in your setup, as it's the only way that people can express their opposition, since (if they express it openly) you're down on them like a ton of bricks.
> So let's get this clear: blame humour for the resistance that your way of working creates. Then you don't have to blame your way of working. This is important. And I mean that solemnly. Your dignity is no laughing matter.
> Second: keeping ourselves feeling irreplaceable involves cutting everybody else down to size, so don't miss an opportunity to undermine your employees' confidence.
> A perfect opportunity comes when you're reviewing work that they've done. Use your authority to zero in immediately on all the things you can find wrong. Never never balance the negatives with positives, only criticize, just as your school teachers did.
> Always remember: praise makes people uppity.
> Third: Demand that people should always be actively doing things. If you catch anyone pondering, accuse them of laziness and/or indecision. This is to starve employees of thinking time because that leads to creativity and insurrection. So demand urgency at all times, use lots of fighting talk and war analogies, and establish a permanent atmosphere of stress, of breathless anxiety, and crisis.
> In a phrase: keep that mode closed.
> In this way we no-nonsense types can be sure that the tiny, tiny, microscopic quantity of creativity in our organization will all be ours!
> But! Let your vigilance slip for one moment, and you could find yourself surrounded by happy, enthusiastic, and creative people whom you might never be able to completely control ever again!
> So be careful.
https://www.youtube.com/watch?v=Pb5oIIPO62g
https://www.brainpickings.org/2012/04/12/john-cleese-on-crea...
This was a great article and I recognize most of the issues described but I feel it doesn’t criticise leadership enough (even though it does somewhat). This is the biggest failure in the industry. A lot
of the best people are too busy putting in the hard work to be put in leadership roles due to mismanagement from the current leadership. Why is crunch always a thing in games dev? It’s poor planning that endures for years and years.
Good article and I agree with most points, however I am not sure I like the articles tone. Maybe I am a precious butterfly but it was a bit "all-knowing", with too much emphasis on how you save the day. I am sure your phone will start ringing after publications too :) Thanks anyway though, I read it :)
I don't understand why Unreal Engine 4 is in the title. It feels click-baity since you're expecting to read something about the "dark side" of UE4 or something.
First sentence of the article: "I have been working professionally with Unreal Engine for over 9 years."
Second paragraph: "If a game company in Los Angeles is having an Unreal Engine 4 problem that no one can solve, I usually end up getting that call. I am writing this article to explain why I get that call, how one would prevent the need for that call, and what I normally do after getting that call."
So while a lot of these things are pretty useful strategies for any large project (we had asset naming conventions for parts of the Flash cartoons I was working on around the turn of the century), they are all common problems he's encountered in his career of being That Guy You Call When Your Big Unreal Project Is Broken.
Second paragraph: "If a game company in Los Angeles is having an Unreal Engine 4 problem that no one can solve, I usually end up getting that call. I am writing this article to explain why I get that call, how one would prevent the need for that call, and what I normally do after getting that call."
So while a lot of these things are pretty useful strategies for any large project (we had asset naming conventions for parts of the Flash cartoons I was working on around the turn of the century), they are all common problems he's encountered in his career of being That Guy You Call When Your Big Unreal Project Is Broken.
Yea I read the article >.>
I still read it as him trying to suggest what better practices you can adopt specifically pertaining to UE4 that would "prevent the need for that call"
I still read it as him trying to suggest what better practices you can adopt specifically pertaining to UE4 that would "prevent the need for that call"
Have someone who has a clue how to design and integrate complex systems from a high level on your team. Seems to be the point of the whole article.
I think it is because UE4 (and Unity) make it possible to get 50-80% there on projects without solid engineering. So people think they can get by without having the skills on the team.
I think most people with UE4 experience or running a studio know that you need solid programmers working on your project. The author is saying that even with that, project ambitions still exceed team bandwidth/capability.
Wow, this hits way too close to home. On the last few projects I've been hired as a lead Unity Dev, my main job was to untangle the mess left by previous maintainers and finally put some good engineering practices in place. Project I'm working on right on had it's multiplayer "just added" on top of single player gameplay mechanics, and now my team finds itself in the middle of the complete architectural rewrite instead of making any new features.
Most of this just sounds like standard software engineering best-practices, though from what I've gathered it seems like standard SE challenges get multiplied five-fold in game development. It takes an engineer braver than I to work at a triple-A game studio.
About this:
“responsibilities than they can handle, and development suffers from this person being fatigued, or worse, leaving altogether. Neither the engineer nor team can grow because of this, as new features or changes get increasingly more resource-demanding to implement”
This is said about mid-level engineers who burn out, but I think it happens to senior level engineers as well.
“responsibilities than they can handle, and development suffers from this person being fatigued, or worse, leaving altogether. Neither the engineer nor team can grow because of this, as new features or changes get increasingly more resource-demanding to implement”
This is said about mid-level engineers who burn out, but I think it happens to senior level engineers as well.
One of the most common things I’ve seen over and over is under powered hardware. It’s so cheap in comparison but since it’s easy for an accountant to total compared to lost time it’s very frequently done. It’s not a solve all but in so many cases it would make a significant difference ignoring all else.
"If you're reading this as an engineer (but this may apply to many fields), or as someone looking to grow an engineer, the most important thing you can do is identify current issues and weaknesses."
I learned a very different things, but thank you nonetheless: there are companies that care about these things! I just got a full-time job that is basically like this, a hell on Earth (I've been freelancing before). So the lesson I learned is, I should be looking for a new job :)
My coworker (we are 2 in the team) is already jumping ship even when he joined later than me.
I learned a very different things, but thank you nonetheless: there are companies that care about these things! I just got a full-time job that is basically like this, a hell on Earth (I've been freelancing before). So the lesson I learned is, I should be looking for a new job :)
My coworker (we are 2 in the team) is already jumping ship even when he joined later than me.
Since I've lived, almost to the letter, a few times many of the scenarios detailed here, I cannot upvote more this post.
My question now is: is there any company mid-big size not guilty of those sins? Does it even exist or it is an utopia wished but not achievable? (just asking)
My question now is: is there any company mid-big size not guilty of those sins? Does it even exist or it is an utopia wished but not achievable? (just asking)
When I got the call to fix this dilemma literally a couple of weeks before release, the simple act of cleaning up the source control server, and purging these unneeded cache files from source control, fixed this audio issue that was plaguing them for months and had costed the company tons of money and time.
This sort of thing drives me insane (or at least used to; my current colleagues, from junior to senior, are all actually pretty good). This is something that everyone knows, just by looking at it, is a bad thing. Everyone who saw this must have known it wasn't a great idea, even if they didn't know why. Even without being able to elucidate why, everyone (should) get the feeling that having these things in the repository is asking for trouble; I'd expect a junior to have at least a bad feeling about it, and a senior to be able to specify what kind of problems it could cause. I don't know about the exact circumstances, of course, but it's even surprising that nobody found this problem while they were investigating. I've certainly gone as far as "are we definitely using the exact same binary files on the different systems?" when chasing odd problems that only manifest on some installations and not others, so what went so wrong that nobody got that far in their debugging?
So unless the company in question had by chance a room full of people with this particular massive blind spot, people saw this happening and they knew it was a bad idea and they didn't do anything about it. Culture failure. I don't know why. Maybe they were afraid to mention bad things because pointing out possible problems is seen as not being a team player. Maybe they were just so sick of it all they didn't care anymore. Maybe there were a couple of forcefully strong people in charge of the repository who saw any question as a personal attack and people were just sick of dealing with the egotistical prima donnas. Maybe the process for doing anything about it was laughable painful and went through someone who would shoot it down on the grounds that we're so busy firefighting things that are broken and you want to waste time tinkering with something that looks fine to me?
If Allar is reading, could you say a few words about this? How many times do you get called in to fix something that could have been averted with a decent programming culture, even if the specific technical chops weren't there? We've all seen teams with good culture deliver software that they really didn't know how to build when they started through good culture, and we've all seen teams of people who all know what they're doing deliver a flaming wreck because the culture was a disaster. How often are you called in to deal with something that the existing team really couldn't have foreseen without some particular piece of specific knowledge or experience, and how often is it something that they could have dealt with months ago if they had just been allowed and guided to work sensibly and professionally?
This sort of thing drives me insane (or at least used to; my current colleagues, from junior to senior, are all actually pretty good). This is something that everyone knows, just by looking at it, is a bad thing. Everyone who saw this must have known it wasn't a great idea, even if they didn't know why. Even without being able to elucidate why, everyone (should) get the feeling that having these things in the repository is asking for trouble; I'd expect a junior to have at least a bad feeling about it, and a senior to be able to specify what kind of problems it could cause. I don't know about the exact circumstances, of course, but it's even surprising that nobody found this problem while they were investigating. I've certainly gone as far as "are we definitely using the exact same binary files on the different systems?" when chasing odd problems that only manifest on some installations and not others, so what went so wrong that nobody got that far in their debugging?
So unless the company in question had by chance a room full of people with this particular massive blind spot, people saw this happening and they knew it was a bad idea and they didn't do anything about it. Culture failure. I don't know why. Maybe they were afraid to mention bad things because pointing out possible problems is seen as not being a team player. Maybe they were just so sick of it all they didn't care anymore. Maybe there were a couple of forcefully strong people in charge of the repository who saw any question as a personal attack and people were just sick of dealing with the egotistical prima donnas. Maybe the process for doing anything about it was laughable painful and went through someone who would shoot it down on the grounds that we're so busy firefighting things that are broken and you want to waste time tinkering with something that looks fine to me?
If Allar is reading, could you say a few words about this? How many times do you get called in to fix something that could have been averted with a decent programming culture, even if the specific technical chops weren't there? We've all seen teams with good culture deliver software that they really didn't know how to build when they started through good culture, and we've all seen teams of people who all know what they're doing deliver a flaming wreck because the culture was a disaster. How often are you called in to deal with something that the existing team really couldn't have foreseen without some particular piece of specific knowledge or experience, and how often is it something that they could have dealt with months ago if they had just been allowed and guided to work sensibly and professionally?
Literally no engineers were allocated to the issue and majority of the project.
20% of the time I work on novel unforseeable issues. R&D and future tech stuff.
80% of the time its culture, professionalism, and generally "are you fucking serious" issues.
I've dealt with literally every situation you've described however. Just because this particular reason was that there were no engineers, you could swap in any other situation and still have a true story.
20% of the time I work on novel unforseeable issues. R&D and future tech stuff.
80% of the time its culture, professionalism, and generally "are you fucking serious" issues.
I've dealt with literally every situation you've described however. Just because this particular reason was that there were no engineers, you could swap in any other situation and still have a true story.
Url changed from https://80.lv/articles/confessions-of-an-unreal-engine-4-eng..., which points to this.
I consider the case, which the author describes, where the consultant actually talks to engineers, a happy day scenario. I was witness to a fast growing startup hiring a well known global consulting firm to help them set up company structure and processes. The startup was unable to do this on its own because of lack of leadership and unwillingness of the owners to empower their own people. The consulting firm sent over a couple of their junior consultants who consulted their 3-ring binder of best practices for the nearest matching industry category and gave their recommendations, which were followed quite strictly by a newly hired junior management. All without talking to a single engineer.
More often than not, employees are considered a necessary means to an end and treated as such, with the least amount of reward and care for which they are willing to work. Their sense of professional honor, responsibility and guilt is often exploited in an attempt to achieve higher output. Does the company's products or services suffer because of this? Of course, but nobody at the top cares as long as there is a metric that tells a good story. And when not, the lowest ranks are the first to be pruned and restructured.