Ask HN: Was hired to improve company's devops, founder won't listen to my ideas
77 comments
If the founder is who you report to then you need to schedule a 1-on-1 with them in private and ask them to clarify why they hired you.
If the person you report to is someone else, you need to have this conversation with them first. Then have a follow up meeting with your boss and the founder and clarify your role and position to the founder with your manager.
Just be upfront, and say hey I’m the new guy, and I’m getting pushback, and want to understand my role better so I can contribute my expertise and create value for the team. Not here to step on anyone’s toes, just trying to gather more information as I haven’t been on boarded properly and I’ve been here nearly 3 weeks.
Then let them tell you what they want.
If the response is not what you agreed to when hired and the role they want you to fill is not what you thought then look for another job. Better figure this out now then waste months or years working in a toxic environment. DevOps is one of the most sought after positions in the world right now, don’t settle for being bullied or working in a toxic environment.
At the same time, you need to establish a clear line of communication first. Collect data, and then analyze what the situation is.
Many startups don’t know why they’re hiring people and often aren’t sure why or what they’re hiring for. For example, I got recruited for business development from a company then in the interview realized they weren’t looking for business development. They were looking for an office manager to babysit the developers so they could not be in the office. When I asked about how I would take calls in an open office next to the dev team, and what kind of flexabity I would have to broker deals and deal structures they didn’t even know what I was talking about. The founders basically just wanted an exec assistant with a fancier title.
It’s entirely possible you thought you were hired for devops but they don’t even know what devops is and think you’re just an IT guy or sys admin. That’s why you need to gather more info and figure out if this is something you even want to be involved in.
If the person you report to is someone else, you need to have this conversation with them first. Then have a follow up meeting with your boss and the founder and clarify your role and position to the founder with your manager.
Just be upfront, and say hey I’m the new guy, and I’m getting pushback, and want to understand my role better so I can contribute my expertise and create value for the team. Not here to step on anyone’s toes, just trying to gather more information as I haven’t been on boarded properly and I’ve been here nearly 3 weeks.
Then let them tell you what they want.
If the response is not what you agreed to when hired and the role they want you to fill is not what you thought then look for another job. Better figure this out now then waste months or years working in a toxic environment. DevOps is one of the most sought after positions in the world right now, don’t settle for being bullied or working in a toxic environment.
At the same time, you need to establish a clear line of communication first. Collect data, and then analyze what the situation is.
Many startups don’t know why they’re hiring people and often aren’t sure why or what they’re hiring for. For example, I got recruited for business development from a company then in the interview realized they weren’t looking for business development. They were looking for an office manager to babysit the developers so they could not be in the office. When I asked about how I would take calls in an open office next to the dev team, and what kind of flexabity I would have to broker deals and deal structures they didn’t even know what I was talking about. The founders basically just wanted an exec assistant with a fancier title.
It’s entirely possible you thought you were hired for devops but they don’t even know what devops is and think you’re just an IT guy or sys admin. That’s why you need to gather more info and figure out if this is something you even want to be involved in.
What is the negative consequence of breaking the site for an hour? Is it "some people are a little annoyed", "nobody notices", or "customers start calling and are very upset" or even "customers file lawsuits"?
The amount of friction you add to the development velocity has to be proportioned to the potential downsides. A system that shows doctors xray images in the ER cant go down or people might die. A site like twitter can go down and slightly improve the life of its users. It sounds like you are recommending big company super risk averse solutions at a startup, which is just silly. Besides, staging environments never work, for a long list of reasons. You should be recommending continuous integration and a focus on proper unit tests.
The amount of friction you add to the development velocity has to be proportioned to the potential downsides. A system that shows doctors xray images in the ER cant go down or people might die. A site like twitter can go down and slightly improve the life of its users. It sounds like you are recommending big company super risk averse solutions at a startup, which is just silly. Besides, staging environments never work, for a long list of reasons. You should be recommending continuous integration and a focus on proper unit tests.
Having a staging environment is pretty standard these days. We set one up from day one at our startup. It doesn't really take much effort and your app benefits from being decoupled from the domain. This allows you to run app instances concurrently without relying on domain specific details. It also allows you to physically test changes before pushing to production.
I can hardly see how a staging environment is "recommending big company super risk averse solutions at a startup, which is just silly". A staging environment has never slowed us down; if anything it gives us the confidence to push to master sooner.
I can hardly see how a staging environment is "recommending big company super risk averse solutions at a startup, which is just silly". A staging environment has never slowed us down; if anything it gives us the confidence to push to master sooner.
I think most of the positive statements across this thread are generally correct, and most of the negative statements show a lack of awareness/inexperience:
My experiences are:
1. Staging environments are typically imperfect and have issues
2. Staging environments often help with product stability despite (1.)
3. Having staging envs is common, and could be considered standard in many companies, but..
4. Staging environments are a cost-benefit trade-off and should be considered on a product-by-product case.
5. CI and Unit testing are also common approaches to increasing product stability, and should be subjected to the same cost-benefit analysis per-product (yes, even unit tests), It's not an either-or with staging environments
6. staging envs should be prod-1 but often they end up being (prod-1 + some workarounds + some cruft left over from a failed deployment + other noise) over time.
7. Running a fully prod-equivalent staging environment can be prohibitively expensive (exact replica of prod configuration including infrastructure, realistic copies of prod data at volume, etc..) so usually some compromises are made to give a best-effort simulation of prod (fewer resources, etc..).
8. 'Battle testing' in staging is not always done, it's not uncommon for staging to be largely not looked at, except for whatever automated testing/alerting is running in that environment
9. It's ok for staging environments to test multiple changes at once, the typical intent of a staging area is to identify the presence of problems before they hit prod. The debugging & isolation of what caused problems is a secondary issue.
Your experiences may differ from the above, but they still stand
[edit: formatting]
My experiences are:
1. Staging environments are typically imperfect and have issues
2. Staging environments often help with product stability despite (1.)
3. Having staging envs is common, and could be considered standard in many companies, but..
4. Staging environments are a cost-benefit trade-off and should be considered on a product-by-product case.
5. CI and Unit testing are also common approaches to increasing product stability, and should be subjected to the same cost-benefit analysis per-product (yes, even unit tests), It's not an either-or with staging environments
6. staging envs should be prod-1 but often they end up being (prod-1 + some workarounds + some cruft left over from a failed deployment + other noise) over time.
7. Running a fully prod-equivalent staging environment can be prohibitively expensive (exact replica of prod configuration including infrastructure, realistic copies of prod data at volume, etc..) so usually some compromises are made to give a best-effort simulation of prod (fewer resources, etc..).
8. 'Battle testing' in staging is not always done, it's not uncommon for staging to be largely not looked at, except for whatever automated testing/alerting is running in that environment
9. It's ok for staging environments to test multiple changes at once, the typical intent of a staging area is to identify the presence of problems before they hit prod. The debugging & isolation of what caused problems is a secondary issue.
Your experiences may differ from the above, but they still stand
[edit: formatting]
Good points. Your description is largely similar to my own experiences. On number #6, do you mean that the code base deviates from what is in production? IMO if staging has deviated from production then your pipeline is broken. There is not a single piece of staging specific code in our app. Whatever works on staging works on production. This is accomplished with ENV variables instead of hard coded environment specific details. We actually push directly from staging into production.
> On number #6, do you mean that the code base deviates from what is in production? IMO if staging has deviated from production then your pipeline is broken.
I'm not the person you're replying to, but in my experience, this only happens if you get 100% buy in to the dev ops approach. If you get less than 100% buy in, you'll often end up with greasy environments with the extreme end being a founder type writing code directly in prod.
I'm not the person you're replying to, but in my experience, this only happens if you get 100% buy in to the dev ops approach. If you get less than 100% buy in, you'll often end up with greasy environments with the extreme end being a founder type writing code directly in prod.
> A site like twitter can go down and slightly improve the life of its users.
Ha!
Ha!
> Besides, staging environments never work, for a long list of reasons
Can you go into more detail on this? I would have thought that having a staging environment where all changes get battle-tested by the wider team, before being deployed to prod, would be a good thing.
Can you go into more detail on this? I would have thought that having a staging environment where all changes get battle-tested by the wider team, before being deployed to prod, would be a good thing.
But in that case you have qualified people actually performing battle-testing. If you don't, then a staging area can be more than useless because developers think that making sure their code is bug free is someone else job. Also in most companies testing is deemed inferior to programming, so the least capable is assigned testing duties while the better programmers write the code. Which is wrong because battle-testing software takes just as much skill as developing it.
Unless you can spin up and tear down a staging environment provisioned with data and instrumentation I too don’t believe in it. Without that you’ll be using something shared, where either one thing at a time is subjected to testing or you’re testing multiple things at once which could conflict (including falsely positive) with eachother.
Hope that provides a new perspective. If the team is small, or the product is not susceptible to concurrent development you might not have this phenomenon.
Hope that provides a new perspective. If the team is small, or the product is not susceptible to concurrent development you might not have this phenomenon.
Staging env should be (prod - 1) version. The whole point is to test thing together with other changes. Stage-environment is created to eliminated "works on my machine" problems.
[deleted]
It sounds like you're in a rough spot. Luckily, it sounds like from what you're talking about that there's a lot you can offer and there's a lot of low hanging fruit that can be picked.
Here's what I suggest:
1. Find the easiest things to do that will save people time, or fix problems that people have been talking about. Especially useful if founder is one of the people helped.
2. Once you've found some easy things to do (<1wk), figure out which ones you can do without any help, resources, or authority.
3. Do it, and show it to him, or talk it up with the team and save them some time.
4. Tell the founder you've saved people time, and him money. Invest in some of your next ideas to save more time. Don't ask for permission. Just say you're doing it.
5. Success builds on success.
One thing I'd definitely point out is sometimes you have to go skunkworks. It doesn't sound like you've been there long and they are harping on you to ship something. You likely have plenty of time to just do some easy things, like automating paperwork or forms.
In terms of your staging workflow, again I'd apply concepts that I don't need permission for. Instead of blocking commits to master or even needing a staging branch, you could write some automation that takes a git hash. There's lots of ways to work around needing permissions or official things. Never underestimate the use of a button to do something, rather than typing it into a terminal.
It also sounds like you may have people problems. Is your founder stressed? It sounds like he's having a rough time with scaling growth. Find out what is stressing him out at work and use your talent to eliminate it.
If all else fails, use FIRE. (and quit.)
Good luck.
Here's what I suggest:
1. Find the easiest things to do that will save people time, or fix problems that people have been talking about. Especially useful if founder is one of the people helped.
2. Once you've found some easy things to do (<1wk), figure out which ones you can do without any help, resources, or authority.
3. Do it, and show it to him, or talk it up with the team and save them some time.
4. Tell the founder you've saved people time, and him money. Invest in some of your next ideas to save more time. Don't ask for permission. Just say you're doing it.
5. Success builds on success.
One thing I'd definitely point out is sometimes you have to go skunkworks. It doesn't sound like you've been there long and they are harping on you to ship something. You likely have plenty of time to just do some easy things, like automating paperwork or forms.
In terms of your staging workflow, again I'd apply concepts that I don't need permission for. Instead of blocking commits to master or even needing a staging branch, you could write some automation that takes a git hash. There's lots of ways to work around needing permissions or official things. Never underestimate the use of a button to do something, rather than typing it into a terminal.
It also sounds like you may have people problems. Is your founder stressed? It sounds like he's having a rough time with scaling growth. Find out what is stressing him out at work and use your talent to eliminate it.
If all else fails, use FIRE. (and quit.)
Good luck.
It sounds like you have a different idea of what you were hired to do than the founder does. Talk to the founder and figure out what they want you to be doing.
If your manager is a different person than the founder, your manager should be able to help you resolve this.
Often companies at some stage realize they need better devops procedures, but also realize they can't slow down their pace of progress, and there are tricky tradeoffs to be made. I encourage you to not be insulted when people have concerns - they may just be concerned that you do not yet have all the context about the company as they do, rather than critical of your experience.
If your manager is a different person than the founder, your manager should be able to help you resolve this.
Often companies at some stage realize they need better devops procedures, but also realize they can't slow down their pace of progress, and there are tricky tradeoffs to be made. I encourage you to not be insulted when people have concerns - they may just be concerned that you do not yet have all the context about the company as they do, rather than critical of your experience.
As you said, the "earn trust" doesn't really make sense, so I'd guess there is actually some other reason. Is the way you've implemented the staging plan making life more difficult in some way? Does the founder have some other objective in mind?
Once I worked with a company that had a super smart founder who hired a super smart sales manager. The sales manager quickly noticed that the company did not follow the usual industry practices around quoting projects, and on his own initiative started rolling out a new system, including training sales people, and changing the company's software systems. There was increasing friction with the founder over everything, and eventually the sales manager was let go/quit.
Somehow the sales manager never understood that the core issue the founder had with his plan was that even though 95% of the industry operated with a secretive, price-what-the-customer-can-pay sales process, the whole company had been built over decades on transparent pricing for everyone. It was big part of the trust the customers had for the company and quite a competitive advantage.
Which is all to say that to you, devops means "production doesn't go down", but to the founder, it could mean "Anyone can push to production even faster on more servers." :)
Since the founder has basically said he doesn't trust you, it's likely there's something you are doing that he doesn't like. I hear that you are offended by the founders lack of trust, but being offended is a really good way of missing the actual meaning when he is talking. If things continue as they are, both of you are going to be very unhappy.
Given that there's this disconnect, I'd really spend some time with him to work out what it is that he actually wants -- and when he says he wants something, push deeper to find out the why behind it! Sometimes people ask for a "universal plugin system for our app, so any can extend it" when all they really wanted was "we can translate the app to French".
It's aways possible that the founder is a loon in this area, but I'd say that's only a 10% chance. 80% bet that there is a miscommunication.
Once I worked with a company that had a super smart founder who hired a super smart sales manager. The sales manager quickly noticed that the company did not follow the usual industry practices around quoting projects, and on his own initiative started rolling out a new system, including training sales people, and changing the company's software systems. There was increasing friction with the founder over everything, and eventually the sales manager was let go/quit.
Somehow the sales manager never understood that the core issue the founder had with his plan was that even though 95% of the industry operated with a secretive, price-what-the-customer-can-pay sales process, the whole company had been built over decades on transparent pricing for everyone. It was big part of the trust the customers had for the company and quite a competitive advantage.
Which is all to say that to you, devops means "production doesn't go down", but to the founder, it could mean "Anyone can push to production even faster on more servers." :)
Since the founder has basically said he doesn't trust you, it's likely there's something you are doing that he doesn't like. I hear that you are offended by the founders lack of trust, but being offended is a really good way of missing the actual meaning when he is talking. If things continue as they are, both of you are going to be very unhappy.
Given that there's this disconnect, I'd really spend some time with him to work out what it is that he actually wants -- and when he says he wants something, push deeper to find out the why behind it! Sometimes people ask for a "universal plugin system for our app, so any can extend it" when all they really wanted was "we can translate the app to French".
It's aways possible that the founder is a loon in this area, but I'd say that's only a 10% chance. 80% bet that there is a miscommunication.
The way I've had to explain this to coworkers when dealing with our former manager -- Take what he says, then figure out what he really means, and from there deliver what he needs. But dress what he actually needs into something that looks like what he meant to say, ignoring what he actually said.
A manager that cannot communicate what they need clearly is not a good manager.
An employee who can't understand what a manager is communicating is not a good employee.
More seriously communication is a two way street and problems can occur at either end.
More seriously communication is a two way street and problems can occur at either end.
I disagree.
It is the sender who is in control of the format. You have to ensure you’re sending something that the reciever will understand.
Good receivers probe and negotiate a better format but ultimatly it is the sender who is responcible for the communication.
Between peers I think it is more acceptable to require more negoatiating the format but in a manager -> employee dynamic (or any other power imbalance) it is the managers responcibility to be understood.
Reliquishing that control is disempowering and stunts personal growth IMO.
I think my view point requires a specific belief about peoples ability to do things and grow. Without that I can see how judging others for their ability to understand makes sense.
I believe in people though.
It is the sender who is in control of the format. You have to ensure you’re sending something that the reciever will understand.
Good receivers probe and negotiate a better format but ultimatly it is the sender who is responcible for the communication.
Between peers I think it is more acceptable to require more negoatiating the format but in a manager -> employee dynamic (or any other power imbalance) it is the managers responcibility to be understood.
Reliquishing that control is disempowering and stunts personal growth IMO.
I think my view point requires a specific belief about peoples ability to do things and grow. Without that I can see how judging others for their ability to understand makes sense.
I believe in people though.
> Given that there's this disconnect, I'd really spend some time with him to work out what it is that he actually wants…
This. Spend time with the guy, try to understand where he's coming from. When you do talk I'd suggest you:
- Make it clear that your motives are to help the company. I.e., you both have the same interests
- Listen, don't try to win the argument, don't push your solutions. The aim is to understand what he thinks is needed
- You can very well explain how you're feeling. Maybe you feel that you're not really sure what you're doing there given that none of your suggestions are accepted. Tell him
- But don't blame, don't presume ill feeling or ignorance. Ask him for suggestions
There's a chance that this is never going to work out and as somebody else said, you're set up for failure. I think you can figure this out if you do as much as you can on your side to understand him, and he still doesn't do his part to help you do well at your job (whatever that is! It may not be what you think).
Something else…
I'm not sure this is part of the issue but I've seen this a lot: you work on an amazing new way of doing things. You figure it all out, document it, dot the i's and cross the t's. And then when you present it nobody is interested. Why? Regardless of whether it's a great idea, you didn't involve anyone else. You need to bring people along with you: ask about their problems, ask for input, collaborate as much as possible…
This. Spend time with the guy, try to understand where he's coming from. When you do talk I'd suggest you:
- Make it clear that your motives are to help the company. I.e., you both have the same interests
- Listen, don't try to win the argument, don't push your solutions. The aim is to understand what he thinks is needed
- You can very well explain how you're feeling. Maybe you feel that you're not really sure what you're doing there given that none of your suggestions are accepted. Tell him
- But don't blame, don't presume ill feeling or ignorance. Ask him for suggestions
There's a chance that this is never going to work out and as somebody else said, you're set up for failure. I think you can figure this out if you do as much as you can on your side to understand him, and he still doesn't do his part to help you do well at your job (whatever that is! It may not be what you think).
Something else…
I'm not sure this is part of the issue but I've seen this a lot: you work on an amazing new way of doing things. You figure it all out, document it, dot the i's and cross the t's. And then when you present it nobody is interested. Why? Regardless of whether it's a great idea, you didn't involve anyone else. You need to bring people along with you: ask about their problems, ask for input, collaborate as much as possible…
I'm doubtful any of these "technical" solutions proposed here will work.
What I mean is this: it sounds like the CEO is completely scattered. Nothing you put in front of him will be enough to shake him out of it. He is only getting 10% of what you are telling him. No matter how well written the document you put out there, no matter what consensus you build, it won't change his relationship to what he is dealing with.
It's terrifying to be the CEO. He's failing constantly and fearful of being exposed as a fraud.
What everyone else deals with, yes, but this time it seems more significant.
My suggestion: find other people who have traveled the same path as he has, specifically the issues you were brought into solve. Find a way to connect them with him. Storytelling from a powerful communicator is the only thing that will break him free of this state.
Once you have that in place, the solutions you put in front of him will be easy.
What I mean is this: it sounds like the CEO is completely scattered. Nothing you put in front of him will be enough to shake him out of it. He is only getting 10% of what you are telling him. No matter how well written the document you put out there, no matter what consensus you build, it won't change his relationship to what he is dealing with.
It's terrifying to be the CEO. He's failing constantly and fearful of being exposed as a fraud.
What everyone else deals with, yes, but this time it seems more significant.
My suggestion: find other people who have traveled the same path as he has, specifically the issues you were brought into solve. Find a way to connect them with him. Storytelling from a powerful communicator is the only thing that will break him free of this state.
Once you have that in place, the solutions you put in front of him will be easy.
You're trying to do too much but moving too slow.
You're asking for permission instead of just doing things.
You wrote up a huge document? This is a startup not an enterprise!
What's the worst thing that can happen? prod breaks right? Make sure it's being backed up! Teach them to tag the branches before merging into master, at least they can have a reference to rollback. Make sure rollback on master can happen with one command.
Build the stage without permission, make it a CI/CD. Show them, taadaah! When you push, you can go test on stage, see? huge documents, calendars? meetings? this is a fast growing startup, all your ideas are great but you are executing it too slow and slowing everyone else down that's why the push back is happening.
You're asking for permission instead of just doing things.
You wrote up a huge document? This is a startup not an enterprise!
What's the worst thing that can happen? prod breaks right? Make sure it's being backed up! Teach them to tag the branches before merging into master, at least they can have a reference to rollback. Make sure rollback on master can happen with one command.
Build the stage without permission, make it a CI/CD. Show them, taadaah! When you push, you can go test on stage, see? huge documents, calendars? meetings? this is a fast growing startup, all your ideas are great but you are executing it too slow and slowing everyone else down that's why the push back is happening.
So you've been given the responsibility but not the authority. You've been set up for failure. Demand clarification of what's expected from you and the backing to carry it out, because right now you are not able to do your job.
Presumably you made it through a rigorous process to be hired? And maybe have some background in the space (it looks like you do).
I'm a former coder turned CEO. What I see here is a leader who is afraid to let go. And at the same time blocking proven models. And after that you carry the fail-bag.
One time, 15+ years ago I joined a small company as an IT Director - reports to CEO - had a pretty good time investment through the interview process. The second day I had similar feeling to what you've expressed. I quit. Sharp talent can find a new gig - quick.
Like is too short for you to waste time explaining that 2>1 to folks who are paying you to tell them that - and then ignore you.
I'm a former coder turned CEO. What I see here is a leader who is afraid to let go. And at the same time blocking proven models. And after that you carry the fail-bag.
One time, 15+ years ago I joined a small company as an IT Director - reports to CEO - had a pretty good time investment through the interview process. The second day I had similar feeling to what you've expressed. I quit. Sharp talent can find a new gig - quick.
Like is too short for you to waste time explaining that 2>1 to folks who are paying you to tell them that - and then ignore you.
What could help you stay sane is to recast how you view your job.
Here is one way to think about it: your job is to help establish a culture of improved reliability. You are accountable for improving the process but you don't have the authority to decide the process.
A large part of being successful in this role is sales.
There is a pattern that works fairly well at starting a new engagement. Initially, the main goal is not to fix things; instead you need to focus on building trust. As you've found out, without trust, you won't have success.
The easiest way to build trust is to find a problem that the team acknowledges and is small. Tell them how you will solve the problem and have them agree. Solve the problem. Then repeat with another problem until you have some trust.
A good indication that you have enough credibility is when they stay asking for your input. Now, you need to find a problem that they are blind to or have given up on solving. Convince them it's a problem worth solving. Metrics are your friend here. Then get buy-in for your proposed solution. Keep visibility add the work is solved. Sacrifice overall speed for immediate partial wins.
At this point, hopefully you've reached a credibility tipping point and established a good working relationship.
I also had advice for dealing with change resistance, but that is not your problem. Your problem is building your personal credibility by delivering something the team cares about and wants solved.
Try not feel insulted. It's not possible. They're only human. :)
Here is one way to think about it: your job is to help establish a culture of improved reliability. You are accountable for improving the process but you don't have the authority to decide the process.
A large part of being successful in this role is sales.
There is a pattern that works fairly well at starting a new engagement. Initially, the main goal is not to fix things; instead you need to focus on building trust. As you've found out, without trust, you won't have success.
The easiest way to build trust is to find a problem that the team acknowledges and is small. Tell them how you will solve the problem and have them agree. Solve the problem. Then repeat with another problem until you have some trust.
A good indication that you have enough credibility is when they stay asking for your input. Now, you need to find a problem that they are blind to or have given up on solving. Convince them it's a problem worth solving. Metrics are your friend here. Then get buy-in for your proposed solution. Keep visibility add the work is solved. Sacrifice overall speed for immediate partial wins.
At this point, hopefully you've reached a credibility tipping point and established a good working relationship.
I also had advice for dealing with change resistance, but that is not your problem. Your problem is building your personal credibility by delivering something the team cares about and wants solved.
Try not feel insulted. It's not possible. They're only human. :)
Sounds like a very toxic company. You're there for a reason, if you have 7 years of experience you do not deserve this attitude. In the West Coast, you can probably find a new job in 3 minutes anyway. Leave and find a new job.
It doesn't sound like a great situation to be in. It sounds like the founder has strong opinions. I suspect he subscribes to Trunk Based Development[1] and Continuous Delivery[2]. He also likely wants to follow the practises that "leading organisations" follow in "Accelerate: State of DevOps Report"[3]. You might find it beneficial to research those subjects prior to discussions with him.
1 - https://trunkbaseddevelopment.com/
2 - https://continuousdelivery.com/
3 - https://cloudplatformonline.com/2018-state-of-devops.htmlJust some friendly advice. When you're referencing stuff elsewhere on the web do it like this:
[1] https://trunkbaseddevelopment.com/
[2] https://continuousdelivery.com/
[3] https://cloudplatformonline.com/2018-state-of-devops.html
This means the links are clickable. Using the code indentation thing you did there causes all sorts of grief for mobile device users.
[1] https://trunkbaseddevelopment.com/
[2] https://continuousdelivery.com/
[3] https://cloudplatformonline.com/2018-state-of-devops.html
This means the links are clickable. Using the code indentation thing you did there causes all sorts of grief for mobile device users.
Thanks! Was faffing with it... the "help" isn't very helpful on how it formats. Sadly can't edit now. :/
Can't believe this comment is so low.
I would guess this is exactly what is happening, an it sounds like OP is trying to push git flow or similar. I would push back against that as well. Trunk based development beats git flow every time.
I would guess this is exactly what is happening, an it sounds like OP is trying to push git flow or similar. I would push back against that as well. Trunk based development beats git flow every time.
It's like reading the same story, but with different protagonist and type of job.
I had a similar situation like yours, but I was in the traditional Sys Admin category.
Basically I had to deal with toxic behavior for a number of years and in reality what happened was to stay where I first started for the whole time.
When they had an issue, I would suggest solutions and useful alternatives, but I would get nothing but "No"s and "don't you dare do such thing!", so they could do their ways. Now, when they would mess up badly with something, they would come back running to me to fix their shit.
I was never trusted nor had they believed in me and my skills to improve our entire infrastructure.
I got tired arguing and fighting for nothing as I was getting nowhere.
Eventually, I automated my tasks to the point of not working, like doing nothing at all for like...4 years or so(?) and right before leaving, I took notes based on my scripts' implementations and then deleted them for good.
Lastly, the headquarters decided to close down our branch.
I don't think I would like to work for such type of environment anymore, but I'm afraid the majority are like this more or less.
I had a similar situation like yours, but I was in the traditional Sys Admin category.
Basically I had to deal with toxic behavior for a number of years and in reality what happened was to stay where I first started for the whole time.
When they had an issue, I would suggest solutions and useful alternatives, but I would get nothing but "No"s and "don't you dare do such thing!", so they could do their ways. Now, when they would mess up badly with something, they would come back running to me to fix their shit.
I was never trusted nor had they believed in me and my skills to improve our entire infrastructure.
I got tired arguing and fighting for nothing as I was getting nowhere.
Eventually, I automated my tasks to the point of not working, like doing nothing at all for like...4 years or so(?) and right before leaving, I took notes based on my scripts' implementations and then deleted them for good.
Lastly, the headquarters decided to close down our branch.
I don't think I would like to work for such type of environment anymore, but I'm afraid the majority are like this more or less.
Wouldn't the company own the scripts you wrote to automate their processes while you worked there? Sounds kind of spiteful not to hand them over, although I can empathize with your frustration (been in similar situations as well).
I guess, but I am pretty sure if they knew, they would fire me therefore I could not risk it.
So, the best thing to do was to delete them as a whole and move on with my life.
So, the best thing to do was to delete them as a whole and move on with my life.
Yeah I gotcha, sometimes that's all you can do. Sounds like they were terrible managers. I hadn't even considered the (obvious) fact that automating your own job can be dangerous.
I did the same thing of automating my own tasks. Only lasted about a year though because I got too bored.
Maybe he just doesn't like what you're selling. Find out what direction the company wants to go and help them get closer to that.
I would recommend temporary feature branches, that frequently merge into master and also release branches for the code that you want to push to production. You can always do hotfix branches off of the release branch.
There's always that wet dream of no branches at all, every edge case automatically tested and pushing right to production if the tests pass. But this is really not practical for most places of business.
But if you can promote some meaningful amount of: 1) Unit tests 2) Integration tests 3) Functional tests That automatically run when code is pushed, that's a good first step.
Also, if you can get them on a regular release cadence of monthly or better, that would also be preferable, and you should be able to release code to pretty much any environment with the click of a button and with little to no manual intervention.
Oh and use Vault. :)
Stick with the basics and it will take them a long way.
I would recommend temporary feature branches, that frequently merge into master and also release branches for the code that you want to push to production. You can always do hotfix branches off of the release branch.
There's always that wet dream of no branches at all, every edge case automatically tested and pushing right to production if the tests pass. But this is really not practical for most places of business.
But if you can promote some meaningful amount of: 1) Unit tests 2) Integration tests 3) Functional tests That automatically run when code is pushed, that's a good first step.
Also, if you can get them on a regular release cadence of monthly or better, that would also be preferable, and you should be able to release code to pretty much any environment with the click of a button and with little to no manual intervention.
Oh and use Vault. :)
Stick with the basics and it will take them a long way.
What’s Vault?
HashiCorp Vault.
It's for storing secrets, keys, certificates, tokens, etc.
Leave. There are better environments to work in.
This. You aren’t happy and you can’t make your boss happy. Accept that it was a mistake, “admit” that it was your fault for not understanding what they wanted, and go your separate ways. Don’t try to get a “dig” in on your way out, just leave on a good terms as you can.
I agree. Especially when the boss starts talking about "trust" and not listening to your proposals. Getting blamed for other people's decisions when you've already warned about the consequences is a flag that you've already lost the battle. Some people just can't be reasoned with.
Granted, saying leave is easy to do but the OP might not be able to due to their financial situation or other circumstances. So maybe leaving is not the right option but I don't believe this relationship can be salvaged so checking out(either mentally or physically) is the only way forward, as I see it.
Granted, saying leave is easy to do but the OP might not be able to due to their financial situation or other circumstances. So maybe leaving is not the right option but I don't believe this relationship can be salvaged so checking out(either mentally or physically) is the only way forward, as I see it.
Alternatively, get ready to leave.
But before you do, push back against the founder. The next time he pushes directly to master, immediately revert the commit, but cherry-pick it over to a development or staging branch instead. Then notify him what you've done and why (including that you're trying to prevent breaking changes being made on master, like happened previously).
How the founder responds to this will determine whether you leave, or whether you stay a bit longer and perhaps gain the founder's trust. But still expect you'll have to leave. :/
But before you do, push back against the founder. The next time he pushes directly to master, immediately revert the commit, but cherry-pick it over to a development or staging branch instead. Then notify him what you've done and why (including that you're trying to prevent breaking changes being made on master, like happened previously).
How the founder responds to this will determine whether you leave, or whether you stay a bit longer and perhaps gain the founder's trust. But still expect you'll have to leave. :/
If it were me, I would want to run a dev system off the master branch and then tag versions that get the OK and are deployed to the production system. That would give the founder some sense of immediate gratification while insulating the production system from goofs. disclaimer << I am not a devops person
Don't make suggestions. Tell them, what they are doing wrong and why.. and then show them what to do. Trust seems like an excuse for being too lazy to adapt to changes.
Get aligned on values as others here have said. We had a similar situation with types of tests and coverage for a product that did not yet find fit. In retrospect everyone could see that it was overkill. After the pivot we now commit to master run fast unit tests in CI, fast acceptance tests in staging and most valuable journey tests for core user actions, like sign up/in do the main action, get notified. If it passes you can push to prod or do some more ad-hoc in staging. Pipeline is fast, have stability, and everyone's happy. Top pri after prod issue is not blocking CI. If not a quick fix, revert til green. We also have a separate dev integration env for testing changes with some uncertainty. Go fast and only break minor things.
You're new to the organization and were hired to help the company go from rough, ship-first, relatively low quality practices to high quality and scalable best practices.
In order to make this a reality, you're going to need to first be very understanding and cognizant of the existing team and the reason they do what they do.
This is the only way to inform your decisions on:
1. Which changes are most high priority ie business impact
2. Which changes will require significant team buy-in
3. Which changes will make the most sense for your team
I very strongly recommend you start with some easy wins:
- Bad key management? Use env variables
- Autoscaling (as you mentioned)
- Clean up no longer in-use aws services to save $$$
Once you've made an improvement or two, then start tackling some of the problems that require more buy-in.
This is anything that affects others' workflows or pace of work.
If the founder is cowboy-coding and pushing things directly to prod, you're going to need to get his buy-in to change his workflow. Come up with a document, plan, or RFC for anything that requires buy-in. Get comments from relevant parties for your plan. Then, and only then, implement the plan.
3 weeks really isn't that long. You still have plenty of time to right this ship.
In order to make this a reality, you're going to need to first be very understanding and cognizant of the existing team and the reason they do what they do.
This is the only way to inform your decisions on:
1. Which changes are most high priority ie business impact
2. Which changes will require significant team buy-in
3. Which changes will make the most sense for your team
I very strongly recommend you start with some easy wins:
- Bad key management? Use env variables
- Autoscaling (as you mentioned)
- Clean up no longer in-use aws services to save $$$
Once you've made an improvement or two, then start tackling some of the problems that require more buy-in.
This is anything that affects others' workflows or pace of work.
If the founder is cowboy-coding and pushing things directly to prod, you're going to need to get his buy-in to change his workflow. Come up with a document, plan, or RFC for anything that requires buy-in. Get comments from relevant parties for your plan. Then, and only then, implement the plan.
3 weeks really isn't that long. You still have plenty of time to right this ship.
I find it amazing how people can make so many assumptions from this Ask HN post and already assume how the CEO is, what the start up needs, if OP is right or wrong and whatnot.
DevOps is much about culture and it's not a one solution fits-it-all that you can grab from your past 7 years and readily apply to a company you just joined. I was responsible for a very big devops transformation in a medium company (3k employees, 80 devs), took around a year and a half to go from ssh to server to deploy to everyone being on Ansible and automated deploys with CI/dev/staging/prod.
I would say to chill :), you just joined the company, 3w is not a lot of time to understand the culture, the devs, what they should optimize for, what is causing them pain, what is taking them time, what is making them money and etc.
Also, have some real 1:1s with the CEO (or whoever you report to) and be very direct about how you are feeling and what the expectations are.
Case in point, I really recommend the two books in this link:
- https://landing.google.com/sre/book.html
I don't believe any of that can be applied blindly to a company that isn't google size.
DevOps is much about culture and it's not a one solution fits-it-all that you can grab from your past 7 years and readily apply to a company you just joined. I was responsible for a very big devops transformation in a medium company (3k employees, 80 devs), took around a year and a half to go from ssh to server to deploy to everyone being on Ansible and automated deploys with CI/dev/staging/prod.
I would say to chill :), you just joined the company, 3w is not a lot of time to understand the culture, the devs, what they should optimize for, what is causing them pain, what is taking them time, what is making them money and etc.
Also, have some real 1:1s with the CEO (or whoever you report to) and be very direct about how you are feeling and what the expectations are.
Case in point, I really recommend the two books in this link:
- https://landing.google.com/sre/book.html
I don't believe any of that can be applied blindly to a company that isn't google size.
Only thing I can think of: If you have freedom of action (you can choose what to do with your time), then do something small, perfect, and seamless. Present it as a fait accompli. That's your foot in the door. Repeat.
Later note: reading through all the responses, I'm impressed at what a great resource the minds and experience of HN are.
>founder has pushed breaking changes directly to master
Is founder also CEO? If founder is ceo and is coding rather than working on sales / marketing / traction, then you need to run briskly away
Is founder also CEO? If founder is ceo and is coding rather than working on sales / marketing / traction, then you need to run briskly away
why? if he is a tech founder, and there are co-founders that work on sales and marketing, why should he give up the CEO position?
or do you mean to say that from his behavior he is obviously not qualified?
greetings, eMBee.
greetings, eMBee.
Put simply: Lead Developer != CEO.
Unless it is super early stage (pre-employees, pre-revenue, pre-everything), the CEO should not be hands-on-keyboard. If they are, there are several problems:
1) Delegation. They are demonstrating a management failure to delegate.
2) Even in a technology-oriented business, the job of the CEO is not to cut code. The function of the CEO is to manage and lead all the things related to the growth and success of the business (think: sale, marketing, hiring and recruiting, raising capital, interacting with investors, and a whole lot more). If they want to undertake the technical functions of the business, they should become CTO / VP of Engineering / etc and turn the reigns over to a CEO who can execute on the business.
3) Even if there are other team members engaged in sales and marketing, see point #2 - the CEO is fundamentally responsible for the growth and financial success of the business and it is highly unlikely that technical tasks should be their priority.
Unless it is super early stage (pre-employees, pre-revenue, pre-everything), the CEO should not be hands-on-keyboard. If they are, there are several problems:
1) Delegation. They are demonstrating a management failure to delegate.
2) Even in a technology-oriented business, the job of the CEO is not to cut code. The function of the CEO is to manage and lead all the things related to the growth and success of the business (think: sale, marketing, hiring and recruiting, raising capital, interacting with investors, and a whole lot more). If they want to undertake the technical functions of the business, they should become CTO / VP of Engineering / etc and turn the reigns over to a CEO who can execute on the business.
3) Even if there are other team members engaged in sales and marketing, see point #2 - the CEO is fundamentally responsible for the growth and financial success of the business and it is highly unlikely that technical tasks should be their priority.
Concurred.
My suggestion is similar to the ones specified here.
Part of the DevOps concepts is to try to take a scientific unbiased approach that is to experiment, measure and make more changes. You could always treat implementing these changes as experiments to see what works.
Simplicity, trust, team rapport and support from top down really helps in order to implement change.
If the team, founder and manager aren't really interested in changing you can either work on influencing and correcting their mindsets, fight them on it or move on.
Part of the DevOps concepts is to try to take a scientific unbiased approach that is to experiment, measure and make more changes. You could always treat implementing these changes as experiments to see what works.
Simplicity, trust, team rapport and support from top down really helps in order to implement change.
If the team, founder and manager aren't really interested in changing you can either work on influencing and correcting their mindsets, fight them on it or move on.
My guess as to what's going on here is that what the founder actually wants is for you to be the expert on the deployment environment & related tools to help ship things faster (in the short-run) and fix things when they break. On the other hand, what you actually want is to spend a bunch of time up-front building tools and processes to keep things stable. Somehow what each of you expects and wants from the other was miscommunicated.
Some leaders are control freaks, and they resist anything that reduces their control of things. Some are so hungry for control that they behave in irrational and arbitrary ways, often undermining themselves and their companies. I wrote about this at length here:
https://www.amazon.com/Destroy-Tech-Startup-Easy-Steps/dp/09...
https://www.amazon.com/Destroy-Tech-Startup-Easy-Steps/dp/09...
Put yourself in the founder shoes and think. He/She already has some way of doing things, which is working till now, and you are trying to break it. There will be some resistance to this inertia.
When you talked about the standard, people tend to make themselves as exception saying that "our case is different". Check if this is the case.
It seems that you are starting big. Trying to push the changes, which people are not used to. Start with small changes, something like statistical analysis tools or some small automated test case tool. Something which will be simple and not much time consuming for developers. Once they see its value, may be then they can "trust" you with big changes.
Then there is psychological approach for making changes. You can present your ideas in such a way that the person buying (in your case founder), feels that he/she has a choice or a major role in decision making. But then you present choices, such that the one you prefer comes out as obviously best. Check if this approach can work in your case.
Finally, it's your life and your job. Mental stability is more important that the company you are working in. Don't hesitate to fly away on first opportunity.
When you talked about the standard, people tend to make themselves as exception saying that "our case is different". Check if this is the case.
It seems that you are starting big. Trying to push the changes, which people are not used to. Start with small changes, something like statistical analysis tools or some small automated test case tool. Something which will be simple and not much time consuming for developers. Once they see its value, may be then they can "trust" you with big changes.
Then there is psychological approach for making changes. You can present your ideas in such a way that the person buying (in your case founder), feels that he/she has a choice or a major role in decision making. But then you present choices, such that the one you prefer comes out as obviously best. Check if this approach can work in your case.
Finally, it's your life and your job. Mental stability is more important that the company you are working in. Don't hesitate to fly away on first opportunity.
What about making production branch which goes live and is tested before and let them push into master if they like it so much? Practically, it does not matter which branch and how it is called. Important is that you don't push untested changes to live.
Also, writing large document is not how you convince people. If they dont think your ideas are good, they perceive it just as an attempt to force them to do something that does not make sense to them.
> since the founder has pushed breaking changes directly to master and then I got flack since I am now the "systems" guy and am responsible for the health of the app. I'm trying to think of the best way to approach this diplomatically. Do any HNers have some pointers or advice to give?
This is large red flag. Toxic politics warning. Document what you do, document discussions decisions etc. Diplomatic is partly way to go, especially since it is founder, but if situation call for it be ready to defend yourself.
Also, if that sort of founder behavior becomes usual, run away. No matter what your tech skills are, this will only get worst as other employees will naturally take clues from founder and imitate behavior.
This is where production branch I suggested up there can really help.
> He also keeps harping that I need to "earn his trust."
Another red flag warning. Sometimes it is not about who you are and what you do, sometimes it is about who they are. And who they are suggests they will not build productive effective workplace. It will generate ugly office politics.
If that happens, leave before you are so frustrated that you lost confidence or burned out.
Also, writing large document is not how you convince people. If they dont think your ideas are good, they perceive it just as an attempt to force them to do something that does not make sense to them.
> since the founder has pushed breaking changes directly to master and then I got flack since I am now the "systems" guy and am responsible for the health of the app. I'm trying to think of the best way to approach this diplomatically. Do any HNers have some pointers or advice to give?
This is large red flag. Toxic politics warning. Document what you do, document discussions decisions etc. Diplomatic is partly way to go, especially since it is founder, but if situation call for it be ready to defend yourself.
Also, if that sort of founder behavior becomes usual, run away. No matter what your tech skills are, this will only get worst as other employees will naturally take clues from founder and imitate behavior.
This is where production branch I suggested up there can really help.
> He also keeps harping that I need to "earn his trust."
Another red flag warning. Sometimes it is not about who you are and what you do, sometimes it is about who they are. And who they are suggests they will not build productive effective workplace. It will generate ugly office politics.
If that happens, leave before you are so frustrated that you lost confidence or burned out.
Get out while you can. You were brought on as a checkbox on someone's list, either a BoD, external investors, someone's due diligence. You'll have little to no impact with your domain expertise sadly. I have seen a few times where the boss would not listen to any standard reasoning or best practices.
I worked with a start-up where owner confessed he is just looking to get out as quickly as possible. We were not supposed to build processes that would slow down the development at the cost of huge technical debt - but that wouldn't be for him to worry. He did have a number of hires so that on paper it would look nice to investors. We had an experienced and old CEO whose job was to smile and drink with investors, number of developers to fiddle with code, two senior devs that were doing actual coding and a devops guy who spent time on Reddit and baby sitting juniors so they don't break the app too badly when deploying to production and marketing team that was making up ideas what they could shoe next to investors. Company was paying "customers" to use the product. I left early when I realised the equity package is made up. I'd say run! Not worth your reputation.
"If you can’t measure it, you can’t improve it."
I would suggest starting by gathering data to show you CEO where you think you can make improvements. Sometimes just applying a standard processes is not enough to convince management of the benefits if they are not yet bought in.
I would suggest starting by gathering data to show you CEO where you think you can make improvements. Sometimes just applying a standard processes is not enough to convince management of the benefits if they are not yet bought in.
I find myself in a similar situation. I took a job as a software developer at a hardware shop. They have different processes to what I am used to. I keep pushing for more robust project management but I am getting a lot of push back ("don't tell me how to do my job" level push back).
There are two parts to this:
1. Build a better relationship with the founder (carefully)
2. Focus on adding value. New processes are not going well, so scrap them and do the best you can with the processes they have. Your arguments for change will go over better if you are adding a lot of value to the company. That is were credibility comes from in small groups like this.
There are two parts to this:
1. Build a better relationship with the founder (carefully)
2. Focus on adding value. New processes are not going well, so scrap them and do the best you can with the processes they have. Your arguments for change will go over better if you are adding a lot of value to the company. That is were credibility comes from in small groups like this.
> He also keeps harping that I need to "earn his trust."
If he doesn't trust you, he won't listen to your advice. Use a collaborative model. Try to figure out the 'pictures in his head'.
Good food for thought on this subject from Stuart Diamond > https://www.microsoft.com/en-us/research/video/getting-more-...
If he doesn't trust you, he won't listen to your advice. Use a collaborative model. Try to figure out the 'pictures in his head'.
Good food for thought on this subject from Stuart Diamond > https://www.microsoft.com/en-us/research/video/getting-more-...
[deleted]
How does the founder behave with other employees? Especially the new ones.
Focus on listening and validating their thoughts. i.e. "What are your concerns?" "Sounds like you're worried about X. I can see where you're coming from".
You don't have to agree with them just make them feel like they are being heard.
Rinse and repeat.
If this doesn't work..... Find a new job.
Focus on listening and validating their thoughts. i.e. "What are your concerns?" "Sounds like you're worried about X. I can see where you're coming from".
You don't have to agree with them just make them feel like they are being heard.
Rinse and repeat.
If this doesn't work..... Find a new job.
If you're willing to relocate, the company I work for is in need of someone like you...
"So after two weeks, I got the approval to create a staging branch." - lol
i think a little conflict is fine, working through conflict can often times build trust, use this as a learning experience because you're always going to have conflict, even if this conflict feels like a core belief mismatch. i find the founder is a bit too super direct and lacks tact, i'm a senior director, and have never told anyone to earn my trust, it's a given, i gain nothing by explicitly stating this, but that's besides the point, different strokes, different folks.
i also see a bit of a power struggle here, because your role, perhaps at other companies, has been more control and process oriented, this is a startup with few processes and control seems to be completely with the founder.
here's what i suggest, change is inevitably hard to deal with, it takes time and it's very natural for people to resist, so the first thing for you is to decide whether you're ok with this taking a long time, if not, there are a ton of other opportunities out there, don't waste time. if you're willing to stick it out and see value in the business, in the role, then figure out how to do things in incremental steps aka non-earth shattering, it shouldn't be, adhere to my god-like, every company does it this way devops pattern or else (even if it is god-like). baby steps where it's easier for the team/founder to accept, less friction/inertia.
another thing is through statistics, founders change caused this which impacted that, we had 3h downtime because founder's change hosed the system, this caused roughly $20,000 hit to revenue, or we lost x users as a results of this. if he's ok with this after repeatedly pointing this out, maybe it's because his goal is different, he's looking for speed of execution as opposed to short term revenue gain, in which case you need to try to align and figure out ways to break things even faster, but with simple ways of rolling back. maybe he/she wants a/b type testing with quick rollbacks in case stuff hits the fan. obviously aligning with the founder is going to be critical.
i also suggest a book titled "what got you here won't get your there". you should treat every new role with an open outlook, there are always new challenges to sharpen your skills and experience, if everything's similar to what you did before, you're essentially not learning much.
i also see a bit of a power struggle here, because your role, perhaps at other companies, has been more control and process oriented, this is a startup with few processes and control seems to be completely with the founder.
here's what i suggest, change is inevitably hard to deal with, it takes time and it's very natural for people to resist, so the first thing for you is to decide whether you're ok with this taking a long time, if not, there are a ton of other opportunities out there, don't waste time. if you're willing to stick it out and see value in the business, in the role, then figure out how to do things in incremental steps aka non-earth shattering, it shouldn't be, adhere to my god-like, every company does it this way devops pattern or else (even if it is god-like). baby steps where it's easier for the team/founder to accept, less friction/inertia.
another thing is through statistics, founders change caused this which impacted that, we had 3h downtime because founder's change hosed the system, this caused roughly $20,000 hit to revenue, or we lost x users as a results of this. if he's ok with this after repeatedly pointing this out, maybe it's because his goal is different, he's looking for speed of execution as opposed to short term revenue gain, in which case you need to try to align and figure out ways to break things even faster, but with simple ways of rolling back. maybe he/she wants a/b type testing with quick rollbacks in case stuff hits the fan. obviously aligning with the founder is going to be critical.
i also suggest a book titled "what got you here won't get your there". you should treat every new role with an open outlook, there are always new challenges to sharpen your skills and experience, if everything's similar to what you did before, you're essentially not learning much.
Your profile says you're in WA. How close are you to Portland? We should have a beer and swap stories. I'm in a similar situation.
> especially since the founder has pushed breaking changes directly to master and then I got flack since I am now the "systems" guy and am responsible for the health of the app
> use of a staging environment - is met with suspicion and doubt
If he doesn't want to give you authority to design the deployment pipeline, then he doesn't get to make you responsible for it. Responsibility/authority mismatch is a classic management mistake, they cover it in lower division business supervision and management classes. It's generally a result of one or more personality defects and/or something interfering with proper reasoning. If you can't get it fixed by figuring out how to work with the founder, then you need to find another job because it's only going to get worse as things ramp up.
Based on what you're saying about wanting more research and suspicion when presented with change, it sounds like they have Groucho Marx syndrome due to either a self-esteem issue or impostor syndrome. The usual line of reasoning is:
1. I need expert help. I will look outside my organization for it.
2. I hired expert help, and they are now working for my company.
3. It is self-evident that this person does not know what they are doing because they are working for my organization, and I have a low opinion of myself and/or my organization.
4. I need expert help.
Rinse and repeat.
I'm currently working with a founder who constantly pushes to master, and I didn't push back because he earned my trust by being a really decent programmer. He's broken things in the past, but in general he fixes things faster than I can detect them so I'm ok with it. The engineering organization doesn't have many complaints about it because the company needs him free to work on the hiring pipeline, which is crucial because our staff keeps getting poached.
If he was mediocre, I'd have privilege separated his accounts a long time ago and enabled annoying 3FA on his account with master commit privileges.
> use of a staging environment - is met with suspicion and doubt
If he doesn't want to give you authority to design the deployment pipeline, then he doesn't get to make you responsible for it. Responsibility/authority mismatch is a classic management mistake, they cover it in lower division business supervision and management classes. It's generally a result of one or more personality defects and/or something interfering with proper reasoning. If you can't get it fixed by figuring out how to work with the founder, then you need to find another job because it's only going to get worse as things ramp up.
Based on what you're saying about wanting more research and suspicion when presented with change, it sounds like they have Groucho Marx syndrome due to either a self-esteem issue or impostor syndrome. The usual line of reasoning is:
1. I need expert help. I will look outside my organization for it.
2. I hired expert help, and they are now working for my company.
3. It is self-evident that this person does not know what they are doing because they are working for my organization, and I have a low opinion of myself and/or my organization.
4. I need expert help.
Rinse and repeat.
I'm currently working with a founder who constantly pushes to master, and I didn't push back because he earned my trust by being a really decent programmer. He's broken things in the past, but in general he fixes things faster than I can detect them so I'm ok with it. The engineering organization doesn't have many complaints about it because the company needs him free to work on the hiring pipeline, which is crucial because our staff keeps getting poached.
If he was mediocre, I'd have privilege separated his accounts a long time ago and enabled annoying 3FA on his account with master commit privileges.
Sounds like there is a miscommunication about what your role is and possibly what each of you mean by "devops".
Also add that 'flake tests' were a sore point until it became clear which ones were actually the system failing and which tests didn't reflect actual usage. By keeping stats on test faulures it was clear which were causing distraction time and fix them or system cause first. After that we got stability much more quickly than always treading seemingly random errors.
I've had similar issues in the past, and here's what I did to resolve them:
1. Get the others on board. I assume there are other developers in your team. Pitch your ideas to them, and not management, and get whoever is in charge of the developers to agree that your plan is the way to go. If there isn't a lead dev or dev team manager, just get them all to agree on a date where they switch to your way of doing things.
2. Do, rather than ask. I had the same issues with a director that fancied themselves as a coder. To get around this, I implemented a plan, set up a continuous delivery lifecycle, and put it into place. Their FTP access was revoked, and when they went through the (stupid) process to get FTP access again to tweak some CSS the next deployment wiped their changes out.
3. Let them fuck up, and be accountable for their fuck-ups. Ultimately, you can't stop someone in power from abusing it, so when he/she does and it inevitably fucks up, tell them they fucked up and the team will fix it. Going back to my FTPing director, he tried to fuck with some HTML on a Razor template and ended up breaking a page. We had a basic script that checked the sitemap periodically for anything other than 200 responses, and alarm bells soon rang. I asked around on Slack to see who had made live changes without going through the process, and after he (eventually) owned up, we re-deployed the last release and fixed the error. That public log of their fuckup was enough to ensure they stuck to the process after.
4. Own it, but take your time to come up with a solution. I think your biggest mistake is to immediately jump in and start making changes. I know they're a startup, but if I were you I would've committed to a week or two of simply watching and helping the team go about their business as it is. Get everything to tell you what the problems with their current setup is, and lead them towards the answer you want to provide. After that time expires, deliver your findings, backed up with all the feedback you've received from the entire team, and move to step 1.
5. Be prepared to walk if things don't improve. Sometimes, there's only so much you can do. You can have literally everyone on board, except one person in charge, and they'll be more than happy to derail everything in order to do what they want. If you reach a point where you feel that there isn't a way around a blocker, and you're absolutely sure that it's not your (or your teams) fault, then walk. Line up an interview, and simply tell them that you've accepted a job elsewhere. Don't rub their noses in it, or say the obvious reason why you're leaving. The team will probably already know why. Just leave, and thank them for hiring you.
1. Get the others on board. I assume there are other developers in your team. Pitch your ideas to them, and not management, and get whoever is in charge of the developers to agree that your plan is the way to go. If there isn't a lead dev or dev team manager, just get them all to agree on a date where they switch to your way of doing things.
2. Do, rather than ask. I had the same issues with a director that fancied themselves as a coder. To get around this, I implemented a plan, set up a continuous delivery lifecycle, and put it into place. Their FTP access was revoked, and when they went through the (stupid) process to get FTP access again to tweak some CSS the next deployment wiped their changes out.
3. Let them fuck up, and be accountable for their fuck-ups. Ultimately, you can't stop someone in power from abusing it, so when he/she does and it inevitably fucks up, tell them they fucked up and the team will fix it. Going back to my FTPing director, he tried to fuck with some HTML on a Razor template and ended up breaking a page. We had a basic script that checked the sitemap periodically for anything other than 200 responses, and alarm bells soon rang. I asked around on Slack to see who had made live changes without going through the process, and after he (eventually) owned up, we re-deployed the last release and fixed the error. That public log of their fuckup was enough to ensure they stuck to the process after.
4. Own it, but take your time to come up with a solution. I think your biggest mistake is to immediately jump in and start making changes. I know they're a startup, but if I were you I would've committed to a week or two of simply watching and helping the team go about their business as it is. Get everything to tell you what the problems with their current setup is, and lead them towards the answer you want to provide. After that time expires, deliver your findings, backed up with all the feedback you've received from the entire team, and move to step 1.
5. Be prepared to walk if things don't improve. Sometimes, there's only so much you can do. You can have literally everyone on board, except one person in charge, and they'll be more than happy to derail everything in order to do what they want. If you reach a point where you feel that there isn't a way around a blocker, and you're absolutely sure that it's not your (or your teams) fault, then walk. Line up an interview, and simply tell them that you've accepted a job elsewhere. Don't rub their noses in it, or say the obvious reason why you're leaving. The team will probably already know why. Just leave, and thank them for hiring you.
You are probably a scapegoat. Pull the ripcord.
Since joining I've received considerable pushback on my ideas. Obviously I am the new guy and don't expect to just dictate people around nor is that my intention. However what really gets me is that the founder just straight up does not seem to trust me. Every suggestion I give - even things as industry standard such as the use of a staging environment - is met with suspicion and doubt and the insistence that I hadn't done enough proper "research" on the idea. I find this highly offensive considering I've been working in DevOps for almost seven years and before that was a SysAdmin.
So after two weeks, I got the approval to create a staging branch. I wrote up a huge document on the staging workflow, covering edge cases such as hotfixes and rollbacks. I put time on people's calendars and walked them through it including the founder. Yet he still continues to resist the change and pushes directly to master, ignores the new processes I put in place, and generally just does what he wants. He also keeps harping that I need to "earn his trust." I find this baffling - I mean, I was hired for a reason right? Why would they hire me if they didn't trust me?
I am starting to get frustrated, especially since the founder has pushed breaking changes directly to master and then I got flack since I am now the "systems" guy and am responsible for the health of the app. I'm trying to think of the best way to approach this diplomatically. Do any HNers have some pointers or advice to give? I've worked with difficult clients before but I've never faced such a huge lack of trust.