I'm only working with logs across 8 servers with a lot of cross talk and ELK covered the logging side brilliantly, but falls over on metrics from what I have seen so far. I'll check out the TICK stack, that sounds like it may be a welcome replacement.
Never heard of Elastalert, I'll have to check that out. By the name I'm guessing it's triggers off an elasticsearch server, which would cover infrastructure I already have - so that's nice.
Hah. I went to college with you. Small world. Will check out your library, I've been doing a lot of work with chatbots over the last year, focused on Slack. I've only really been working with Java solutions though, so a python library is a nice change of pace.
I've been meaning to use grafana for more than homelab server monitoring.
I've always found it so satisfying to sit back after setting up these types of systems and to watch everything churn out metrics / logs. I'll check out the kibana plugin, I'm using that in production at the moment.
I use slack for a lot of notifications too. We strapped a python chatbot on as an integration and gave the bot access to our product which made asking the bot what's happening really easy.
Chatops are catching on, and I'm intrigued. Thanks for the response!
I've been dancing over the line between devops and backend development. It sounds like I am where you were historically, I'm using ansible to deploy docker images to EC2 instances, and a few monitoring scripts in python to do a little more finely tuned orchestration.
Lately I've been using home-brew chatops to manage products so it's nice to hear what other people are using. Operable looks really interesting, I'm going to give it a shot. Thanks for the example cog!
My big issue was getting multi-node deployments working well in AWS. I hit walls of configuration issues, DNS issues, poor documentation on fields, and generally could not make much forward progress. Running locally or on a single node OpenShift was fantastic, the haproxies for ingress were easy to configure and launching new services was impressively easy.
I was leveraging EFS as NFS mounts for my persistent volumes and had good results.
You might check out fabric8 if only for their visualizations of what is going on in your openshift / kubernetes environment.
Thanks for the youtube link! I'll be sure to check it out
Well that's exciting. I've been pushing towards Kubernetes with the company I'm at right now, got half the battle done by dockerizing our services and using Harbor to manage the containers.
Do you have any advice, or gotchas on getting into kubernetes development?
Yes, but for the unique reason that I have worked on web tracking analytics packages and they are terrifying.
From what I have seen working on those packages, the vast majority of users do not care, or do not understand what they are in for and will say "Sure, track me, what's the worst that could happen?"
We used the dreaded zombie-cookies to track when users would hit our pages and to track their passage across content because it gave us a lot more information on how our products were used, and as such gave us a lot of information to hand over to marketing to up our lead gen strategies. From there, we could wow investors.
From a company perspective, we loved that most users would blindly click yes. Enough did so that we never even bothered looking into the minority that opted for do not track.
I currently work as a self-employed independent contractor in the cloud-engineering and devops automation sector in the greater Boston area. I have a strong focus on minimizing cost while making sure the customer achieves that they intend to, and that has given me an edge over fellow contractors that like to drop a known pattern in place and call it a day.
Here's an example.
I had a meeting a few days ago where I was asked to optimize a section of a product that made no sense to me (eliminating DNS and SSL tunnel start-up times). I have a firm believe in how I operate that I do not want to waste my customer's time, money, or my time in the work that I do.
I came back with a data-driven response showing other areas of the product with low hanging fruit that got seconds of time back instead of milliseconds on the area they wanted. I implemented one solution as an example, and landed several more months of contract work with them as a result.
I don't want to waste my time to get paid, I want to solve interesting problems and get paid solving them. I also want to stand behind my work, and stand proud.
So for me planning my career is as simple as finding companies that have problems I can address, for me that is low level network protocol analysis, distributed systems, cloud deployments, and java optimization. I hit those hard, and then build up quotes from customers I have worked with to show potential leads what I am capable of. Understanding that anyone you are talking to is a potential lead was huge for me. Just because it's some guy you ran into at a bar, some woman talking about her start-up, whatever - they all are potential customers, and you need to hold yourself as a professional.
Don't be an ass, establish credibility, and show that you actually care. Money follows.
I have personally found that anyone taking issue with this is a pain to work with in general. If you understand what the developer is talking about, and you know that this is a pedantic comparison, then what is the issue?
The only times I have seen someone take issue was because of an ego-related response. I don't have time to deal with that in my day to day, so I generally just say "Whatever floats your boat" and fall in with their term, simmilar to how I would document code and fall in place with code style while working on someone's project with existing code.
Unless there is something I see that would actually cause real problems for future development, customers, or myself, I won't take issue with the little things. There are a lot of other hills that I would rather die on than spaces and hyphens.
I think that this is a question that will draw a lot of opinionated responses.
There are new technologies driving every day both on the front end and back-end that demand attention to stay 'current' with what is going on. I don't think that it is fair to say that either is 'harder' which will not earn me a lot of love.
I am a back-end engineer currently doing freelance independent contacting, and I am fully self employed in the U.S. and I have developed a lot of respect for front end developers in the last two years while pursuing a true full-stack set of skills. I work with Java, ansible, docker, kubernetes, and AWS for my development cycles and I find the front-end work to be the trickiest personally. I say that because it is typically what customers touch every day, so it gets the most scrutiny. My back-end code can be solid and the front-end engineer can screw up the UI and I will still take the fall, so I think it is worth noting that regardless of difficulty you need to think about what the customer will be looking at, at the end of the day.
I personally think that with traditional back-end coding you have to learn good coding practices to be able to fit into dev ecosystems, and you get very active feedback on when you're doing something wrong. I have not had the same experience with the front-end, I have seen a lot more of "Does it work? Cool. Sure we need to fix it, but we can do that later" while working on that side of products. In my experience that has led to rushed code that has bugs that are not easily found, or is just slow in general.
So which one is harder? I want to say back-end because I am a back-end engineer, every front-end engineer I know says the front-end is worse. So at this point I honestly think it more depends on what you know as an individual.
I am a not so social software engineer, so take it with a grain of salt. I think I would divide it into three categories
* Perks and incentives
These are things like coffee, having an espresso machine at work, getting access to free snacks. Gym access is another one I have commonly seen. I have also seen things like programs that give you access to free classes or certification programs (when the hell am I supposed to have time to do that?) These little things can make my day better and lighten my mood when things are going well at a company.
* Caring about you as a person (?)
I would call this roaming the halls - or when management walks around asking people how they are, what did they do over the weekend, are they okay. This can be a great way to gauge what's going on in a small to medium business. That is until it's the same question day in and day out, or you're explaining your hobby to the same person for the fifth time and they obviously aren't listening, then it feels forced and the illusion fades. That said, I'd rather have the person that makes the effort as a manager to get to know me at a least a bit, than to have a suit treat me like a number on a piece of paper. This is another one where when things are going well at a company, I think that management showing they care is important, and it does make me feel more motivated to do my job.
* Gatherings
I call these "Mandatory fun", I have never been one to want to go hang out at a bar with coworkers after work. It feels forced and awkward to me. When you're spending the majority of your week pulling your hair out trying to get a product working, or bugs fixed, or put out fires, or come back from a layoff - it's hard for me to find these to be fun. That said, I know a lot of other people do find them fun, so I understand why they happen. But using a pizza party to try to cheer up employees instead of communicating what is going on at the company is a half measure that just erodes trust over time.
Where this all falls apart for me is when a company starts to have issues. I have personally found that when you don't have transparency in a company, you lose trust. Is my boss being nice to me because they really don't want me to quit? Or because they're a nice person? Staying motivated when you're facing impending failure is what made me want to respond to this. When everything is going well, it's easy to stay motivated, when things get rough, everything changes.
The perks start to vanish as budgets get cut, the lazy conversations about how much fun you had over the weekend are replaced with monosyllabic responses - you don't have time because you have to ship that feature yesterday. You start to realize that you're dreading going to work. You start to realize you don't know what you're doing anymore. You start you realize this isn't what you thought it would be.
How do you stay motivated during that?
The only thing I have found is sheer willpower, fighting through it and just telling yourself to keep going. Carefully managing what you can and can't do so you don't burn out. I have yet to have a company offer anything that really motivated me during those times other than clear and honest explanations of where we are, what we need to do, why we need to do that, and how that will make my life better.
Transparency and honesty has been the only real motivator for me when things are tough - but the coffee is nice too.
Thanks for the suggestion!