Can we use Jenkins for that?(engineering.simondata.com)
engineering.simondata.com
Can we use Jenkins for that?
http://engineering.simondata.com/can-we-use-jenkins-for-that/
49 comments
We were in the same boat as we moved towards a more Dockerized production environment on Kubernetes. There is a great Jenkins+Kubernetes whitepaper detailing how to string all of the various plugins together, check all of the right boxes, fill all of the right fields in. But it was apparent that the setup and maintenance costs were going to be higher than we wanted. And nobody outside of the engineering group would probably be able to figure out how to use the thing (we have a lot of other teams writing code that aren't "engineering" teams).
Right before I was about to give up, I found Drone: http://try.drone.io/ (GitHub: https://github.com/drone/drone)
With Drone, builds are done within Docker containers. Cool enough, nothing super special there. The real magic is that all of the plugins (http://addons.drone.io/) themselves are also ran in Docker containers. So we can write plugins in whatever language we want, they can be pinned to specific tags or auto-upgraded at build time.
That and the setup process was basically "run this Docker image, pass in some env vars". The UI, while still very basic is easy enough for our other teams to use, and the small but growing community is very receptive to feedback and pull requests.
Check it out if you aren't opposed to a Docker-heavy build system. If you are running a Kubernetes setup like us, here are some starter manifests for you (http://gc-taylor.com/blog/2015/10/27/example-drone-ci-kubern...)
Right before I was about to give up, I found Drone: http://try.drone.io/ (GitHub: https://github.com/drone/drone)
With Drone, builds are done within Docker containers. Cool enough, nothing super special there. The real magic is that all of the plugins (http://addons.drone.io/) themselves are also ran in Docker containers. So we can write plugins in whatever language we want, they can be pinned to specific tags or auto-upgraded at build time.
That and the setup process was basically "run this Docker image, pass in some env vars". The UI, while still very basic is easy enough for our other teams to use, and the small but growing community is very receptive to feedback and pull requests.
Check it out if you aren't opposed to a Docker-heavy build system. If you are running a Kubernetes setup like us, here are some starter manifests for you (http://gc-taylor.com/blog/2015/10/27/example-drone-ci-kubern...)
I agree with all this. Unfortunately I haven't found anything less bad :(
Jenkins + the Job DSL Plugin [0] has been amazing. We don't do any configuration in the web interface, instead everything lives in a groovy file on BitBucket. Every time we push a change Jenkins grabs it and auto runs the groovy file, regenerating all of our jobs. We have it set up so that a job is created for every branch in our repo, so much easier to work with. All a developer has to do to get his branch building is push it to the repo.
[0] https://wiki.jenkins-ci.org/display/JENKINS/Job+DSL+Plugin
[0] https://wiki.jenkins-ci.org/display/JENKINS/Job+DSL+Plugin
I also used JobDSL, and thought it worked pretty well. I don't think my teammates agreed, they seemed to just want to change things in the UI and this was just something getting in the way of their work. Some were willing to learn it, others weren't interested.
I think the new Jenkinsfile approach looks very promising: http://jenkins-ci.org/content/pipeline-code-multibranch-work...
I think the new Jenkinsfile approach looks very promising: http://jenkins-ci.org/content/pipeline-code-multibranch-work...
We do similar thing but with workflow plugins.
You only need recognition-memory to use Web and GUI interfaces, whereas you need recall-memory to use command line interfaces and scripts, which is far more difficult. That's why clicking in a Web GUI or graphical XML-generator replaced scripting. You need a lot of previous information in your memory for recall to write stuff like the following example taken from your linked page:
def project = 'quidryan/aws-sdk-test'
def branchApi = new URL("https://api.github.com/repos/${project}/branches")
def branches = new groovy.json.JsonSlurper().parse(branchApi.newReader())
branches.each {
def branchName = it.name
job {
name "${project}-${branchName}".replaceAll('/','-')
scm {
git("git://github.com/${project}.git", branchName)
}
steps {
maven("test -Dproject.name=${project}/${branchName}")
}
}
}
GUIs are more discoverable than command lines. Text files are easier to save in version control. So it's common to have a system where the GUI hooks new users, but they eventually have enough configuration in it that the GUI is no longer a reasonable way to control it all, and they need a way to extract the information in a reasonable format. Hence the reason several Jenkins plugins have evolved to do that.
At work we use Jenkins Job Builder http://docs.openstack.org/infra/jenkins-job-builder/#content... to maintain jobs using JSON or YAML files. As a result we rarely have to deal with the Jenkins web UI and that's alright :)
Jenkins Job Builder is, IME, the only way to keep complex Jenkins configs sane. Did you know that Jenkins is constantly rewriting its own XML config? We've had Jenkins itself corrupt its own config (with some combination of active plugins).
Keeping your configs in YAML in source control is such a win. Not a fan of Jenkins, BTW.
Keeping your configs in YAML in source control is such a win. Not a fan of Jenkins, BTW.
Every time I install Jenkins, I install the thinBackup plugin. Losing your configuration is way too common, but it's easy to restore.
In line with the other suggestions, there is also jenkins-autojobs [1] (disclaimer: I wrote it). It's a set of scripts that listen for changes in a git|hg|svn repository and create|update jobs from template jobs and the rules defined in a yaml config file [2].
[1] https://github.com/gvalkov/jenkins-autojobs [2] http://jenkins-autojobs.readthedocs.org/en/latest/examples.h...
[1] https://github.com/gvalkov/jenkins-autojobs [2] http://jenkins-autojobs.readthedocs.org/en/latest/examples.h...
We use this to generate builds from our (many, many) feature branches, its amazingly useful, bravo!
Using Job DSL + Jenkins Job Builder here, trying to move as much stuff off Jenkins as possible.
Have to second all of this. There plugin ecosystem looks full but the vast majority of plugins are unsupported, qality varies enormously without any good way to judge it, and god help you if you want to mix and match plugins which weren't made to work together.
The configuration is hostile to separation of concerns. The configuration of your remote build nodes goes in the same XML as the configuration that groups your jobs into "views" on the front end.
I could go on. I don't think I'd build anything complex with Jenkins again. Id prefer to build a job system from scratch, honestly.
The configuration is hostile to separation of concerns. The configuration of your remote build nodes goes in the same XML as the configuration that groups your jobs into "views" on the front end.
I could go on. I don't think I'd build anything complex with Jenkins again. Id prefer to build a job system from scratch, honestly.
the Gerrit plugin
There are projects out there with workflows where people submit to Github so the CI can run, then it gets pulled into Gerrit for code review on a website with no visibility into the result of that CI.
I'd love to move all code review to Gerrit but I'd need to see something disrupt this space first.
Except Gerrit's not really all that great either! It just looks good when everything else has been dumbed down to parity with GitHub.
This start up seems like they are trying to sort of package and iterate on the value and power of Gerrit: https://www.gitcolony.com
This start up seems like they are trying to sort of package and iterate on the value and power of Gerrit: https://www.gitcolony.com
I literally came across this just now:
https://github.com/google/git-appraise
Although it may well be a decent Gerrit alternative - it still talks about interfacing with Jenkins.
https://github.com/google/git-appraise
Although it may well be a decent Gerrit alternative - it still talks about interfacing with Jenkins.
> compared to the simplicity of modern tools like Travis (which uses YAML).
But is Travis really a tool or just a service?
Having just learned of Travis on this thread, I can't seem to find its source code.
Somebody else mentioned Drone (https://github.com/drone/drone ) which could be a proper Jenkins replacement.
But is Travis really a tool or just a service?
Having just learned of Travis on this thread, I can't seem to find its source code.
Somebody else mentioned Drone (https://github.com/drone/drone ) which could be a proper Jenkins replacement.
> Having just learned of Travis on this thread, I can't seem to find its source code.
https://github.com/travis-ci/travis-ci
I don't think there's a quick and easy way to set up your local version of Travis, but it does seem possible.
> But is Travis really a tool or just a service?
I would certainly argue it's more on the 'service' end of things. If I had to guess, most people who find hosting their code on Github acceptable (Travis only works with Github at the moment) also find using the Travis web service acceptable as well so there isn't a lot of incentive to use Travis as a locally installed tool rather than using their as-a-service offering.
https://github.com/travis-ci/travis-ci
I don't think there's a quick and easy way to set up your local version of Travis, but it does seem possible.
> But is Travis really a tool or just a service?
I would certainly argue it's more on the 'service' end of things. If I had to guess, most people who find hosting their code on Github acceptable (Travis only works with Github at the moment) also find using the Travis web service acceptable as well so there isn't a lot of incentive to use Travis as a locally installed tool rather than using their as-a-service offering.
Cool ideas here even though a lot of comments are pouring water on it.
Reminds me of this post by Ted dzubia where he uses makefiles for data processing [1].
I like reading about novel uses of tools other than their original intent.
1- http://widgetsandshit.com/teddziuba/2011/02/stupid-unix-tric...
Reminds me of this post by Ted dzubia where he uses makefiles for data processing [1].
I like reading about novel uses of tools other than their original intent.
1- http://widgetsandshit.com/teddziuba/2011/02/stupid-unix-tric...
Thanks sheraz - we certainly don't expect Jenkins to serve as core infrastructure for many of these problems long term.
The flexibility of Jenkins continually impresses me. Given :
1) Simple UI (for simple usecases) 2) Easy setup on single node or multi-node scenarios 3)Automation capabilities
Unfortunately there are some gaps that make it just enough of a pain to really take up in a Production 24x7 env.
1) In a distributed setup, there are very minimal node management capabilities unless you manually integrate with say Zookeeper or something.
2) the plugins for backing up and restoring configurations are "lacking" to put it politely
3) Very hard to change the Master machine in a master slave set-up
4)etc etc
So while Jenkins is a like the Swiss army knife for CI be careful that you don't take it to a (multi-node production) Gunfight - to stretch the analogy.
1) Simple UI (for simple usecases) 2) Easy setup on single node or multi-node scenarios 3)Automation capabilities
Unfortunately there are some gaps that make it just enough of a pain to really take up in a Production 24x7 env.
1) In a distributed setup, there are very minimal node management capabilities unless you manually integrate with say Zookeeper or something.
2) the plugins for backing up and restoring configurations are "lacking" to put it politely
3) Very hard to change the Master machine in a master slave set-up
4)etc etc
So while Jenkins is a like the Swiss army knife for CI be careful that you don't take it to a (multi-node production) Gunfight - to stretch the analogy.
This is actually my problem with Jenkins. It's great for running arbitrary jobs when triggered to do so. But it's super frustrating for managing many permutations of of the same class of jobs, with component tasks, permutations of different input, and orderly storage/display of output categorized on input. Or in other words: it sucks at building and testing software.
I'm almost to the point of using it just to manage generic "do X sort of stuff" tasks across many nodes, but the jobs that rely on an external system to run parameterized builds, and then the job then stores the results into the external system.
I'm almost to the point of using it just to manage generic "do X sort of stuff" tasks across many nodes, but the jobs that rely on an external system to run parameterized builds, and then the job then stores the results into the external system.
This is part of the problem I have with Jenkins. It does far too many things. In many organizations it seems to occupy an awkward position between dev and production, where it's used for both and so ends up as the most fragile part of a production pipeline. Its config is very awkward, and partly as a result of that I've never seen it properly backed up or even a staging instance for testing changes, yet alone blue/green instances.
So I prefer to have clear segregation. Jenkins as a build tool only. Rundeck for deployment. If I had a big need for scheduling, I'd want a dedicated system for doing that too.
So I prefer to have clear segregation. Jenkins as a build tool only. Rundeck for deployment. If I had a big need for scheduling, I'd want a dedicated system for doing that too.
While not the best answer, I've used Jenkins extensively in the past for some rather creative purposes. Besides the standard "watch a SCM repo and react" type functions I've used it in combination with libvirt to spawn VMs self service for users (as well as coordinate the accounting and access control around this), watch an etag for a file to change in object storage and react, etc. I agree with some of the other commentary in that it's most useful aspects are around not using it with a traditional CI pipeline.
We use jenkins for CI, but only on our QA server. I haven't found a way to do CI without giving Jenkins full credentials on our server.
Jenkins is very powerful, but I would not trust it (or any of the myriad plugins we have installed) to not have security holes.
Jenkins is very powerful, but I would not trust it (or any of the myriad plugins we have installed) to not have security holes.
Yep. Not long ago a remote code execution hole in Java was found. Fortunately Jenkins was quickly patched: https://jenkins-ci.org/content/mitigating-unauthenticated-re...
At my previous employer I used Jenkins with build matrices quite heavily. Lately I've been using TeamCity and we're paying for the enterprise variant. TeamCity has its share of flaws but FWIW it seems to handle more complex workflow tasks better.
In particular I like the flexibility around snapshot versus artifact dependencies, the APIs are decent (and you can do a lot of troublingly clever things if you invoke the API from within a build), and the metarunner concept seems strictly more powerful than the Jenkins equivalents, albeit with a somewhat steeper learning curve.
In particular I like the flexibility around snapshot versus artifact dependencies, the APIs are decent (and you can do a lot of troublingly clever things if you invoke the API from within a build), and the metarunner concept seems strictly more powerful than the Jenkins equivalents, albeit with a somewhat steeper learning curve.
We've been using TeamCity for many years, and recently evaluated Jenkins as a possible replacement. Stability was a huge concern and decided to stick with TC. I agree, TC does have its share of flaws, but it's way ahead of Jenkins in many areas.
I would be extremely wary of doing something like this considering that Jenkins stability is not stellar and bugs are aplenty. The codebase is fairly old and is littered with outdated code, failed refactorings etc. In the course of daily operation Jenkins is extremely susceptible to race conditions, potential memory leaks, and that is just core - without even running any plugins. Entrusting your company's critical infrastructure to it seems to me a bit dangerous.
Does anyone consider Jenkins as a replacement to CRON?
We use it to run a bunch of periodic report-building tasks. Compute internal metrics, archive support tickets, that sort of things. Compared to plain cron, we get proper tracking and history of the runs that failed, and a web UI to access the results (generated reports, for us).
I wouldn't use it for production-sensitive crons, but for administrative stuff, it works fine.
I wouldn't use it for production-sensitive crons, but for administrative stuff, it works fine.
Absolutely - we (i.e. engineering @ Simon) view Jenkins as crontab that's easier to maintain, to track historical runs, and to keep tabs on failures (with full logging).
We do at my work. It has much more robust options. You can keep multiple jobs from running at the same time if one takes too long. You can auto kill it after a time limit. You get a web UI for all logging from the job. You get great built in tools for auto deleting logs after a certain number of builds or time period. You get a very robust email solution for sending out alerts when jobs run or don't run or fail. It adds some complexity vs CRON but it has many more features.
it uses a huge amount of resources compared to cron, though, which is my biggest problem with this approach.
There's a nice little write-up on that subject: http://davedevelopment.co.uk/2015/06/04/scheduled-tasks-with...
We have tried that as it is easy to learn the status of the job and view logs. But had problems while running many jobs simultaneously. Some jobs will hang. So now we are using Cron + Monitoring External Jobs Plugin in Jenkins(https://wiki.jenkins-ci.org/display/JENKINS/Monitoring+exter...). This gives the performance of Cron and the above mentioned advantages of Jenkins.
Yes, We do. We primarly use it as Job scheduling engine. I have multiple teams asking for data extracts, and I fashioned Jenkins to run extracts both adhoc as well as fixed schedule basis.
The question appears to be "how can we abuse Jenkins today"? For small teams and orgs, Jenkins is fine, but scales horribly.
Totally agree. When you have critical systems driven from Jenkins jobs, and they're called all hours of the day, how do you manage to upgrade Jenkins without trashing all the running jobs? (A: pay CloudBees for the Multi-Master plugin).
Jenkins skimps on so many basic problems (cleaning workspaces for deleted jobs, resource requirements for jobs, API which can handle 10 requests a second, config store that doesn't require parsing hundreds of XML files on startup). Plus it has its own git implementation, which has unconfigurable checkout behavior (i.e. fetch all branches then checkout), which makes it a huge disk hog on busy repos.
The plugin system is half-baked: we have to check every new plugin (& version) on a test instance to verify that it doesn't interact horribly with any existing plugin.
When Jenkins triggers a downstream job & blocks on it before continuing, it logs the job it triggered, but not the build (until after the job completes). This makes watching a build in progress extremely tedious, and you have to hunt through the builds of the triggered job to find the relevant build. None of this is exposed directly by the API, your client has to derive it itself.
Jenkins skimps on so many basic problems (cleaning workspaces for deleted jobs, resource requirements for jobs, API which can handle 10 requests a second, config store that doesn't require parsing hundreds of XML files on startup). Plus it has its own git implementation, which has unconfigurable checkout behavior (i.e. fetch all branches then checkout), which makes it a huge disk hog on busy repos.
The plugin system is half-baked: we have to check every new plugin (& version) on a test instance to verify that it doesn't interact horribly with any existing plugin.
When Jenkins triggers a downstream job & blocks on it before continuing, it logs the job it triggered, but not the build (until after the job completes). This makes watching a build in progress extremely tedious, and you have to hunt through the builds of the triggered job to find the relevant build. None of this is exposed directly by the API, your client has to derive it itself.
This article feels like it's from years ago. Using Jenkins today is an anachronism - poor UI, poor configuration, poor scalability, poor distribution. My current employer has poured so much well intentioned effort into Jenkins' black hole. Never again.
What are you going to use instead?
I've been meaning to take a closer look at http://concourse.ci/ at some point. It looks like it could be a viable alternative to Jenkins for many things.
Migrating to Travis and a SWF-based solution, all driven by versioned configuration and code.
In addition to CI, we use it for all of our Selenium testing and for jobs that would otherwise just be cron jobs. The UI for logging and built in email alerting when jobs fail are invaluable.
I hope your Jenkins server is well protected, if you're running so many things through it. One [JVM] to rule them all.
The Multi-Configuration Project abstraction (IE build matrices) is clunky and the plugin ecosystem doesn't respect it well (eg. the Gerrit plugin is extremely popular but very brittle here). So you wind up with O(n) projects anyway and still needing to copy and paste configuration among them.
Also Jenkins configuration itself is pretty nuts - settings splattered all over the web UI, backed by XML - compared to the simplicity of modern tools like Travis (which uses YAML).
And Jenkins' UI I would definitely categorize as typically-poor open source UI, having evolved and grown more complex over many years with no strong guiding vision.