Faster Maven Builds(blog.frankel.ch)
blog.frankel.ch
Faster Maven Builds
https://blog.frankel.ch/faster-maven-builds/1/
35 comments
Laughs in horrible Gradle Kotlin DSL build times.
Yeah, don't do that.
If it wasn't for Google's sponsorship it would have been gone by now.
Android is probably the only platform whose conferences to this day have a sessions about improving build times, thanks Gradle.
When doing Windows CE, Pocket PC, Windows Phone 7, WinRT, Symbian, iOS, I never needed a gaming rig as development workstation.
Android is probably the only platform whose conferences to this day have a sessions about improving build times, thanks Gradle.
When doing Windows CE, Pocket PC, Windows Phone 7, WinRT, Symbian, iOS, I never needed a gaming rig as development workstation.
It is android’s fault, not gradle’s.
When Gradle requires a background daemon consuming at least 2GB and SSD disks to achieve usuable builds, that lies on Gradle not Android.
Why exactly? It stores caches, that gets used for the build process. How else should it operate?
The problem is the amount of dependencies/build tasks, not the tool managing those.
The problem is the amount of dependencies/build tasks, not the tool managing those.
The excuses with Gradle are always the kind of "you're holding it wrong".
Thankfully outside Android, I never have to put up with it on regular Java projects.
Thankfully outside Android, I never have to put up with it on regular Java projects.
Which would show you how is it a very fast build tool, but if you prefer not even learning about it, do stay in your ignorance.
Yet again the "holding it wrong" and "you're an ignorant" argument that always comes up from Gradle people.
Yeah, it is very fast, when one has all the knobs to help Gradle pretend it is fast.
I have been setting up Java CI/CD pipelines since 2005, have used all major CI/CD products in the Java world, including your beloved "blazing fast" Gradle.
Yeah, it is very fast, when one has all the knobs to help Gradle pretend it is fast.
I have been setting up Java CI/CD pipelines since 2005, have used all major CI/CD products in the Java world, including your beloved "blazing fast" Gradle.
[deleted]
Gradle is one of the fastest build systems out there (certainly the fastest for Java projects because the tight integration with the compiler), given people know what they are doing. Most often than not they will write build logic as configuration which runs each time.
But when configured properly, it actually has parallel builds, with fine-grained build dependencies, so only the actually changed classes will be recompiled.
But when configured properly, it actually has parallel builds, with fine-grained build dependencies, so only the actually changed classes will be recompiled.
Java build systems are too archaic and complex at this moment. They should be purged and replaced with new simple tiny tool, compiled with GraalVM. Java could be compiled extremely fast, javac does not do any fancy optimizations, it just spews bytecode directly derived from AST. 99% of maven/gradle run is mindless abstraction overhead.
You are coming from a good place but... please god no, not another build system.
They are complicated because they do a fuckton of things. Please don't give me another proof-of-concept build system which then turns out to do 5% of what I needa and have no plugins available.
Gradle is fine, just use it. It's not the best. It's OK. If you hate it, use Maven. It's also OK. Just copy-paste some answers off stackoverflow and move on... we'll all be better off.
They are complicated because they do a fuckton of things. Please don't give me another proof-of-concept build system which then turns out to do 5% of what I needa and have no plugins available.
Gradle is fine, just use it. It's not the best. It's OK. If you hate it, use Maven. It's also OK. Just copy-paste some answers off stackoverflow and move on... we'll all be better off.
I sometimes contribute to a jMonkeyEngine, it is an open source game engine written in Java. This was my workflow in the past: fork > clone > added engine feature > deploy locally > added local deployment to my game's Gradle build file to test the newly added feature.
I recently found out that I can just use Github cli to clone from upstream, and add one line in my game's Gradle build file to tell Gradle to use the cloned folder instead of looking at local cache. In Gradle world it is known as Composite build and, it has really make me like Gradle despite their convoluted build files.
I recently found out that I can just use Github cli to clone from upstream, and add one line in my game's Gradle build file to tell Gradle to use the cloned folder instead of looking at local cache. In Gradle world it is known as Composite build and, it has really make me like Gradle despite their convoluted build files.
Maven is great as long as you buy into their philosophy of how builds ought to work - phases and actions happening at a particular phase.
If your build has really custom stuff, and you cannot fit your custom work into those phases (e.g., it interferes with some other custom stuff), then you have a maven problem that's hard to solve. One usually resolve this by writing a maven plugin, and that kinda sucks. But hey, at least if you do it, you can contribute this to the whole plugin ecosystem and thus, improve it for somebody else.
If your build has really custom stuff, and you cannot fit your custom work into those phases (e.g., it interferes with some other custom stuff), then you have a maven problem that's hard to solve. One usually resolve this by writing a maven plugin, and that kinda sucks. But hey, at least if you do it, you can contribute this to the whole plugin ecosystem and thus, improve it for somebody else.
Usually the solution is to break your project into smaller modules, each of which can follow the normal lifecycle, and express your special-case requirement through module dependencies (e.g. maybe you have an integration test module).
IME the benefits of fixed phases outweigh the costs: any developer can pick up any maven project and immediately know how to build it, how to run the tests, and so on. A lot of other build systems are really bad for knowing where to start; allowing arbitrary task names and arbitrary relationships between what runs when just ends up with projects having pointless differences between what they call things - do I need to run "gradle build", "gradle install", or what? (and similarly for sbt etc.)
IME the benefits of fixed phases outweigh the costs: any developer can pick up any maven project and immediately know how to build it, how to run the tests, and so on. A lot of other build systems are really bad for knowing where to start; allowing arbitrary task names and arbitrary relationships between what runs when just ends up with projects having pointless differences between what they call things - do I need to run "gradle build", "gradle install", or what? (and similarly for sbt etc.)
> If your build has really custom stuff
That's the biggest problem. Nearly every developer thinks their build has *really* custom stuff whereas it's wrong in 99% of the cases.
Then comes Gradle with its so-much-vaunted flexibility and deliver you exactly what you think you need. And who is going to maintain the clusterfuck of spaghetti code that your build has become?
That's the biggest problem. Nearly every developer thinks their build has *really* custom stuff whereas it's wrong in 99% of the cases.
Then comes Gradle with its so-much-vaunted flexibility and deliver you exactly what you think you need. And who is going to maintain the clusterfuck of spaghetti code that your build has become?
Don't. Just don't.
It's already a clusterfuck in JS land with grunt/gulp/webpack/TSC/rollup/parcel/snowpack/esbuild/rome etc, do not bring that madness to java as well.
It's already a clusterfuck in JS land with grunt/gulp/webpack/TSC/rollup/parcel/snowpack/esbuild/rome etc, do not bring that madness to java as well.
that java madness has already happened...10 yrs ago!
maven is at the level of maturity where you don't need to even refer to it by it's version - it's just maven now (there's been two previous incarnations of maven, called maven 1 and maven 2). Then there's ANT, and there's a bunch of bespoke tools as well that i'm not too familiar with - ANT is still useful, but i would say that it plugins into the maven toolchain acceptably OK.
maven is at the level of maturity where you don't need to even refer to it by it's version - it's just maven now (there's been two previous incarnations of maven, called maven 1 and maven 2). Then there's ANT, and there's a bunch of bespoke tools as well that i'm not too familiar with - ANT is still useful, but i would say that it plugins into the maven toolchain acceptably OK.
Yeah we had this stable period, but now get ready for Maven 4. There will be breaking changes.
Could you provide examples of better solutions for other platforms?
I've been doing software dev professionally in 5+ languages for the past 20 years and speaking from experience java has one of the best build/dep management systems to choose from.
> Java build systems are too archaic and complex at this moment
I feel the opposite. For 99% of use cases, a plain maven pom file works. If you need something custom, you should think hard about why the existing workflow doesn't fit you before moving on. I think maven mostly gets hate from hip developers because it uses xml, but the tool itself works fine.
I've spent ages more getting simple js apps to build correctly. It "just works" in java, and has for ages (thus archaic).
I feel the opposite. For 99% of use cases, a plain maven pom file works. If you need something custom, you should think hard about why the existing workflow doesn't fit you before moving on. I think maven mostly gets hate from hip developers because it uses xml, but the tool itself works fine.
I've spent ages more getting simple js apps to build correctly. It "just works" in java, and has for ages (thus archaic).
Parallelizing builds and tests is definitely useful. Not only does it make things run faster but it can force concurrency issues to the surface that you would otherwise not find until after you deploy the code. This works especially well with integration tests that each might take a few seconds to run. Most of that is IO so you can actually utilize more threads than CPU cores here and still expect a speedup.
However, it requires designing your tests such you can do this. One key trick is using randomized data and test fixtures. Another one is avoiding the overhead of initializing databases, etc. over and over again. Of course, all that stuff is useful in just about any language. But some languages make this easier than others.
However, it requires designing your tests such you can do this. One key trick is using randomized data and test fixtures. Another one is avoiding the overhead of initializing databases, etc. over and over again. Of course, all that stuff is useful in just about any language. But some languages make this easier than others.
100% agreed re. bringing concurrency issues to the surface.
Another nasty one I encountered a couple years ago: After switching the build servers to using ramdisks, the app started breaking in subtle ways. Turns out there were two conflicting dependencies, but building on a server using ext4 and with the specific number of concurrent threads caused the "right" one to appear first the .jar's central directory.
Gotta add filesystem directory entry ordering to the list too.
Another nasty one I encountered a couple years ago: After switching the build servers to using ramdisks, the app started breaking in subtle ways. Turns out there were two conflicting dependencies, but building on a server using ext4 and with the specific number of concurrent threads caused the "right" one to appear first the .jar's central directory.
Gotta add filesystem directory entry ordering to the list too.
Nice write-up, and good to see that maven also evolves. The use of the daemon is really significant for the build time. Is there any clue what would take 90 seconds everytime you execute the maven process? Is it the creating of the build graph and its dependencies? What would happen when I change my dependencies or module structure, does the daemon still stay on 30 seconds?
For my projects, building the jar-with-dependencies takes forever. And as there is a zip file from the previous build just sitting there, it’s a real shame the system isn’t clever enough to examine and reuse vast swathes of it.
A jar-with-dependencies is not really working with the grain of maven - you're meant to use the "exploded" version and use maven to run with the right classpath. Caching and reusing build outputs sounds clever, but my experience with Gradle is that the cache wastes far more time than it ever saves, either by taking longer to search the cache than to rebuild the thing, or by causing nondeterministic misbuilds that take days to debug.
Can't you just 'install' it then make it a dependency itself?
It would be great if the mvnd authors provided some instructions about how to set up mvnd to be used in an IDE like Eclipse or IntelliJ on the github repo [0]
[0] https://github.com/mvndaemon/mvnd
[0] https://github.com/mvndaemon/mvnd
Imho, mvnd is a step in the right direction.
Next is how to make maven cache work more reliably and aggressively, as maven is still too slow at incremental build: A zero-change build can take seconds, while Bazel finishes almost instantly.
Next is how to make maven cache work more reliably and aggressively, as maven is still too slow at incremental build: A zero-change build can take seconds, while Bazel finishes almost instantly.
I think it’s more of the fault of needing to invoke a number of plugins bound to different cycles, rather than the compiler deciding if it needs to recompile files or not. I don’t believe Maven’s domain model has any concept of incrementalism or caching - only the compiler does.
Step 1: if you use Windows, just use anything else.
Even a Linux VM running on top of the same Windows will probably be faster.
(I have not tested exactly that but I have tested in dual boot configuration and I have tested .Net cli apps and seen them running faster under a Linux VM on top of Windows than it did natively on Windows.)
Even a Linux VM running on top of the same Windows will probably be faster.
(I have not tested exactly that but I have tested in dual boot configuration and I have tested .Net cli apps and seen them running faster under a Linux VM on top of Windows than it did natively on Windows.)
We switched to buildr. Build scripts massively simplified and build times drastically reduced.