“Call me Ishmael” (this is your hint that this intro is going to have the intricate or excruciating detail of Moby Dick).
In OpenRewrite, we heavily use nullability annotations to provide developers (and their IDEs) with detailed hints about our expectations on how to use our APIs. We generally adopt non-nullability by default and explicitly mark fields, type parameters, method return types, etc. as “nullable” whenever they can hold or produce null values.
In a perfect world, these annotations would have been incorporated into the language itself, and indeed an effort was made to include them in JSR-305. But the JSR was never completed, unfortunately. As one of the Chief Architects on the Java Platform Group at Oracle says:
Later, the famous (or infamous depending on your opinion) JEP-200 Project Jigsaw introduced modularity into JDK 9, along with a huge litany of breaking changes from “javax” to “jakarta” packages and dependency coordinates. The result was that “JSR-305 annotations” (henceforth we call them this in shorthand even though technically JSR-305 never completed) became unusable in the Java module system because they shared the same package javax.annotation as other annotations in the core JDK. Jigsaw did not permit split packages, an opinion it in turn inherited from OSGi.
Nevertheless, IDEs still looked to the JSR-305 annotations as a source of truth for their own nullability analysis and hinting. As framework authors (e.g., Spring, Micrometer, OpenRewrite), we were caught in the middle. We wanted simple nullability assertions to make our communities’ lives easier, but we didn’t want a dependency on annotations from a library like Guava for general dependency hygiene reasons. So several of us used a little known technicality of the Java module system: creating our own nullability annotations that themselves were annotated by the JSR-305. These are called meta-annotations. Amazingly, IDEs see the JSR-305 annotations as being applied through the meta-annotations and so we get to have our cake (nullability annotations) and eat it too (no Jigsaw problems).
We’re leaving this messy world behind and moving to JSpecify in OpenRewrite, which we feel is finally stable enough and has enough industry participation that we don’t need to maintain our own anymore.
The impact analysis: Finding JSR-305 meta-annotations
First, it would be nice to identify all the uses of OpenRewrite’s internal JSR-305 meta-annotations, just so we’re sure we understand what the usage patterns really look like. Looking at our package, all of the annotations have the word “Null” in them somewhere, so we can find them all at once with OpenRewrite’s find types recipe, using the pattern org.openrewrite.internal.lang.*Null* to capture all of them at once across all the repositories.
You can see why we wouldn’t want to do this change by hand. Just in the OpenRewrite repository alone we have uses of our nullability meta-annotations in 683 files!
In addition to marking every occurrence, the recipe also produces a data table. Very quickly we understand the prevalence of every annotation in use on not just one but all of these repositories. It’s easy to see why doing this by hand would be super boring (5,159 call sites need to be changed after all).
Planning the JSpecify migration
Unsurprisingly, JSpecify’s annotations are equivalent in behavior to the OpenRewrite meta-annotations, so really we’ve just got a mapping exercise here. In some cases, the types are named the same but with different packages. of course:
- @NonNullApi -> @NullMarked
- @Nullable -> @Nullable
- @NonNull -> @NonNull
There doesn’t seem to be a corollary to @NonNullFields. Running find types again just for this type allows us to see how that is used.
What we find is that @NonNullFields and @NonNullApi often go together, with the former only applying to fields, and the latter to methods, types, and so on.
Unfortunately, JSpecify doesn’t yet have an annotation for fields, and that’s ok. We’ll just update the definition of @NonNullFields to be a meta-annotation on JSpecify’s @NonNull instead of JSR-305 and leave the references to @NonNullField for the foreseeable future.
There is one last bit here which is to add the JSpecify dependency itself. We have a choice: do we want to add a direct dependency on JSpecify when it is directly used or only if it is directly used and is not transitively included already?
Most OpenRewrite repositories have rewrite-core included either directly or transitively, and we’ll be adding JSpecify itself as a direct dependency of rewrite-core. Personally, I’m comfortable in this case allowing JSpecify to come in transitively for all the dependency recipe repositories, though I could also respect a contrary opinion that it should be specified everywhere as a direct dependency.
Either way, luckily the “Add Gradle or Maven dependency” recipe is flexible enough to support our opinion. First we specify the group, artifact, and version (the special latest.release version will automatically pick up whatever the latest release version is in the artifact repositories available to each repository):
The “Only if using” option allows us to constrain the recipe to only add the dependency if one of the JSpecify annotations is demonstrably in use:
And the “Accept transitive” option is the last bit which controls whether the dependency is added if it already exists transitively.
There are many other options on this recipe than these, and together they represent many of the decisions that we as engineers make when we’re manipulating dependencies.
Doing the initial manual work
We will deprecate any OpenRewrite that has a corollary in JSpecify. Also, we’ll make even the now deprecated annotations a meta-annotation of JSpecify instead of JSR-305. And add the dependency to rewrite-core directly.
We don’t need to automate this part, we just need to make some hand-edits to pave the way for the mass migration of all the use-sites to follow.
This work was done as part of this PR. As one example, we are deprecating our own meta-annotation, rewiring it to JSpecify (so that any of OpenRewrite’s users implicitly also start using JSpecify), and just otherwise leaving it around for a while.
The CI builds of all the downstream OpenRewrite recipe repositories use snapshots of rewrite-core, so once this PR is merged and its CI build is run, we would expect that the dependency-adding recipe wouldn’t have much work to do.
A new custom recipe is required
The OpenRewrite @Nullable annotation can be specified on either a field or a type. In the following two examples, the first application of the @Nullable annotation is applied to the field, the second to the type.
When the type is an outer class, these are indifferentiable in syntax and the compiler accepts them both:
While I suspect most Java developers don’t even know it is possible to apply an annotation to an inner class identifier like we see in the type annotation example above, it is quickly becoming the fashionable style. IntelliJ autocompletions favor type over field annotations, for example, when both are possible.
In fact, this pattern is so fashionable that JSpecify doesn’t even permit its equivalent of the @Nullable annotation to be specified on a field. So since it is more strict, we have to first write a recipe to move the original OpenRewrite nullable annotations from field annotations to type annotations.
It’s not uncommon for migrations to involve some custom recipe development work on top of using the basic building blocks of "Change type" like we did so far.
When combined with the "Change type" recipes we now have a more complex behavior that does both:
Now the fun part: mass migration to JSpecify
The foundation has been laid now to migrate. We can assemble the steps that we laid out above into an OpenRewrite declarative recipe:
I used the Moderne Recipe Builder to make this easier:
Running the recipe on all 44 repositories leads to very satisfying results. Thousands of call-sites updated, and I’m ready to mass PR. Of the 44 OpenRewrite repositories under management in Moderne, 39 of them require changes. In one operation, I can create PRs for all of them. The primary author on the commit to the branch that this PR is created from will be me. After all, mass change shouldn’t be driven by a bot, but by a human. A very, very efficient human.
The Moderne platform lets me see all the PRs in one view, with a column indicating their status. Let this sink in for a moment—this is a multi-repository view of a change.
That’s it. Over 5,000 use sites edited across 39 repositories. All of the commits are attributed to me, the engineer responsible for promulgating this change (if anything goes awry folks will know who to ask). When I started this blog post, I honestly thought this was going to be a very trivial migration. As often happens, I found complexities in the position of annotation types that made this more complicated than I knew. I know of no other refactoring technology that could have done it accurately at scale. Parts of it, maybe, but not the whole thing. And certainly not with the same degree of confidence. The general strategies that we employed are the same even for those that are much more complex than this. We study a problem at scale, plan a migration to the actual uses we’ve identified, execute the plan at scale, and monitor it to completion—all in one platform designed for multi-repository operation. Contact us to learn more about how you can tackle those tough and tedious migrations.
