Open source libraries distill the collective wisdom and toil of untold legions of software engineers into powerful, convenient libraries. In most circumstances you would be crazy to forgo dependencies. But with dependencies come dependency problems. Each library has its own cadence for breaking changes, vulnerability disclosure, and thankless lone maintainers unexpectedly getting run over by buses. Every dependency you take is like an alert waiting to pop up telling you that all your plans for the day (or week, or more) are canceled because now you have to deal with an urgent issue. Companies end up spending huge amounts of money, talent, and tools just to try and quarantine dependency disruption.
An interesting thing about the problem space is that two wildly different libraries can cause the same sorts of headaches regardless of their actual domain. You’ll get many of the same problems regardless of whether the libraries themselves deal with HTTP communication or trigonometry or cryptography or anything else, including:
- Breaking code changes
- Security vulnerabilities
- Transitive version conflicts (e.g., Diamond dependency problem)
- Lack of updates (e.g., a maintainer decides to open a craft brewery rather than respond to issues from ingrates)
- Bad updates (e.g., new owners full of creativity and ambition… when it comes to extracting revenue from you, the user they believe is locked-in to their technology)
The more dependencies you have the more exposed to all of these potential problems you are. This is where OpenRewrite and Moderne step in with unique capabilities for managing dependencies—direct and transitive—removing unused dependencies, performing upgrades, and more. All the tools we need to automate our way out of dependency hell.
Finding and fixing vulnerable dependencies
When an explosive vulnerability is dropped like a bomb into your sprint, the first step is to assess the damage. Large software development organizations can have thousands or even tens of thousands of repositories, each with their own graph of hundreds of dependencies. Manually figuring out your exposure could itself be a lengthy and expensive process, during which time bad actors can continue to exploit the prolonged problem.
One of OpenRewrite’s most popular recipes will scan through all of your dependencies, direct and transitive, checking for corresponding entries in the Github Security Advisory database. That database includes the National Vulnerability Database, which is the source of truth for those CVE numbers we all know and love. Whenever you are on a potentially vulnerable version that has an available fix only a patch version away, it will suggest an upgrade. Suggesting patch version bumps is like picking the low-hanging fruit off the security tree: The library maintainers have asserted that there are no breaking changes accompanying the fixes.
This screenshot shows a change suggested by “Find and Fix Vulnerable dependencies” in a Gradle-built project where Spring 6.1 is in use. The Spring dependency management plugin necessitates that a custom resolution strategy be used to manage the versions of transitive dependencies. In other Gradle projects the recipe would suggest different changes. Recipes can be sensitive to context and designed to handle a variety of situations.
I ran this recipe on a selection of over 200 repositories, which collectively had over 8000 results when comparing their dependency graphs to the vulnerability database. Imagine trying to fix all of that by hand—you’d never finish!—which is exactly the situation our industry is in as a whole and why it is so vital to bring automation to bear on problems of this scale. The full results of the recipe can be seen in its data table:
This recipe produces a data table showing every dependency in use that has an entry in the database. I’ve highlighted in green rows wherein patch version bumps alone are sufficient to get the fix. Data tables are available as CSV, Excel, and JSON so as to be convenient to post-process or otherwise analyze in your tools of choice. One such tool is Jupyter notebooks, a popular open source suite of data analysis and visualization tools. Moderne can embed these visualizations to provide easy, reusable lenses through which to see and understand what the data wants to tell you.
For example this visualization shows how big of a version bump is required to get from the dependency versions in use to those with fixes. In this case the largest column—almost half of the total—is the column for patch version upgrades. The recipe itself will make patch version upgrades for you.
Managing dependencies—insights and upgrades
I once maintained a library used by other teams internally. One day I got a message from a consumer of our library saying that they were getting errors out of our code. They insinuated—in that faux-polite, “per my last email” sort of way—that we had messed up. The stack trace clearly showed our code was involved, yet we had made no change at all. The library was in maintenance mode. However, our library depended on another library, in this case the Jackson family of serialization tools, and so did an unrelated third party library also in the dependency graph. That unrelated third library had upgraded to a newer version of Jackson, and Gradle’s “newest wins” conflict-resolution strategy had put a newer version of Jackson on the classpath than we had tested with. A classic “Diamond dependency problem.”
And if one consumer of our library was running into a diamond dependency problem I knew it wouldn’t be long until everyone in the company was. This was before OpenRewrite and the “Dependency insight for Gradle and Maven” recipe, so it was hard to find all of the places Jackson and our library was used across all repositories. Making the code change to our library to be compatible with the new Jackson release was the fastest, easiest part of the whole process. Figuring out the impact, contacting people, and otherwise propagating the fix was 9/10ths of the work.
Now when a problem like this comes up it takes only seconds to run the Dependency insight recipe and use its search results, data table, and visualizations to see the spread of versions in use for a particular module across all repositories and take action accordingly. Indeed, it takes only seconds to find exactly which versions of exactly which modules are in use in exactly which repositories, visualized in a handy violin chart.
Then armed with that information it’s a simple matter to use a recipe like “upgrade dependency” or “upgrade transitive dependency” to propagate a dependency upgrade organization-wide. If code changes are required, a recipe to effect the required changes can be bundled together with the dependency upgrade.
Researching dependency issues with type-aware code search
Sometimes a dependency problem manifests in code, when a particular method or class is used or a particular data access pattern exists. For these cases knowing what dependency version is on the classpath is necessary but not sufficient. Fortunately, recipes can be used to search code with accuracy and precision unknown to textual search tools. Resolving the problem with Jackson versions in the previous anecdote required changing our configuration of Jackson’s ObjectMapper class. The dependency insight recipe is great for showing me which repositories use Jackson, but not where within those repositories it is used, or if the piece of it of interest is there.
With OpenRewrite and Moderne it is a simple matter for me to use the Find types recipe to search any number of repositories for where this particular type is in use:
In only a few seconds I’ve searched dozens of repositories, identifying all of the relevant call sites with perfect accuracy. Now you might not be impressed, reasoning correctly that it isn’t too hard to do a text-based search for the term “ObjectMapper.” Of course you would be right that it is easy to search for that term—and thus would your naivety and hubris betray you, as such a text search will miss relevant subclasses like JsonMapper or XmlMapper. A foible that type-aware search fully avoids:
In the future, your dependencies will update themselves
The tools I’ve presented to you today are useful and powerful, but they’re only a sketch of a future where your dependencies manage themselves. Imagine the next big CVE comes out with a recipe that makes all the necessary changes to upgrade your code and upgrade your dependency versions. By the time you get a panicked call about it, the fix is already rolling out to production. Or when your programming language or key libraries introduce a big breaking change, this doesn’t have to break your code if it comes with an upgrade recipe. With OpenRewrite recipes distributed via the Moderne platform we, collectively as software engineers, can make it happen.
Contact Moderne if you want to get your dependency management on the right track.
