In 2013, it was revealed that Spain’s first submarine in decades, the S-81 Isaac Peral, was overweight by nearly 100 tons due to a decimal place error in an early design calculation. This mistake would have made the submarine unable to surface once it submerged. Adding additional length to the sub seemed like a reasonable solution until 5 years later when Spanish authorities discovered that the Isaac Peral was too big for the port of Cartagena, where it’s currently being built. As my grandfather would have said… “measure twice, cut once.”
When planning large scale code changes, it’s exciting to get directly to the business of writing the automation with an OpenRewrite recipe that’s going to immediately do the work. We’ve found over time, though, that first building the recipe to focus on identification of all the potential edit sites is time well spent. It gives us a full impact analysis of what we are about to do before we do it. Real-world code is complex and nuanced. Remember, “measure twice, cut once.” Without a proper accounting of this nuance, the transformational recipe would not be as accurate as we wish.
Curiously, this means that many recipes start their life as search recipes, and evolve into transformational recipes. In the latter stage, the transformation serves both purposes (after all, why search if you can fix?). In the earlier stage, the recipe informs the implementation of what later becomes the transformation. And so it is that Olga Kundzich had an inspired moment back in 2021 that generalized the application of Markers to any part of the Lossless Semantic Tree (LST) and OpenRewrite accidentally backed into code search (which later led to the innovation of the aggregate form of impact analysis, Data Tables).
Independently, Google identified “how an API is used” as the most common Code Search task—the ability to research the code to understand usage and learn by example:
“The most frequent use case—about one third of Code Searches—are about seeing examples of how others have done something. Typically, a developer has already found a specific API (e.g., how to read a file from remote storage) and wants to see how the API should be applied to a particular problem (e.g., how to set up the remote connection robustly and handle certain types of errors).”
Software Engineering at Google, Chapter 17, O’Reilly Media 2020
The Moderne SaaS Platform has been able to scale this important search experience ‘in platform’ but now we’re bringing it into the developer IDE, which is revolutionary for developers.
We’re introducing the Moderne IDE plugin for JetBrains IntelliJ—a new, powerful tool for developers to work at scale without leaving their flow. Expanding the code scope within the IDE across multiple (even 1000’s) of repositories enables developers to expand their mental model of the code in real-time and make more informed, accurate changes.
Sign up for Beta version of this plugin, and read on to learn more about the advanced coding you can do when you can be omnipresent in your codebase.
Challenges of IDE-based multi-repo search
Despite the fact that Moderne has long supported running a wide variety of search recipes at scale on tens of thousands of repositories and billions of lines of source code, the developer experience remained firmly “in platform” in a web UI and outside of the IDE. There is historical precedent for this as well. Google couldn’t imagine how they could deliver multi-repository search in the IDE either:
“…the Google codebase is so large that a local copy of the full codebase—a prerequisite for most IDEs—simply doesn’t fit on a single machine. Even before this fundamental barrier is hit, there is a cost to building local search and cross-reference indices for each developer, a cost often paid at IDE startup, slowing developer velocity. Or, without an index, one-off searches (e.g., with grep) can become painfully slow. A centralized search index means doing this work once, upfront, and means investments in the process benefit everyone.”
Software Engineering at Google, Chapter 17, O’Reilly Media 2020
It honestly took a long time, and a lot of repetition watching developers trying to develop refactoring recipes with this two-step approach, for us to see how to connect the “where” of recipe development to the “where” of executing that recipe.
After all, Google said it was impossible, and isn’t that practically as unassailable as Newton’s Third Law. So what has changed?
It’s been staring us in the face for a while. We have been practicing and training Moderne customers to mass produce LSTs and store them in their artifact repositories as binary artifacts. One of the key things Moderne adds on top of OpenRewrite is the ability to serialize and store LSTs as artifacts in your artifact repository, allowing you to effectively bear this computational burden centrally and once for all engineers in your organization. We have been using these artifact-repository-sourced LSTs to deliver a multi-repository experience in the form of the Moderne SaaS now for a while—why not in the IDE?
Researching an API usage at large with the Moderne IDE plugin
Let’s look at that most common search use case. Suppose I am trying to change the API of a method Preconditions.check. I don’t understand how to use it. I’m typically looking at one occurrence of it. I want to know more because I think I might want to use it.
An actual occurrence of the use of an API is the jump-off point for the “Find Usages” action that developers use all the time in the IDE. But normally, “Find Usages” finds usages in just the code of the repository that is currently open in the IDE (or at best in the repository that is open currently plus its binary dependencies). That’s great when the repository I am editing or its (sometimes significant) set of transitive dependencies happens to use the API in a way that is instructive. But frequently enough, it isn’t. I think I physically tense up in those moments hoping that one of the usages that are captured are sufficient for helping me generalize my understanding. Because if it is, then I’m going to stay in flow. And if not, I’m going spelunking.
As Google pointed out, “Find Usages” works as well as it does in the IDE because the IDE has indexed the call sites of the repository and its dependencies. What about all the other usages of that API in repositories that are in my scope of responsibility (or in the scope of my larger organization’s responsibility) but don’t have a binary dependency relationship with the repository I have open?
This is where it would be amazing to have a wider scope to search with, but that has never existed. Despite Google’s attempt later in that chapter to convince themselves and everyone else that a separate web UI was really the best place (conveniently because they felt a technical limitation that required a web UI), I really don’t want a totally separate UI for this. I don’t want to learn another DSL I’ll forget tomorrow. I just want to click “Find Usages” on the example I already have staring at me.
Now, with the advent of the new Moderne IDE integration, engineers can configure their scope of responsibility, regardless of which part of that scope of responsibility they happen to be editing. This configuration is IDE-wide and not project-specific. This set, which we call the “multi-repo,” serves as the scope by which searches can be performed on code that isn’t part of the IDE window that I have open or its dependencies and complements those existing search scopes. The multi-repository could contain dozens or even thousands of repositories.
The IDE isn’t responsible for indexing the contents of these repositories, but rather Moderne tooling will download from your Artifactory the pre-existing LSTs for each of them. Since the LST is really just a binary representation of the text of the source code plus everything the compiler knows about that code (plus some other metadata), it is a more than capable index to feed multi-repository search.
If you don't have a centrally managed Moderne solution in your organization, you can still build the LSTs outside of the IDE on your machine using the Moderne CLI. The key is that the IDE is not responsible for building indexes. That is done once outside the IDE.
And that multi-repository search is glorious. The Moderne IDE plugin is simply exercising existing OpenRewrite recipes for different kinds of searches and displaying the matched code in the IDE’s existing usage views that are already familiar. The only display difference is an extra layer of nodes in the usage tree that represent the repositories that each file with search results belongs to. All the other usage grouping facilities, opening in an editor, etc. works exactly the same as a normal IDE “Find Usages” action.
Creating a recipe from the Moderne IDE plugin for building and tuning code searches on the fly
We’ve gone even one step further. Because OpenRewrite added support for pattern-based matching in the form of transpiling Google Refaster templates to OpenRewrite recipes, we can actually select any expression in the code and find matching patterns!
A good example of this is finding all occurrences of a SLF4J logger that has primary argument concatenation of some string and an exception. By supplying the exception as a separate argument instead, we would get a much more useful log message containing the stack trace as well. It’s an easy mistake we’ve all made (luckily Copilot never makes mistakes right? right?!). The Moderne plugin adds a “Create OpenRewrite Recipe” menu option to the refactor menu that helps us identify and remediate this pattern:
All of this structure and the before template is filled out for us. We just need to decide what the “after” should look like. This template now works for any logger call that happens to look like this, independent of the name of the logger or the expression that provided a string in the first part of that concatenation (i.e., a string literal, a method call). If we remove the “after” method, then the recipe acts as a search.
How to get started with the Moderne IDE plugin
To try out the Moderne IDE plugin, sign up here. Watch this video to see the plugin in action:
We’re just at the beginning of this brave new world, but you can be sure that more context-specific searches and refactoring operations are on their way. We’d very much welcome your feedback on what operations you want on multi-repos.
For questions and feedback on the Moderne IDE plugin, please reach out on the OpenRewrite Slack.
