Rf: an experimental refactoring tool for Go(pkg.go.dev)
pkg.go.dev
Rf: an experimental refactoring tool for Go
https://pkg.go.dev/rsc.io/rf
51 コメント
It would be great if go.mod file’s replace directive also supported a replace that applies to all downstream users (url to url , not just url to local path) This would allow us to maintain simple forks (ones where you are only fixing a few lines and awaiting PR approval to upstream) but still don’t want to wait for the whole cycle to finish for your own internal company dependencies. I feel this is better than changing all imports.
If you are truly forking with no intention to merge back, isn’t the manual find-and-replace by IDE just fine?
If you are truly forking with no intention to merge back, isn’t the manual find-and-replace by IDE just fine?
The replace directive does allow you to use a different import path, not just a local path. Unless there's something I'm missing about your use case?
https://golang.org/ref/mod#go-mod-file-replace
https://golang.org/ref/mod#go-mod-file-replace
It's important to note that it still doesn't apply to downstream users (those adding your package to their go.mod), only to those building your package directly. I think this decision makes sense, so this is not a complaint.
Can you elaborate? We use `replace` everywhere in our codebase for modules that we've forked + modified.
Assume module A depends on module B, and module B has a dependency on module github.com/C, and a replace rule for github.com/C => company.internal/C.
When you build module A, it will pull module B and module github.com/C, instead of B and company.internal/C, unless A has its own replace directive for C.
When you build module A, it will pull module B and module github.com/C, instead of B and company.internal/C, unless A has its own replace directive for C.
Oh I never knew! Thanks. That does simplify a lot
The problem I had was that I was forking a monorepo package, so I had to update its internal dependencies to point to the new place. It's possible I missed something, but I don't think I was able to do this with go.mod.
And yes, manually doing the rename "is fine", but I don't think that has any reason to be stated in a discussion about a refactoring tool.
And yes, manually doing the rename "is fine", but I don't think that has any reason to be stated in a discussion about a refactoring tool.
> I don't think that has any reason to be stated in a discussion about a refactoring tool.
Find and replace is the baseline, and for many refactorings is insufficient and fragile, however in replacing a stable, unique, and isomorphic token it is basically as good as it gets, so I think it is relevant. Its probably the least useful thing a refactoring tool could do.
Find and replace is the baseline, and for many refactorings is insufficient and fragile, however in replacing a stable, unique, and isomorphic token it is basically as good as it gets, so I think it is relevant. Its probably the least useful thing a refactoring tool could do.
I am not sure. You want to make sure that you are only modifying import statements and those can’t necessarily be identified from just a single line (else you might change an errant URL in a string literal), and you also need to move the files themselves, and update the package line in the files. So sure, you can do it yourself, but why are we building a refactoring tool at all?
Not to mention if the package name itself changes...
I used to joke at work that the fastest, simplest way to get promoted to a staff engineering position was to finish a 1 yr refactor.
Getting existing codebases to do new things is hard, and most of enterprise software engineering is basically "rework this API (in the general sense, not web-API sense) for a new product use case".
This tool seems to borrow the philosophy of Go's keeping the language simple, with the tradeoff of a more complex (but fairly well supported) ecosystem to account for missing language features.
IME, Golang (some would say paradoxically due to a lack of generics) is one of the best languages to refactor because of:
- Forced error handling semantics so that unhappy paths are easily enumerated
- Tooling like `rdep` to quickly grok the impact of a package refactor
- String templates/codegen being a first-class "blessed" part of the ecosystem (i.e. you can find tooling by the Go team as a reference when writing your own)
Some would balk at things like codegen being how Go handles this, but for the iteration loops at most companies at scale (where a few tools to do codegen are written & maintained by one team and consumed by others) it works well. For paradigms where you want everyone to contribute to the tooling, it works slightly less well since the barrier of understanding the ecosystem is greater than just understanding the language. Nevertheless, it seems to be the right tradeoff for most enterprise use cases.
Getting existing codebases to do new things is hard, and most of enterprise software engineering is basically "rework this API (in the general sense, not web-API sense) for a new product use case".
This tool seems to borrow the philosophy of Go's keeping the language simple, with the tradeoff of a more complex (but fairly well supported) ecosystem to account for missing language features.
IME, Golang (some would say paradoxically due to a lack of generics) is one of the best languages to refactor because of:
- Forced error handling semantics so that unhappy paths are easily enumerated
- Tooling like `rdep` to quickly grok the impact of a package refactor
- String templates/codegen being a first-class "blessed" part of the ecosystem (i.e. you can find tooling by the Go team as a reference when writing your own)
Some would balk at things like codegen being how Go handles this, but for the iteration loops at most companies at scale (where a few tools to do codegen are written & maintained by one team and consumed by others) it works well. For paradigms where you want everyone to contribute to the tooling, it works slightly less well since the barrier of understanding the ecosystem is greater than just understanding the language. Nevertheless, it seems to be the right tradeoff for most enterprise use cases.
Doesn't that just mean that, while Go the language is simple, to become productive one must not only learn the language but an ecosystem that is bigger than the ecosystems of other languages?
I'm a Googler, and while that might be true at another place, unfortunately this is not true at Google. Refactoring/Tech Debt/Eng Excellence work will not get you promoted pass L4 for the majority of teams. In some team, it will get you to L5, but that's rare. Getting to L6 based on "1 yr refactor" is... simply extraordinary. I haven't seen or heard of one case in 7 years here.
I'm pretty sure that the comment you replied to was talking about "the general rule" and not "the exception to the rule." you don't need to call out that your specific employer is different than other employers. everyone here already knew that.
I’m guessing this was really just a way to brag about where they work. Seems oddly common among people that work at that company.
For what it’s worth, I’ve been at more than a few companies and this lines up. Managing a refactor or migration that’s long term and creates significant impact across an org chart or product area is definitely in the purview of promotion to senior or staff, whatever it’s called.
For what it’s worth, I’ve been at more than a few companies and this lines up. Managing a refactor or migration that’s long term and creates significant impact across an org chart or product area is definitely in the purview of promotion to senior or staff, whatever it’s called.
Given that the tool mentioned in this thread, rf, is written by Russ Cox, who works at Google, to help refactor go, a language developed and maintained by Google, how Google addresses these things feels not totally irrelevant.
I also think that "migration" and refactor are completely different concepts. And only the first is going to get you promoted (in the context of Google), because there's so much tooling for making even large refactoring relatively trivial. Pretty much every language has something like this, the build system also supports it too, so the process of, for example, taking a file, renaming it, querying and updating all dependencies (both source files and the build system), testing and sending out pull requests is essentially entirely automated. Unless there's something unique about a refactoring, it's not challenging enough to be promo-worthy.
I also think that "migration" and refactor are completely different concepts. And only the first is going to get you promoted (in the context of Google), because there's so much tooling for making even large refactoring relatively trivial. Pretty much every language has something like this, the build system also supports it too, so the process of, for example, taking a file, renaming it, querying and updating all dependencies (both source files and the build system), testing and sending out pull requests is essentially entirely automated. Unless there's something unique about a refactoring, it's not challenging enough to be promo-worthy.
Refactors are rarely as simple as moving a few file names or symbols around, at least where I work. While we do have some automation to help (and not break existing consumers), it’s significantly more involved in an SOA architecture for tier1 services. These sorts of big refactors also generally come with data model changes, performance profiling etc. This tool seems fantastic for parts of that, but not all of it.
Just as a friendly aside, your tone seems strangely defensive (though this may just be my fault reading this tired at 3AM) - Levels tells me you make on the order of half a million a year, while I’ll be lucky to make half of that in the next decade at my employer, so you really can’t feel persecuted by this when you most likely work half as much as me while being rewarded for it while folks like me struggle to tread water and are considered mentally deficient.
Just as a friendly aside, your tone seems strangely defensive (though this may just be my fault reading this tired at 3AM) - Levels tells me you make on the order of half a million a year, while I’ll be lucky to make half of that in the next decade at my employer, so you really can’t feel persecuted by this when you most likely work half as much as me while being rewarded for it while folks like me struggle to tread water and are considered mentally deficient.
> Refactors are rarely as simple as moving a few file names or symbols around, at least where I work. While we do have some automation to help (and not break existing consumers), it’s significantly more involved in an SOA architecture for tier1 services. These sorts of big refactors also generally come with data model changes, performance profiling etc. This tool seems fantastic for parts of that, but not all of it.
Yes, I'd refer to that as a migration, which absolutely is a lot of effort, but a tool like this isn't going to be particularly helpful with a migration precisely because the majority of the effort with such a migration will be like...doing all the data stuff and traffic stuff. Changing the code is a minimal part. A refactor is often not a thing that even registers at a smaller company, because its on the scale of a few files. But "Large Scale Changes" are a thing when you end up having millions of lines of code (https://www.youtube.com/watch?v=TrC6ROeV4GI), and tools like this one are useful for those kinds of changes. But a single LSC won't get you promoted.
And rf is neat, but in a very go way, it's not particularly exciting if you're familiar with https://github.com/google/pasta or https://github.com/bazelbuild/buildtools/blob/master/buildoz... or clangmr or https://github.com/google/Refaster, or outside of google Semgrep and Bowler and others, since its essentially the same, but built in go for go.
> Just as a friendly aside, your tone seems strangely defensive (though this may just be my fault reading this tired at 3AM)
Not at all. I was however, pointing out that "this was really just a way to brag about where they work. Seems oddly common among people that work at that company." was certainly unnecessary, and perhaps unkind.
Yes, I'd refer to that as a migration, which absolutely is a lot of effort, but a tool like this isn't going to be particularly helpful with a migration precisely because the majority of the effort with such a migration will be like...doing all the data stuff and traffic stuff. Changing the code is a minimal part. A refactor is often not a thing that even registers at a smaller company, because its on the scale of a few files. But "Large Scale Changes" are a thing when you end up having millions of lines of code (https://www.youtube.com/watch?v=TrC6ROeV4GI), and tools like this one are useful for those kinds of changes. But a single LSC won't get you promoted.
And rf is neat, but in a very go way, it's not particularly exciting if you're familiar with https://github.com/google/pasta or https://github.com/bazelbuild/buildtools/blob/master/buildoz... or clangmr or https://github.com/google/Refaster, or outside of google Semgrep and Bowler and others, since its essentially the same, but built in go for go.
> Just as a friendly aside, your tone seems strangely defensive (though this may just be my fault reading this tired at 3AM)
Not at all. I was however, pointing out that "this was really just a way to brag about where they work. Seems oddly common among people that work at that company." was certainly unnecessary, and perhaps unkind.
Those are all good tools.
> perhaps unkind
Unnecessary? I sure don’t see people from my company point it out constantly. The superiority complex seems to be a Google only phenomenon in my experience. I guess that’s caused by constant external validation - perhaps it’s people like you that are “unkind” by flaunting your wealth and status while the rest of us have literally nothing.
The market has determined people like me are worth half of you or less. Please be cognizant of this.
> perhaps unkind
Unnecessary? I sure don’t see people from my company point it out constantly. The superiority complex seems to be a Google only phenomenon in my experience. I guess that’s caused by constant external validation - perhaps it’s people like you that are “unkind” by flaunting your wealth and status while the rest of us have literally nothing.
The market has determined people like me are worth half of you or less. Please be cognizant of this.
Food for thought: there are people on this board that make an order of magnitude less, simply due to location lottery. Half of FAANG level is still far from nothing.
To be clear, I also work at a FAANG, doing very similar work but am consistently condescended to by these folks, literally everywhere from real life to online. The fact that I'm basically only worth 50% of them just adds insult to injury.
[deleted]
Sorry I should have clarified - by "refactoring APIs" I meant the sort of work that would e.g. be unifying all photo abstractions at Facebook behind a new service, not just refactoring package-level abstractions. That sort of work does involve a lot of symbol-level renaming and dependency hell management, but that's just a part of the work.
yeah you gotta replace something in prod that already works perfectly fine - or release a greenfield feature and sunset it in 14mo lmaoo
I feel like I've been involved in quite a few projects like that; Some Guy sold technology X to a company, so now it's all hands on deck.
Over the years, one company I've worked for off and on (as a consultant/contractor) went from a jQuery UI library, to a CSS component library (which was fine), to an AngularJS component library, then some ex-googler (younger than me) swept in and became CTO or similar and Decided that the whole company should use, drumroll please, Polymer. Then Polymer 2 came around and they had to scramble to make everything Polymer 2 compatible. They were just about done when Polymer 3 came around, which did things Differently once again.
But I'm sure the guy made decent scratch and left the company right before things came crashing down to an even more cushy job. That's usually what happens.
Over the years, one company I've worked for off and on (as a consultant/contractor) went from a jQuery UI library, to a CSS component library (which was fine), to an AngularJS component library, then some ex-googler (younger than me) swept in and became CTO or similar and Decided that the whole company should use, drumroll please, Polymer. Then Polymer 2 came around and they had to scramble to make everything Polymer 2 compatible. They were just about done when Polymer 3 came around, which did things Differently once again.
But I'm sure the guy made decent scratch and left the company right before things came crashing down to an even more cushy job. That's usually what happens.
Does Google actually use Polymer much?
It does: https://github.com/Polymer/polymer/wiki/Who's-using-Polymer%...
The whole idea of web components was somehow sold inside Google, hard. They even re-wrote Youtube with web components v0 which was almost immediately deprecated. And Chrome had to support it for years before Youtube rewrote stuff in v1.
The whole idea of web components was somehow sold inside Google, hard. They even re-wrote Youtube with web components v0 which was almost immediately deprecated. And Chrome had to support it for years before Youtube rewrote stuff in v1.
This is from 2018, though.
Since then Google has abandoned Polymer and now has a new toy called lit-html. And is now busy writing stuff in lit-html.
Last year they shipped Constructible Stylesheets by default in Chrome despite the fact that both Safari and Mozilla were against the state of the spec and the proposed implementation.
The only reason? lit-html was going to use it. And the justification for not hiding it back behind the flag? "Oh, our internal metrics show 0.8% of sites using them", which, at the time, was exclusively Google-developed lit-html exclusively on Google-developed properties.
Last year they shipped Constructible Stylesheets by default in Chrome despite the fact that both Safari and Mozilla were against the state of the spec and the proposed implementation.
The only reason? lit-html was going to use it. And the justification for not hiding it back behind the flag? "Oh, our internal metrics show 0.8% of sites using them", which, at the time, was exclusively Google-developed lit-html exclusively on Google-developed properties.
Thanks for the insight.
What is `rdep`? Couldn't seem to find it with a cursory google search.
Reverse dependencies, meaning find all sources that depend on a given file. Likely inspired by the Bazel query function by the same name.
https://docs.bazel.build/versions/master/query.html#rdeps
https://docs.bazel.build/versions/master/query.html#rdeps
The sibling comment touches on this, but the class of tooling that lets you analyze (R)everse (dep)endencies. There's a bunch that do similar things, but examples:
Rdep: https://github.com/axw/rdep Gomod: https://github.com/Helcaraxan/gomod
If you look at what they're doing under the hood, its mostly gluing bunch of official Golang libs that let you inspect package-level information.
Rdep: https://github.com/axw/rdep Gomod: https://github.com/Helcaraxan/gomod
If you look at what they're doing under the hood, its mostly gluing bunch of official Golang libs that let you inspect package-level information.
> Forced error handling semantics so that unhappy paths are easily enumerated
They could have just added pattern matching and exhaustivity checking though, right?
They could have just added pattern matching and exhaustivity checking though, right?
A similar concept named "Tactics" was explored in Haskell.
https://reasonablypolymorphic.com/blog/towards-tactics/
HN Discussion: https://news.ycombinator.com/item?id=24759649
Relevant: https://haskellwingman.dev/ . it's actually possible to use it if you have the haskell language server installed, as a code action (at least i think it is, haven't personally tried it yet tho)
I hope more refactoring tools come into existence in the future. Right now, writing transforms similar to this complexity using Clang would require a fairly large amount of boilerplate code, implementing at least some AST matching code and wrapping it into a frontend action. While having this for Go is good, I hope it grows and inspires others to do similar things in other languages, the same way gofmt once did.
Since you're talking about Clang, I assume you're coding in C? In that case, Coccinelle [https://coccinelle.gitlabpages.inria.fr/website/sp.html] may serve your purposes, and has a long history of use in the Linux kernel to modify shared APIs.
In case of Clang (and more specifically libClang,) I am usually using it to do source-to-source translations on C++ code. In those cases, sometimes I need some fairly advanced knowledge, such as the type of an expression, to match on or perform logic on. However, this still looks quite interesting for some use cases (and I am interested in such tools.)
Ooooh ouch, yeah C++ is generally rough on static analysis tools.
> AST matching code and wrapping it into a frontend action.
I'm working on a an AST visualization tool and accidentally stumbled in to this rabbit hole. Send help!
I'm working on a an AST visualization tool and accidentally stumbled in to this rabbit hole. Send help!
I think this is cool but I wonder what the expected API surface for editor integrations (with mouse-click menu selection to apply a refactor) would work. Would editors code-generate this mini scripting language? Or will there be a more abstract way of providing the same intention to the tool? If editor integrations are expected to send the script-snippets, will there be facilities for programmatically generating valid script snippets?
I've always wanted a tool that finds duplicate code (with minor variable name differences) so I can factor it out. Often when prototyping I'll copy a function thinking it will change a lot or maybe just not knowing what will change. As the application matures I'll go back and clean these up but it shouldn't be too hard for a tool to identify.
https://github.com/mibk/dupl does a great job, and it's included in https://github.com/golangci/golangci-lint (which everyone should use, tbh)
IntelliJ does this.
Shouldn't this be part of gopls or something?
Maybe eventually, but not while it's experimental.
A similar, but general purpose, tool to this is https://comby.dev/.
I've used it before locally for some structural code changes. The author is super helpful/responsive too.
I've used it before locally for some structural code changes. The author is super helpful/responsive too.
I wonder if this is some sort of precursor work for the generics release? Feels like gofmt -r on steroids, and could be eventually used to do large scale transformations to the standard library?
- Replace an import with another import. This is used when forking a go package.
- Rename a package.
It's also pretty unclear from the documentation which operations are "UNIMPLEMENTED", since the label simply falls directly between two different operations. It would be nice if this were made more explicit.