🤖 Add pull request autofix automation - #2767
Draft
JFWooten4 wants to merge 1 commit into
Draft
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
AI has made a big difference in my PR submissions, and it seems the Foundation takes a similar respective view. Namely, Copilot's been added to a plethora of
stellarrepositories over the last year, sometimes even running after a PR's been merged. Those code reviews have done wonders that would otherwise require a substantial and experienced internal team.I appreciate this expanding use of agents because it allows for nonpersonal (and hence nonjudgmental) and fast code feedback. It has been particularly interesting watching it run through suggestions on rudimentary SEP drafting. But it requires specific configuration to properly attribute to, which this PR begins.
Individual author attribution means a lot downstream. In a parallel example, we helped the SEC formulate modernizations to their corporate reporting system, which replaced bulk omnibus issuer changes with individualized access control and more. Functionally, that meant their implementation of SSO with 2FA runs on every individual rather than a set of shared, overarching plaintext credentials.
In adopting these changes, the Commission parroted our remarks about individual accountability, which is a big problem in our industry since a breach of access means potentially millions in losses with spilled MNPI. I say all this because I'd like to start a relatively substantial refactor which appears to affect current operations presently centralized through account-level authorship... or nonexistent. I believe these changes will streamline document contributions, amplify code review quality, and clarify human intent.
To start, we can see an example of automated review in stellar/stellar-dev-skill#99 (comment). This comes just after a Codex review through the ChatGPT connector. I really appreciate the Codex and Copilot reviews because they provide substantive feedback in code comments.
Whether or not the comments are directly related to the lines isn't the biggest factor for me, though it does help. What I love is that threads can continue on specific topics in their own bounded environments without cluttering the repository-style comment chat. The latter option can simply muddy the scope and topic as actual intent gets lost in back and forth.
I've attempted to remedy this pain in dear-github/dear-github#169 (comment), but it doesn't seem this is something the platform is prioritizing per community/community#5633. It was particularly relevant when Jake proposed a substantial governance change in stellar/stellar-protocol#1740 (comment), and I really appreciated the separate space outside of the main PR changes to respond to that substance. I do wish that resolved chats were a little easier to search through or peruse without manually opening each one, but that's another discussion.
Firstly, it's simple enough to ask contributors to add the
co-authored-bytag when working with LLMs. However, this does not alone extend to the review automation Jan's cool Triage Bot employs. If we can start to get this right together in a public repo, anyone can contribute to improving the quality of our review process without pandering for permissions.The problem with posting from a user account is notification tagging, automated triage, and later references all construe messages in the name of a detached individual. This is simply not a problem1 with a separate bot account and hence bot handle. And it makes it much easier for real developers to separate the wheat from the chaff when contemplating consequential network design choices.
@aolieman later commented that 99's "PR discussion reads like a generative brainstorming session" as akin in facebook/docusaurus#12285 (comment). It's an annoyance when solving a little bug. It's a structural weakness when reviewing changes that could later influence protocol development and/or get referenced in a CAP Discussion.
Aside from the bot approach, I've tried to filter down remarks through the connector bot in #2450 (comment). When the tool is used intentionally, it can employ
performed_via_github_appto visually separate the automated comment. For some reason, this looks different than in an issue like #2731 (comment). The latter has a much clearer AI disclaimer, but that could also be attributable to ChatGPT using a transparent happy icon which renders poorly in dark mode.2This would be invaluable to help filter Tyler's bot findings from actual thoughts she has. Although this app appendix does not work for the initial comment in an issue or PR, hence it's perfect for review procedure, but we'd need the full bot suite for actual item creation or specific feedback.
Lastly, and the starting intent of this branch, the bot can commit basic fixes like running Prettier or fixing references. You can see this in 22c636f, and I cannot express how much better this looks to have a clean separation from the drafting author. And hence the insertion in #2584 (review), it removes the need for contributors to wait for local commit checks. Moreover, it can be expanded with smarter logic to apply fixes like #2766 or #2701 (comment) completely automatically.
Authorship matters a lot because it separates out intelligent choices from robotic consensus. If we are to seriously employ GitHub for public collaboration, then there should be a visible line from what respected contributors think and agent hallucinations. When relevant conversations emerge years down the line, clear written attribution is really the only way to tell what a certain PR intended.
Moreover, internal members working on the Community Fund have publicly hinted at funding allocation reputation weight based on GitHub activity. This intent has already been formalized into $800,000+ distributed to repo maintainers based on OSS. It is substantially easier to intelligently allocate monies and trust when the line between developer and agent stays clear.
This PR gets the ball rolling with GitHub Actions, which is on by default and should be free for public repos. Next, I'd like to piece together an official Stellar bot, or better yet something with Raven. That kind of app would give our repos an unprecedented advantage in expanding community collaboration outreach without drowning a select few friends.
Footnotes
45 messages back and forth between agentic bots can really overload the viewport when not condensed into suggestion threads. But it is the perfect opportunity to handle rote procedural code review issues. Copilot's initial finding fails at iterating since it doesn't double-check whether a stated fix actually fixes the root problem, especially when it spans multiple lines and can hence require iterative fixes. ↩
The disclaimer does not directly appear as attribution for inline reviews. ↩