Automated Coder Reviews for New Project Applications - Planning/Development

This wiki page has been set up to continue the 'Automated Coder' discussion initially started at http://groups.drupal.org/node/155084#comment-517764.

Questions

• Do we want automated reviews for every issue opened in the queue?
• Would we prefer a semi-automated approach which requires an applicant or reviewer to specifically trigger the automated code review process?
• How would we want to trigger re-testing of automated code review runs? Would we leverage Issue statuses, similar to the Automated testing process? How would this work into our existing workflows?
• We need to determine some workflow and governance around how and under what conditions we would trigger a Coder review request

Application Pre-Requisites for Leverage Existing qa.drupal.org Functionality

There are a couple of pre-requisites which would have to be satisfied in order to work with the current qa.drupal.org infrastructure:

• The application needs to have a -dev release created (TODO: Check pift_cron_get_release() code, to see if having a -dev branch is enough to satisfy PIFT)
• Contrib modules/applications with dependencies on other contrib modules can not be tested (Global limitation of the current infrastructure)
• Currently, the re-review would be triggered on patches posted to issues in the project issue queue, or on the next/subsequent git commits. We'd need to build our own custom function for the re-triggering of tests based on updates within the 'Project Application Queue' (which isn't actually associated with the project we want tested).

Proposed Approach

With the above pre-requisites in mind, I cut a first draft of a potential approach:

New projectapps Module

Creation of a dedicated drupalorg_projectapps module, to host code related specifically to the Project Application process.

1. Adds the 'Age' column to the '/project/issues/projectapplications' and 'project/issues/search/projectapplications' views.

2. Creates a new 'drupalorg_projectapplications' table to track project application status.

   • Fields: Application Nid, Project Nid, Project Title, Repo Id, Repo Link, Release Exists, CoderCheckPassed, LicenseCheckPassed, SecurityCheckPassed, Closed Date

3. Acts on newly created applications in the 'projectapplications' issue queue:

   • Create a new {drupalorg_projectapplications} entry to represent the application

   • Parse the application for a 'sandbox' link.

     • If a sandbox link can not be found:

       • Auto-posts a "Please provide your sandbox link" comment in the issue
       • Sets issue status to 'needs work'

     • If a sandbox link can be found:

       • Parse it to determine the NID for the sandbox project
       • Determine the project title name (node title from {node})
       • Determine the corresponding repo ID from {versioncontrol_project_projects} (20520)
       • Updates {drupalorg_projectapplications} with appropriate Nid/Title/Repo information

       • Check for a '-dev' release branch (or better) by querying {versioncontrol_labels} with the repo_id (and type = 2)

         • If no dev release found:

           • Adds a 'Please create a -dev release in your repository to facilitate automated testing' comment to the issue queue
           • Sets the issue status to 'needs work'

         • If dev release (or better) found:

           • Updates {drupalorg_projectapplications} 'Release Exists' value
           • Adds a 'Queuing for qa.drupal.org testing' comment to the issue queue
           • Adds tag "PAR:RequestQATestRun"

4. Acts on newly created comments in the 'projectapplications' issue queue

   • Checks for issue state changes

     • Issue change to 'Needs Review':

       • If {drupalorg_projectapplications}.ProjectNid not set:

         • Parse application sandbox comments (most recent comment first), looking for a sandbox link. If found, follow the steps outlined in #3 above

       • ElseIf {drupalorg_projectapplications}.ReleaseExists not set:

         • Check for a '-dev' release branch (or better) by querying {versioncontrol_labels} with the repo_id (and type = 2), and react as per the steps in #3 above.

   • Check for specific PAR (Project Application Review) tags:

     • "PAR:RequestQATestRun":

       • Ensure state is set to 'needs review' (i.e. don't queue test for a 'needs work' application)
       • Trigger one-time, on-demand test request to qa.drupal.org
       • Upon test return, clear flag

     • "PAR:CoderCheckPassed" (ie. Reviewer manually ran the coder check):

       • Ensure tag added by someone other than applicant, who shouldn't update their own status flags
       • Update {drupalorg_projectapplications}.CoderReviewPassed for the application
       • Remove the tag from the application (and add an 'Updated "Code Review" Status Flag' comment)

     • "PAR:CoderCheckFailed" (ie. Reviewer manually reversing Coder Pass Flag):

       • Clear {drupalorg_projectapplications}.CoderCheckPassed if TRUE.

     • "PAR:LicenseCheckPassed" (ie. No 3rd party licensing issues found/remaining)

       • Ensure tag added by someone other than applicant, who shouldn't update their own status flags
       • Update {drupalorg_projectapplications}.LicenceCheckPassed for the application
       • Remove the tag from the application (and add an 'Updated "Code Review" Status Flag' comment)

     • "PAR:LicenseIssueFound"

       • Clear {drupalorg_projectapplications}.LicenseCheckPassed if TRUE.

     • "PAR:SecurityCheckPassed" (ie. No Security Issues found)

       • Ensure tag added by someone other than applicant, who shouldn't update their own status flags
       • Update {drupalorg_projectapplications}.SecurityCheckPassed for the application
       • Remove the tag from the application (and add an 'Updated "Code Review" Status Flag' comment)

     • "PAR:SecurityIssueFound"

       • Clear {drupalorg_projectapplications}.SecurityCheckPassed if TRUE.

5. Adds a table with project application status (ie. based on {drupalorg_projectapplications} columns) to the project application issue view, providing a single-table summary of Coder, Licence, and Security check status.

PIFT/PIFR Testing Enhancements/Integration approaches

Options:

• Add capability for PIFT to trigger on-demand tests for sandbox projects which aren't enabled via the pift_projects table. (How? Button on sandbox page? Specific Tag in Project Issue Queue? Menu callback w/ confirm form?)

  • Dependent on being able to pull a release from a sandbox page (ie. is having a 'branch' enough?)
  • Trigger would need to validate a branch is present, and that there are no contrib dependencies.

• Send test requests directly to PIFR, bypassing PIFT

• Build a manual interface directly on testbots (as per http://drupal.org/node/1176982), and send sandbox tests directly to a testbot (bypassing both PIFT/PIFR)

• Implement a specific testbot environment that only performs coder reviews, and a new PIFT_TEST_* type; as well as a mechanism to tell PIFT/PIFR to send this particular test type only to that testbot

  • Benefit: Coder can run on un-enabled modules, so this gets around any issues with calculating and downloading dependencies.