Special thanks to Sov from Gitcoin and DisruptionJoe from Plurality Labs for review & input and @joanbp and @Griff for feedback as part of the Feedback Commission.
This post outlines a framework for the application review process of future Retroactive Public Goods Funding (Retro Funding) rounds and its application to Retro Funding 4. The application review process is applied to decide which applicants are eligible to participate in retro rounds and is used to enforce application rules and eligibility criteria.
Learnings from Retro Funding 3
You can find an overview of the Retro Funding 3 application review process here 1 and its learnings here 1. Below you find a summary of the main learnings from the application review process in Retro Funding 3:
- The optimistic approach (e.g. a project first needs to be reported) lead to an increase in operational effort. Due to: a. There being no incentive for community members to review all applications and report malicious ones b. There being no effective way of tracking which applications have already been reviewed or reported
- Reviewers sometimes applied criteria outside of application rules (e.g. subjective judgements on who should participate) or applied rules inconsistently across cases.
- A large amount of obvious spam submissions required review by multiple humans
Application Review process framework
Based on learnings from past Retro rounds you can find our goals for iterating on the application review process below:
- Minimize human review: We received tons of obvious spam in Retro Funding 3, which all needed to be reviewed by at least 5 humans. In this iteration, we aim to rely more on automated filtering of low quality submissions…
- Minimize duplicate work: In Retro Funding 3, each application was reviewed by multiple volunteers who could report the application for rule violations. This uncoordinated approach lead to duplicate work. We aim to minimize duplicate work by relying on clear roles and responsibilities.
- Maximize consistency of rule enforcement: In Retro Funding 3, rules were subjectively applied by randomly selected badgeholders, which lead to a difference in rule application across cases. We aim to provide more consistency in how rules are applied.
Proposed Process overview
- Retro Funding eligibility criteria and application rules are established 1.1 Eligibility criteria: For each Retro Funding round, there are pre-defined eligibility criteria. This sets expectations to builders and minimizes the amount of non-qualifying applications. Ideally, eligibility criteria are enforced in the sign up process. 1.2 Application rules: For each Retro Funding round, there is a set of application rules (ideally standardised across rounds), which applications must adhere to. Violating these rules will result in exclusion from the Retro Funding round.
- Application submission: Retro Funding applicants submit their applications by a certain deadline. After this deadline, applications are final and can no longer be edited.
- Application review Phase 1: Applications are excluded via automated filtering and badgeholder review 3.1. Automated filtering: To reduce manual human review of applications, automation must be used to identify submissions which violate application rules or eligibility criteria. Identified applications might be automatically excluded, or flagged to badgeholders for review. 3.2 Badgeholder review: a set of badgeholders review applications for eligibility criteria and application rule violations.
- Appeals: All applications which are excluded in review phase 1 get the chance to hand in an appeal. An appeal can refute the reasons for excluding the application, while application content can not be edited/changed.
- Application review phase 2 (appeal review): Appeals are reviewed by a number of badgeholders.
- Application approval/rejection attestation: Following application review, attestations which indicate approval or rejection are issued for each Retro Funding application.
Note that application content is final following the application deadline. Allowing for edits following the deadline within the application review process could lead to a significant increase in operational effort and complexity.
Badgeholder Review
- Badgeholders are asked to opt-in to participate in the application review process
- From the opt-in list, a number of badgeholders are randomly selected to participate, based on the expected number of applications to be reviewed
- Application review Phase 1: All applications are reviewed by badgeholders to enforce eligibility criteria and application rules 3.1. Rule violations are flagged: Each badgeholder is presented with subset of applications and is tasked with reviewing them for meeting specific eligibility criteria and/or application rules. Each application is reviewed by at least two badgeholders. 3.2. Flagged rule violations are reviewed: If a badgeholder flags an application rule violation or failure to meet eligibility criteria, 4 other badgeholders will be asked to review the case and confirm. 3/5 badgeholders need to agree for an application to be excluded
- Application Review Phase 2 (appeals) 4.1. Each rejected application is notified of their opportunity to hand in an appeal 4.2. Each badgeholder is presented with a subset of the appeals cases and is tasked with reviewing each. 4.3. Each appeal is reviewed by 5 badgeholders, with 3/5 badgeholders needing to signal approval of the appeal to pass. The reviewing badgeholders should be different to the reviewing badgeholders in phase 1. 4.4 Appeals can only dispute reasons for exclusion, application content can not be changed. (e.g. “Oh sorry I was meant to report my grant but forgot to do so” is an invalid appeal).
Retro Funding 4: Application review process
Eligibility criteria and application rules are established
- Retro Funding application submissions: the sign up flow is enforcing some of the eligibility criteria (e.g. applicant needs to provide Github, onchain contracts etc.)
- Application review Phase 1
- Automated filtering: A number of applications should automatically be flagged and excluded, including:
- Applications not meeting eligibility criteria
- Applications containing obv spam: In Retro Funding 4 this concerns project profile metadata (e.g. project name, picture, etc), project github repos (e.g. repo is empty, has been created shortly before applying, etc), project onchain contracts (contract have been newly deployed, contracts only have interactions from low reputation addresses, etc)
- Duplicate Applications, which include the same Github Repo or onchain contracts
- Manual Human review: In Retro Funding 4, all applications (which haven’t been automatically excluded) need to be checked for correct reporting of grants and funding
- Appeals: All applicants excluded in review phase 1 can hand in an appeal
- Application review phase 2 (appeals review): Appeals are reviewed by a number of badgeholders.
- Application approval/rejection attestation: Following application review, attestations which indicate approval or rejection are issued for each Retro Funding application.
Eligibility Criteria (Retro Funding 4)
(See Retro Funding 4: Onchain Builders - round details 2)
In Retro Funding 3, the Retro Funding sign up process saw a large influx of spam and low-quality applications. By setting stronger eligibility criteria in Retro Funding 4, the Foundation aims to provide clarity to builders and reduce the amount of human review required for spam and low-quality application. In addition, qualifying criteria reduce the number of applicants who have not generated sufficient impact to receive rewards (such as the below requirement of a minimum number of unique address interactions). The below eligibility criteria aim to strike a balance between broad inclusivity and sufficient requirements to allow for an operationally viable Retro Funding round.
Builders are eligible who have:
- Deployed their onchain contracts on one or multiple of the following OP chains: OP Mainnet, Base, Zora, Mode, Frax and Metal, and meet the following criteria:
- Onchain contracts have interactions from 420 unique addresses during Jan 1st - May 1st 2024
- Onchain contracts had their first transaction before April 1st 2024
- Onchain contract had more than 10 days of activity during Jan 1st - May 1st 2024
- Verified their onchain contracts in the Retro Funding sign up process
- Made their contract code available in a public Github repo, for which ownership has been verified in the Retro Funding sign up process
- Confirmed that they will comply with Optimism Foundation KYC requirements and are not residing in a sanctioned country
- Submitted a Retro Funding application before June 6th, 2024 and comply with application rules
Application Rules (Retro Funding 4) DRAFT
Please note that the Application Rules are not final and might change. Violating one or multiple rules will lead to the project becoming ineligible for Retro Funding 4:
- Promises of future impact - promises of future deliverables or impact are not allowed.
- False statements & deception - false claims about your contributions, past impact or funding & grants are not allowed.
- Violations of the Rules of Engagement - no discrimination against any person based on identifying features, such as religion, sexuality, ethnicity or geographical location, public or private harassment (as defined here), the use of sexualized language or imagery and unwelcome sexual attention or advances or Intentionally doxxing (as defined here)
- Deceiving badgeholders - Malicious content that could cause harm or unintended consequences to users are not allowed.
- Fraud & Impersonation - Claiming to be a brand or person you are not. The Grant owner must be directly affiliated with the project, and the funds must go to the project.
- Advertising - Using Retro Funding application to showcase something you are selling like a token sale or NFT drop.
- Bribery - Bribing badgeholders or vote buying is strictly forbidden.
- All recipients are subject to KYC - If you do not pass KYC, your grant will be returned to the Retro Funding treasury for future rounds. You can find out more here
- Not meeting Eligibility criteria - contributions that do not meet the eligibility criteria of the relevant Retro Funding round.
- Spam - Applications containing spam, such as irrelevant answers, plagiarized content, broken or unrelated impact metrics and contribution links. Applications in languages other than English*.
- This will help simplify the process as English is the working language of the majority of Badgeholders. Please ensure you translate any content that’s part of the application.
- Duplicate applications - Multiple applications from the same individual, project or group which apply for the same impact.
- Members of contribution paths (Council Members, Ambassadors, NumbaNERDs, SupNERDs, TechNERDs, Translators) or Councils can’t submit individual applications for their work within the relevant workstream, as each workstream will apply as a project.
Next Steps
The Foundation will explore the implementation of the application review process for Retro Funding 4. This thread will be used to update on implementation progress, flag possible changes to the process design and collect feedback from badgeholders and the community.
Apr 24
4 / 4
Apr 26
6d ago
Special thanks to Sov from Gitcoin and DisruptionJoe from Plurality Labs for review & input and @joanbp and @Griff for feedback as part of the Feedback Commission.
This post outlines a framework for the application review process of future Retroactive Public Goods Funding (Retro Funding) rounds and its application to Retro Funding 4. The application review process is applied to decide which applicants are eligible to participate in retro rounds and is used to enforce application rules and eligibility criteria.
Learnings from Retro Funding 3
You can find an overview of the Retro Funding 3 application review process here 1 and its learnings here 1. Below you find a summary of the main learnings from the application review process in Retro Funding 3:
- The optimistic approach (e.g. a project first needs to be reported) lead to an increase in operational effort. Due to: a. There being no incentive for community members to review all applications and report malicious ones b. There being no effective way of tracking which applications have already been reviewed or reported
- Reviewers sometimes applied criteria outside of application rules (e.g. subjective judgements on who should participate) or applied rules inconsistently across cases.
- A large amount of obvious spam submissions required review by multiple humans
Application Review process framework
Based on learnings from past Retro rounds you can find our goals for iterating on the application review process below:
- Minimize human review: We received tons of obvious spam in Retro Funding 3, which all needed to be reviewed by at least 5 humans. In this iteration, we aim to rely more on automated filtering of low quality submissions…
- Minimize duplicate work: In Retro Funding 3, each application was reviewed by multiple volunteers who could report the application for rule violations. This uncoordinated approach lead to duplicate work. We aim to minimize duplicate work by relying on clear roles and responsibilities.
- Maximize consistency of rule enforcement: In Retro Funding 3, rules were subjectively applied by randomly selected badgeholders, which lead to a difference in rule application across cases. We aim to provide more consistency in how rules are applied.
Proposed Process overview
- Retro Funding eligibility criteria and application rules are established 1.1 Eligibility criteria: For each Retro Funding round, there are pre-defined eligibility criteria. This sets expectations to builders and minimizes the amount of non-qualifying applications. Ideally, eligibility criteria are enforced in the sign up process. 1.2 Application rules: For each Retro Funding round, there is a set of application rules (ideally standardised across rounds), which applications must adhere to. Violating these rules will result in exclusion from the Retro Funding round.
- Application submission: Retro Funding applicants submit their applications by a certain deadline. After this deadline, applications are final and can no longer be edited.
- Application review Phase 1: Applications are excluded via automated filtering and badgeholder review 3.1. Automated filtering: To reduce manual human review of applications, automation must be used to identify submissions which violate application rules or eligibility criteria. Identified applications might be automatically excluded, or flagged to badgeholders for review. 3.2 Badgeholder review: a set of badgeholders review applications for eligibility criteria and application rule violations.
- Appeals: All applications which are excluded in review phase 1 get the chance to hand in an appeal. An appeal can refute the reasons for excluding the application, while application content can not be edited/changed.
- Application review phase 2 (appeal review): Appeals are reviewed by a number of badgeholders.
- Application approval/rejection attestation: Following application review, attestations which indicate approval or rejection are issued for each Retro Funding application.
Note that application content is final following the application deadline. Allowing for edits following the deadline within the application review process could lead to a significant increase in operational effort and complexity.
Badgeholder Review
- Badgeholders are asked to opt-in to participate in the application review process
- From the opt-in list, a number of badgeholders are randomly selected to participate, based on the expected number of applications to be reviewed
- Application review Phase 1: All applications are reviewed by badgeholders to enforce eligibility criteria and application rules 3.1. Rule violations are flagged: Each badgeholder is presented with subset of applications and is tasked with reviewing them for meeting specific eligibility criteria and/or application rules. Each application is reviewed by at least two badgeholders. 3.2. Flagged rule violations are reviewed: If a badgeholder flags an application rule violation or failure to meet eligibility criteria, 4 other badgeholders will be asked to review the case and confirm. 3/5 badgeholders need to agree for an application to be excluded
- Application Review Phase 2 (appeals) 4.1. Each rejected application is notified of their opportunity to hand in an appeal 4.2. Each badgeholder is presented with a subset of the appeals cases and is tasked with reviewing each. 4.3. Each appeal is reviewed by 5 badgeholders, with 3/5 badgeholders needing to signal approval of the appeal to pass. The reviewing badgeholders should be different to the reviewing badgeholders in phase 1. 4.4 Appeals can only dispute reasons for exclusion, application content can not be changed. (e.g. “Oh sorry I was meant to report my grant but forgot to do so” is an invalid appeal).
Retro Funding 4: Application review process
Eligibility criteria and application rules are established
- Retro Funding application submissions: the sign up flow is enforcing some of the eligibility criteria (e.g. applicant needs to provide Github, onchain contracts etc.)
- Application review Phase 1
- Automated filtering: A number of applications should automatically be flagged and excluded, including:
- Applications not meeting eligibility criteria
- Applications containing obv spam: In Retro Funding 4 this concerns project profile metadata (e.g. project name, picture, etc), project github repos (e.g. repo is empty, has been created shortly before applying, etc), project onchain contracts (contract have been newly deployed, contracts only have interactions from low reputation addresses, etc)
- Duplicate Applications, which include the same Github Repo or onchain contracts
- Manual Human review: In Retro Funding 4, all applications (which haven’t been automatically excluded) need to be checked for correct reporting of grants and funding
- Appeals: All applicants excluded in review phase 1 can hand in an appeal
- Application review phase 2 (appeals review): Appeals are reviewed by a number of badgeholders.
- Application approval/rejection attestation: Following application review, attestations which indicate approval or rejection are issued for each Retro Funding application.
Eligibility Criteria (Retro Funding 4)
(See Retro Funding 4: Onchain Builders - round details 2)
In Retro Funding 3, the Retro Funding sign up process saw a large influx of spam and low-quality applications. By setting stronger eligibility criteria in Retro Funding 4, the Foundation aims to provide clarity to builders and reduce the amount of human review required for spam and low-quality application. In addition, qualifying criteria reduce the number of applicants who have not generated sufficient impact to receive rewards (such as the below requirement of a minimum number of unique address interactions). The below eligibility criteria aim to strike a balance between broad inclusivity and sufficient requirements to allow for an operationally viable Retro Funding round.
Builders are eligible who have:
- Deployed their onchain contracts on one or multiple of the following OP chains: OP Mainnet, Base, Zora, Mode, Frax and Metal, and meet the following criteria:
- Onchain contracts have interactions from 420 unique addresses during Jan 1st - May 1st 2024
- Onchain contracts had their first transaction before April 1st 2024
- Onchain contract had more than 10 days of activity during Jan 1st - May 1st 2024
- Verified their onchain contracts in the Retro Funding sign up process
- Made their contract code available in a public Github repo, for which ownership has been verified in the Retro Funding sign up process
- Confirmed that they will comply with Optimism Foundation KYC requirements and are not residing in a sanctioned country
- Submitted a Retro Funding application before June 6th, 2024 and comply with application rules
Application Rules (Retro Funding 4) DRAFT
Please note that the Application Rules are not final and might change. Violating one or multiple rules will lead to the project becoming ineligible for Retro Funding 4:
- Promises of future impact - promises of future deliverables or impact are not allowed.
- False statements & deception - false claims about your contributions, past impact or funding & grants are not allowed.
- Violations of the Rules of Engagement - no discrimination against any person based on identifying features, such as religion, sexuality, ethnicity or geographical location, public or private harassment (as defined here), the use of sexualized language or imagery and unwelcome sexual attention or advances or Intentionally doxxing (as defined here)
- Deceiving badgeholders - Malicious content that could cause harm or unintended consequences to users are not allowed.
- Fraud & Impersonation - Claiming to be a brand or person you are not. The Grant owner must be directly affiliated with the project, and the funds must go to the project.
- Advertising - Using Retro Funding application to showcase something you are selling like a token sale or NFT drop.
- Bribery - Bribing badgeholders or vote buying is strictly forbidden.
- All recipients are subject to KYC - If you do not pass KYC, your grant will be returned to the Retro Funding treasury for future rounds. You can find out more here
- Not meeting Eligibility criteria - contributions that do not meet the eligibility criteria of the relevant Retro Funding round.
- Spam - Applications containing spam, such as irrelevant answers, plagiarized content, broken or unrelated impact metrics and contribution links. Applications in languages other than English*.
- This will help simplify the process as English is the working language of the majority of Badgeholders. Please ensure you translate any content that’s part of the application.
- Duplicate applications - Multiple applications from the same individual, project or group which apply for the same impact.
- Members of contribution paths (Council Members, Ambassadors, NumbaNERDs, SupNERDs, TechNERDs, Translators) or Councils can’t submit individual applications for their work within the relevant workstream, as each workstream will apply as a project.
Next Steps
The Foundation will explore the implementation of the application review process for Retro Funding 4. This thread will be used to update on implementation progress, flag possible changes to the process design and collect feedback from badgeholders and the community.
- OP Bulletin: Weekly news and insights on the Optimism Collective
- Optimism Forum Weekly Recap (April 22 - April 28, 2024)
- 8d
- last reply 6d
- 3
- 270
- 2
- 6
- 3
created
replies
views
users
likes
links
Unlisted on Apr 24
Listed on Apr 25
Is the review process still the same as before? And would like to add that if the project is rejected, we as badge holders should add comments on the decision as it will help evaluate the value of the applicant