This post describes the ongoing experiment of impact metric based voting and asks for input and feedback from the community!
Metrics based voting
Currently, voting in Retroactive Public Goods Funding (Retro Funding) is made up of reviewing and comparing individual applications to come up with an allocation of tokens among applicants. This process is intensive in manual labour and highly dependent on the individual experience and expertise of citizens.
In recent Retro Funding rounds, Open Source Observer has outlined an alternative approach to evaluating impact, by leveraging data to allow citizens express what types of impact matter most and how they should be rewarded instead of revieweing individual projects.
Badgeholders are badgeholders because they care deeply about the health and growth of the ecosystem as a whole, not because they know the intricacies of projects. For most badgeholders, evaluating the quality of a portfolio of projects is a much better way of leveraging their time and expertise than evaluating each individual project. This will become even more apparent as the mechanism scales to more projects.
See @ccerv1âs blog posts here 4 and OS Observers work on Impact vectors 6.
Different types of impact require different measurements
While some contributions to Optimism have rich data associated with them, which we can leverage to evaluate impact today, such as onchain contracts and open source libraries, others lack high quality and standardised data for impact evaluation, such as IRL events, education initiatives and offchain tooling.
A metrics based voting experience is only viable for a subset of impact today and canât be applied to evaluate all contributions to the Optimism Collective.
Impact Calculator: Experimenting with a metric based voting experience
To further explore how citizens could leverage data to evaluate the impact of a number of projects, we started by putting up a project idea for a prototype of a metric based voting interface called impact calculator 8. This prototype leverages OS Observer 3 data for impact metrics.
Buidl Guidlâs prototype 44
Screenshot 2024-02-27 at 15.27.21
Via Buidl Guidlâs Impact Calculator 44 a user is able to
- Select Impact Vectors: Allows users to choose from various impact vectors (e.g. metrics) with descriptions and creator names, search functionality, and options to view detailed pages or add vectors to a ballot.
- Ballot View: A ballot system where users can select or deselect impact vectors and edit their configurations. This includes a graphical representation of the allocation of OP among projects.
- Detailed View: Each impact vector has a detailed view, including a name, description, creator, and a link to GitHub. Users can visualize the impact vector and configure it.
- Configuration Option: Users can set a scale or weight to deterimine how flat or skewed the eventual distribution of tokens to projects should be.
TLDR; pick what types of impact you think are important and assign weight to them.
RetroPGFhub.com 9 is in the early stages of developing their own version of the impact calculator protoype and will share for input once ready.
Request for input and feedback
At this early stage input and feedback from the community are very valuable in shaping the iteration of this prototype. For the purpose of this prototype weâre focusing on evaluating the impact of onchain deployments and open source libraries.
- What are your thoughts on impact metric based voting? What excites you about it or what makes you skeptical?
- What do you think of the current selection of impact vectors? What impact vectors would you want to see to reward onchain deployments/builders?
- Does the weighting help you in expressing what impact you find valuable? Would you want to have additional configuration options?
- Does the graph help you visualize the impact of your impact vector weightings? Do you understand what the graph is showing you? Do you want to see how individual projects are impacted by your weightings?
Check out the impact calculator here 44
This thread is used for an open discussion on impact metric based voting and will be used for further updates on the impact calculator prototype.
Screenshot 2024-02-27 at 15.27.21
From a technical project perspective, I would like to learn whether we could count non evm related contributions, education resources reach and quality and supporting github repositories.
Avoid divided system
Iâm concerned about having a divided system, onchain contracts & open source libraries compared with other contributions to OP Stack. For example Gitcoin grants publics good funding is now focusing on open source.
To avoid this we could look to add categories (e.g. events, education, offchain tooling) with metrics that can apply to those categories. e.g. attendees/users, followers etc
We could also encourage projects to have onchain & open source components e.g. NFTs for attendees/participants, open source repositories for project information.
Gameable metrics
Unfortunately many of the metrics are gameable.
Even transaction volume and fees could be gamed with farming for token rewards or future token airdrops from projects.
Given that metrics can be gamed, we should focus on metrics that deliver real impact to the OP stack. RetroPGF is a strong incentive for projects to focus on those impactful metrics.
Great to see this coming into the open. Impact data experiments take us closer to a sustainable balance between objective and subjective decision making.
Over time I expect that the variance between human decisions and impact metric ârankingâ via objective data sources like this to shrink. So rather than answer the questions above, I just want to share one tendency I have seen which is that data (especially in early experiments) can over-correct the way in which people rely on their own expertise and contextual knowledge. Iâd love to see the impact metrics complement human decision making by providing prompts that help create the right voting habits as well as make better decisions. One very simple example would just be âhey, you didnât look at the impact metrics⌠would it help you to review some data?â and more examples of this type would probably create a nice loop for early observation of behaviour
abcoathup:
This is something Iâve been thinking about quite a lot. It would be very easy for incentives to go very wrong if we focus on metrics like this.
As an example, lets imagine we give ExampleSwap a grant of xM OP for some incentive scheme to attract users (via grant process not retroPGF).
Then that incentive scheme is set up so that users can earn more OP rewards for making transactions than they spend in ETH for gas.
Users(/bots) are then effectively paid to use the dApp, and so will pump up transaction numbers even if they arenât getting any benefit from the swaps themselves.
If we then weight transaction numbers / gas spent as an important metric for assigning RetroPGF we reward ExampleSwap for setting this system up. So they are incentivized to use their grants this wayâŚ
But is that actually valuable for Optimism?
Quantifiable metrics can obviously be useful, but if we are not very careful they will be given too much weight, and as you say, gamed to the point of being harmful.
Absolutely, I do think we should provide more clarity on what metrics should be targeted by applicants in order to secure more votes and potentially a higher funding
Jonas: Different types of impact require different measurements While some contributions to Optimism have rich data associated with them, which we can leverage to evaluate impact today, such as onchain contracts and open source libraries, others lack high quality and standardised data for impact evaluation, such as IRL events, education initiatives and offchain tooling. A metrics based voting experience is only viable for a subset of impact today and canât be applied to evaluate all contributions to the Optimism Collective.
To me, this may be the most important point of them all, and I really appreciate you spelling it out here.
While for example ânumber of usersâ may be an appropriate metric for comparing otherwise similar analytics services, all of which target highly skilled investors, it could be a very bad idea to use that same metric indiscriminately for general educational or news services - as that might mean rewarding the most attention grabbing or addictive approaches over those that create high quality content for specific target groups.
If we want Etherâs Phoenix to be in the service of humanity, we really need to think hard about the relationship between quantitative and qualitative metrics.
A good rule of thumb might be that the more we care about certain qualitative metrics (in the context of certain project categories), the more important it is to make sure that they are incorporated into the impact evaluation.
And the more a project category targets / affects âthe whole humanâ, the more we should probably require subjective human judgement to be a crucial evaluation criteria.
To the extend that human experience is the goal, human experience should also be the measuring stick.
Screenshot 2024-02-27 at 15.27.21
Jonas: Different types of impact require different measurements While some contributions to Optimism have rich data associated with them, which we can leverage to evaluate impact today, such as onchain contracts and open source libraries, others lack high quality and standardised data for impact evaluation, such as IRL events, education initiatives and offchain tooling. A metrics based voting experience is only viable for a subset of impact today and canât be applied to evaluate all contributions to the Optimism Collective.
To me, this may be the most important point of them all, and I really appreciate you spelling it out here.
While for example ânumber of usersâ may be an appropriate metric for comparing otherwise similar analytics services, all of which target highly skilled investors, it could be a very bad idea to use that same metric indiscriminately for general educational or news services - as that might mean rewarding the most attention grabbing or addictive approaches over those that create high quality content for specific target groups.
If we want Etherâs Phoenix to be in the service of humanity, we really need to think hard about the relationship between quantitative and qualitative metrics.
A good rule of thumb might be that the more we care about certain qualitative metrics (in the context of certain project categories), the more important it is to make sure that they are incorporated into the impact evaluation.
And the more a project category targets / affects âthe whole humanâ, the more we should probably require subjective human judgement to be a crucial evaluation criteria.
To the extend that human experience is the goal, human experience should also be the measuring stick.