Why SAMbot fell off the digital campaign trail

On May 16, 2022, the Samara Centre for Democracy and Areto Labs observed inconsistencies with how SAMbot was reporting data. To ensure that SAMbot’s findings are shared responsibly and accurately, we made the difficult decision to pause publishing weekly snapshots so we could look into this issue. Further investigation revealed discrepancies in toxicity measurements caused by a recent update to the machine learning model that SAMbot uses to assess a tweet’s toxicity.

While we have not released insights for the final two weeks of the election, SAMbot has continuously tracked tweet mentions throughout the entire election period.

SAMbot uses a natural language processing machine learning model. This means that SAMbot is trained and tested on millions of data points so it can understand whether tweets can be considered toxic, harmful, or insulting.

The model used by SAMbot, like all machine learning models, is updated over time as we have access to more data. These updates allow the model to report more accurate results but can produce slight changes to specific toxicity scores as SAMbot is using an expanded set of data points to analyze tweets. This can lead to slight changes to SAMbot’s findings, which is what SAMbot experienced on May 16, 2022.

SAMbot’s data from the Ontario election period will be re-analyzed using the current machine learning model. In lieu of weekly snapshots, insights will be shared in a broader Ontario Election Snapshot that covers May 4 to June 2, 2022.

Our aim with SAMbot is to explore and demonstrate how AI-driven tools can contribute to civic inquiry in an ethical and productive manner. We are keen to apply the lessons we learn from this experimental work.

We will include details about SAMbot’s version history on all snapshots to publicly track the usage and updates of related tools. The Samara Centre for Democracy and Areto Labs will also continue working together to better anticipate and manage disruptions to SAMbot’s tracking from third party software updates.

The information collected by SAMbot helps advance our understanding of the scale and impact of toxicity in online Canadian political conversations. We appreciate your patience as our work evolves.