Introduction

Every social product under the sun has issues with equity, trust, and safety. It’s partially in the nature of humans, and partially in the nature of software—and as a result, it’s imperative that we consider our normative assumptions about what we build, and guard against the possibility of bad actors with every new feature that we make.

I’d been itching to work on issues of trust and safety since I joined Studio. This year, I had the opportunity to research and pitch a trust and safety plan—one that would hopefully allow us to build capacity for future social features and content moderation, and which would be feasible with a small team.

Baselines: Where we began

We already had a few social features in existence before the writeup of the plan. One such feature was the Peer Group, where students in classes can interact with each other and give each other feedback. Another was Chat, where students who are in classes together can interact directly, in groups or 1:1. There was reporting functionality on all messages, but no mechanism to cut off contact with another user on the platform. Luckily, we’d so far had few reported incidents.

Research

I chose to focus my research in a few areas:

1. Existing case studies from our company: I analyzed known issues with our social infrastructure, and areas where we’d had content flagged. For instance, there is a rather banal issue in some of our classes, where younger teens are matched through a random group generation process in peer groups with adults. There’s never been an explicit issue with this, but it can feel odd and create the potential for harm.

2. Looking at other companies that have a Trust and Safety program that I know and trust, or that are analogous to our situation. I chose two: Twitter (serious issues of trust and safety, and until recently a great safety team) and Kickstarter (analogous because of their creativity focus, and a company I highly respect for many reasons).

3. Reading books about issues of trust and safety. I looked around a bit and landed on The Internet of Garbage by Sarah Jeong, which a fantastic analysis, legal and otherwise, of spam, harassment, and hate on the Internet.

I found The Internet of Garbage to be particularly effective in clarifying my thinking around these issues. I’d never put spam and harassment into the same category, but Jeong argues that both are bad content that make the Internet unusable. The difference? Spam prevention has extensive investment. Harassment prevention does not. Cynically, I think this is because it most affects the vulnerable—women, people of color, and queer people, who have historically been underrepresented in the C-suite of tech and much more pessimistic about the social Internet.

Elements of the final plan

After careful analysis, I put together a written plan that was over 2500 words long 😅. I can’t share the plan with you on my portfolio, but here are some key areas:

1. Shaping up key documents, like terms of service and community guidelines. This is a point that I gathered from Kickstarter’s excellent terms of service and community guidelines, which have versions in plain text. I’m a big proponent of having plain English user agreements and/or plain English versions of all agreements.

2. Create documentation for future content moderators—for example, write out instructions for accessing specific chat messages or peer group posts.

3. Consider making social features opt-in, rather than opt-out: This is a point I took directly from Sarah Jeong, who noted that this subtle change can significantly change harassment patterns.

4. Implement blocking, which allows users to prevent a harassing user from contacting them, before expanding our social feature infrastructure.

5. Make viewing permissions more robust, so that users can more carefully control who sees their content and future content moderators can see any flagged content.

6. Improve tooling around content hiding and removal, so that content moderators can actually take action.

7. Once ready, hire a content moderation team.

Actions taken

Within a week of the plan being accepted, I got to begin work on following and blocking work! 🥳 There was also a push to begin reevaluating our viewing permissions on most posts.

It’s really exciting to see an organization encourage this kind of work, especially that early into its existence. If you work at a startup and have interest in protecting users, I would really encourage you to try and pursue it, drawing on the comprehensive research that already exists. I was really lucky to have a good background in the social issues of the Internet already, but it’s never too late to start! With any safety initiative, we’re able to keep the horizons of the Internet open to more people, making it a better digital place for everyone.