2022-23
Systemizing the features Facebook uses to fight problematic content
Since 2016 Meta has employed a 3-pronged strategy to combat problematic and potentially misleading content — Remove, Reduce, Inform. The inform prong involves giving people visible, in-the-moment context to help them decide what to trust and interact with.
Over time the patterns used to deliver this context on Facebook had become fragmented across surfaces, problem types, and a number of other axis.
In 2022 I worked with the team responsible for Meta’s responses to developing real-world events (e.g. elections, wars, public health crises ect.) to consolidate, standardize and scale inform patterns across Facebook.
CONFIDENTIALITY NOTICE: My work at Meta focused on highly sensitive trust and safety initiatives which are subject to strict NDAs. Did I read them? No. Am I scared of them? Yes.
As such much of my work including, iterative design explorations, specific workflows and non-published metric impacts cannot be shared publicly.
The challenge
People want to feel protected and empowered online, especially when they may encounter content they find distressing, confusing or sensitive.
While Facebook had invested heavily for several years in shipping proactive experiences aimed at empowering people with context and helping them make informed decisions about what to trust and interact with, large swaths of the catalog had been shipped by siloed, problem-specific teams across Meta.
Additionally some features had gone from 0-1 in extremely tight timelines as they were being deployed to mitigate harm around real-time events.
These factors among others meant that much of the catalog was some combination of repetitive, bespoke, unsupported or non-compliant with modern design language. These factors in aggregation were increasingly inhibiting progress on meeting key user needs.
My Role
I joined an internal strike team as 1-of-2 founding product designers with the mandate to simplify, consolidate and scale this disconnected suite of warnings across Facebook, ensuring coverage across the highest-touch surfaces on the app (Feed, Reels, Search ect.) and adherence with Facebook’s existing design system, while maintaining or improving current baseline trust and safety thresholds.
Contributions
-
Audited over 100 live and proposed inform experiences across all of Facebook, and led multiple workshops to categorize, evaluate them.
-
Created a principled unification framework which was used to identify key use cases, current gaps and ultimately prioritize solutions to address them.
-
Utilized the framework to lead the design of 5 canonical patterns that scaled app-wide, and partnered with research and data science to stress test at scale and refine.
-
Drove a 6-month design effort to update dozens of existing inform features across Feed, Reels and Search to newly-ratified canonical patterns and evaluate them against current baselines.
What we Shipped
Inform Labels
Consolidated and unified our patterns for giving unintrusive additional context on posts which may contain fact-checked information, lack context or pertain to sensitive, developing events in the world such as elections, Covid-19 or climate change.
Sensitive Content Covers
Standardized the way Facebook gives people the ability to avoid seeing sensitive, graphic or potentially misleading content without first giving consent.
Low-friction warnings for sharing content
Helped people avoid sharing things they may regret later, by creating a single pattern for warning people that the content they’re sharing is something that’s been fact-checked, is out-dated or pertaining to sensitive topics.
High-friction warnings for seeking out or creating content
We helped people avoid breaking Facebook’s rules, and suffering unintentional consequences to their account by warning them if they post or seek out posts that might be violations.
In-Feed Profile and Page labels
We made it easier for people to understand important information about accounts whose posts they see in feed, such as if their posts were paid for, and by whom, where they were based in the world, whether they use AI or are a state-controlled media outlet among others.
Outcome
Our work resulted in the creation of Meta’s first app-wide trust and safety design system. We shipped a massive wave of updates bringing the bulk of live experiences into compliance and unblocking critical in-coming work over the course of 2023. Our work was so successful that it was eventually adopted across Instagram, Messenger and later Threads.
Fewer bespoke interventions
95%
Our effort consolidated the ecosystem of inform features taking over 100 bespoke implementations and redesigning them using 5 canonical design patterns.
Response time
We exponentially reduced the time and resource overhead associated with responding to rapidly-developing real-world events.
Safety outcomes
Usage of the patterns at scale resulted in no stat-sig regressions in key metrics, and over time saw a steady increase as optimizations could now be rolled out at a system level.
Utilizations per year
50+ avg
The team’s pattern documentation and governance mechanisms were utilized to evaluate an average of 50 proposals a year.
Read more
This work enabled reactive and proactive responses to emergent problems and events across Facebook for years after its initial roll out, and the system remains in use today. While not all usages are publicly sharable, the below links are a representative sample of the kind of work this system enabled.
