Michigan Law ReviewMichigan Law Review
Volume 118 Issue 6
2020
Cyber Mobs, Disinformation, and Death Videos: The Internet as ItCyber Mobs, Disinformation, and Death Videos: The Internet as It
Is (and as It Should Be)Is (and as It Should Be)
Danielle Keats Citron
Boston University School of Law
Follow this and additional works at: https://repository.law.umich.edu/mlr
Part of the Entertainment, Arts, and Sports Law Commons, Internet Law Commons, and the Law and
Society Commons
Recommended CitationRecommended Citation
Danielle Keats Citron, Cyber Mobs, Disinformation, and Death Videos: The Internet as It Is (and as It
Should Be), 118 MICH. L. REV. 1073 (2020).
Available at: https://repository.law.umich.edu/mlr/vol118/iss6/9
https://doi.org/10.36644/mlr.118.6.cyber
This Review is brought to you for free and open access by the Michigan Law Review at University of Michigan Law
School Scholarship Repository. It has been accepted for inclusion in Michigan Law Review by an authorized editor
of University of Michigan Law School Scholarship Repository. For more information, please contact
mlaw.repository@umich.edu.
1073
CYBER MOBS, DISINFORMATION, AND
DEATH VIDEOS: THE INTERNET AS IT IS
(AND AS IT SHOULD BE)
Danielle Keats Citron*
SABRINA. By Nick Drnaso. Montreal: Drawn & Quarterly. 2018.
Pp. 203. $27.95.
INTRODUCTION
Nick Drnaso’s1
graphic novel Sabrina provides a powerful snapshot of
online norms. The picture is not pretty: A young woman goes missing. Her
grief-stricken boyfriend cannot bear to stay in their home and escapes to a
friend’s house. Her sister struggles with the pain of her loss. We learn that
the woman’s neighbor, a misogynist loner, killed her and recorded the murder.
Online, people clamor for the video.
The execution video leaks and goes viral. The media hounds the woman’s
sister and her boyfriend. A conspiracy theorist with a popular radio
show argues that the murder is a deep-state hoax. He gins up a cyber mob to
“investigate” what is really going on.
A cyber mob descends. The woman’s family, her boyfriend, and her boyfriend’s
friend are smeared as crisis actors. They are barraged with death
threats, and their personal information is posted far and wide. The attacks
continue until a shooting massacre captures the conspiracy theorist’s attention.
The cyber mob redirects its wrath at other mourners.
The novel raises important questions about the interaction of human
behavior, culture, and law in the digital age. What compels people to like,
click, and share grotesque execution videos, conspiracy theories, and destructive
falsehoods? We have always been drawn to information that resonates
with us, but the online environment seems to supercharge human
* Professor of Law, Boston University School of Law; Vice President, Cyber Civil
Rights Initiative; Affiliate Fellow, Yale Information Society Project; Affiliate Scholar, Stanford
Center on Internet and Society. I am grateful to Mary Anne Franks and Sara Rimer for their
helpful insights, to Carrie Goldberg for talking to me about her important work, to Ellie Citron
and Brian Flaherty for excellent feedback and research, and to Tyler Gabrielski for his kind
assistance. This project benefited from the support of Boston University School of Law and my
wonderful deans—Dean Angela Onwuachi-Willig and Associate Dean Stacey Dogan. Deep
thanks to the Michigan Law Review, my editors Nina Cahill and Andrew Lanham for their inspired
suggestions, and to Cade Boland and Sarah McDonald for their helpful insights.
1. Author and graphic novelist. Drnaso was nominated for the Man Booker Prize in
2018.
1074 Michigan Law Review [Vol. 118:1073
biases. Why? Platforms structure and shape online activity, so what are they
doing about online abuse? Each and every one of us is ultimately responsible
for liking, clicking, and sharing the destruction. How can we work to change
our behavior?
Right now, it is cheap and easy to wreak havoc online and for that havoc
to go viral. Platforms act rationally—some might say responsibly to their
shareholders—when they tolerate abuse that earns them advertising revenue
and costs them nothing in legal liability.
Combatting cyber-mob attacks must be a priority. Law should raise the
cost of cyber-mob attacks. It is time for tech companies to tackle some of the
negative externalities of their business model. Platforms should not enjoy
immunity from liability for user-generated content unless they have earned
that immunity with reasonable content-moderation practices. Education
should play a role as well. As digital citizens, we need to do better.
I. EXPOSING THE DARK SIDE OF NETWORKED LIFE
Sabrina is an important read. Fiction and visual representations can alter
our understanding of human experiences and struggles.2
The recognition
of human rights owes much to novels, art, and photographs that changed social
attitudes by showing human pain and degradation in a visceral way.3
As
Hillary Chute has argued, hand-drawn pictures forge a personal connection
with readers.4
They help us bear witness to suffering.5
Sabrina does this in spades. Male aggression and fear pervade Drnaso’s
graphic novel. So do dark, dull hues and slow-moving action.6
The novel’s
inhabitants and environs appear muted, contemplative, and restrained except
when the action moves online. Then, texts, emails, blog posts, comments,
videos, Skype calls, and multiplayer games appear in stronger colors.7
Then, frenetic, impulsive, and disturbing developments come to the fore.
The novel’s contrast of quiet introspection offline to the loud negativity
online allows readers to feel how jarring and destabilizing a cyber-mob attack
can be. One minute, people are safely and anonymously proceeding
with the minutiae of daily life. The next, they are caught in the blinding glare
2. LYNN HUNT, INVENTING HUMAN RIGHTS 35–69, 82–92 (2007).
3. MARK PHILIP BRADLEY, THE WORLD REIMAGINED: AMERICANS AND HUMAN
RIGHTS IN THE TWENTIETH CENTURY 9, 26, 48 (2016).
4. HILLARY L. CHUTE, DISASTER DRAWN: VISUAL WITNESS, COMICS, AND
DOCUMENTARY FORM (2016).
5. Id.
6. Nick Drasno told the New Yorker that he “severely restricted his color palette” by
design. He embraced an “ethic of subtraction: ‘If there is a gas station in a comic, usually you
see four cars. In “Sabrina,” there are no cars.’ ” D.T. Max, The Bleak Brilliance of Nick Drnaso’s
Graphic Novels, NEW YORKER (Jan. 14, 2019), https://www.newyorker.com/magazine/2019
/01/21/the-bleak-brilliance-of-nick-drnasos-graphic-novels [https://perma.cc/F5WN-78J9].
7. Id.
April 2020] Cyber Mobs, Disinformation, and Death Videos 1075
of a cyber mob’s attention. They are exposed, maligned, and scared. Sabrina
helps us appreciate what it is like to be in the vortex of a cyber-mob attack.
The novel opens with Sabrina Gallo, whose later absence is a driving
force of the story. Twenty-seven-year-old Sabrina is seen talking to her sister,
Sandra, in their childhood home in Chicago (pp. 2–4, 81). After chatting
about their parents, Sandra asks Sabrina to join her for a bike trip. The idea
is to “[g]et out of the city. Get away from the internet” (p. 8).
Sabrina wonders about the safety of camping out alone (p. 8). After
thinking quietly about the question, Sandra recalls a trip she took by herself
at age nineteen (pp. 8–9). The experience was “a spring break nightmare. . . .
lousy with college date rapists . . .” (p. 9). One night, three boys confronted
her on the beach (p. 9). They were “out hunting,” they said, and asked her to
go to their room (p. 9). One of the boys grabbed Sandra’s arm to prevent her
from leaving (p. 9). Sandra managed to escape to a restaurant where she hid,
crying in the bathroom (p. 9). She reveals the incident to Sabrina for the first
time and says that she has not taken a vacation since (p. 9). Sandra assures
Sabrina that their bike trip would be safe because “[t]he fucking wild animals
stay in hotels” (p. 9).
Two days after the sisters’ chat, Sabrina disappears (p. 58). She is last
seen leaving work (p. 30). As time passes, her loved ones assume the worst
(p. 37). Grief incapacitates her boyfriend of two years, Teddy King, who has
been living with her (p. 16). Teddy flees to Colorado Springs to stay with a
high-school friend, Calvin Wrobel, a cybersecurity analyst in the Air Force
(pp. 12–31).
Teddy is seen in Calvin’s guest room, listless and worried, while Sandra
is seen cradled in a ball on the floor (pp. 54–59). With her hands over her
ears, Sandra repeatedly cries, “I want it to stop” (pp. 62–63). Every night,
Calvin returns home from work and tries to convince Teddy to eat something
(pp. 32, 55). Calvin retreats to his bedroom to play first-person shooter
video games with his Air Force buddies (p. 42). In online chats, someone
says, “Just woke up and I can’t fall back asleep. Killing people always puts me
to sleep . . .” (p. 42).
Soon, Sabrina’s terrible fate is revealed. A twenty-three-year-old neighbor,
Timmy Yancey, abducted and killed her (pp. 70, 81). He is a misogynist
loner, an avid player of video games, and a freeloader whose mother pays his
rent (p. 70). He is modeled after Elliot Rodger, a college dropout who shot
and killed six people, including two sorority women, at the University of
California, Santa Barbara, because, as he raged online, beautiful women refused
to sleep with him.8
Chicago police learn about the killing after a newspaper contacts them
about a VCR recording of the murder (pp. 67–76). The killer sent copies of
8. Id. (noting that Drnaso modeled Sabrina’s murderer after Elliot Rodger); see also
Isla Vista Massacre: What Do We Know About Elliot Rodger’s Rampage?, NBC NEWS (May 26,
2014, 12:02 AM), https://www.nbcnews.com/storyline/isla-vista-rampage/isla-vista-massacrewhat-do-we-know-about-elliot-rodgers-n114416
[https://perma.cc/CV3Q-Y48G].
1076 Michigan Law Review [Vol. 118:1073
the video to several news outlets, Chicago politicians, and a local sportscaster
(p. 76). The police discover that after murdering Sabrina, the killer committed
suicide (p. 72). The killer’s smiling face is seen jutting out of a bloody
bathtub in his apartment (p. 72). Although we never see the execution video,
we learn that the killer wore a black mask and said on the tape, “It has become
increasingly difficult for my voice to be heard above the din of chatter.
This is only a means to an end” (pp. 81, 114).
After a news outlet posts a screenshot of the killer taken from the video
(p. 76), online posters clamor to see the whole thing. Online commenters solicit
links (p. 81). The comment “I NEED to see this,” posted at 1:37 a.m.,
gets 101 likes (p. 81). Trending hashtags of the day include #TimmyYancey,
#SabrinaGallo, #TheAvengers, and #SalmonRecall (p. 81).
Details about the killer’s life appear in posts that garner hundreds of
clicks, comments, and shares. The post “What we know about Timmy Yancey”
reveals that he was “active” on men’s rights message boards and had been
banned from several online communities due to his “vitriolic rants” (p. 81).
The frenzy builds offline as well. Television crews surround Sandra’s
house (p. 99). The family issues a statement to stop people from contacting
them, but to no avail (p. 99). Reporters camp out at Calvin’s house to reach
Teddy (p. 99). Calvin tells a coworker that only after a “big shooting happened
in Buffalo” did the media retreat (p. 99). The shooting massacre diverts
the public’s attention, but just for a while.
After learning about Sabrina’s death, Teddy spends his days listening to
the radio (p. 88). He tunes into the show of conspiracy theorist Albert Douglas,
a takeoff on Alex Jones’s Infowars.9
Teddy is seen hugging a pillow as he
listens to Douglas talk about a “globalist” conspiracy to keep the public repressed
(p. 88). According to Douglas, the government has been staging
school shootings to keep people separated and scared (pp. 101, 118, 121).
Douglas urges listeners to harness their rage in the “right direction”
(p. 88). “[The government is] going to announce a state of emergency. . . .
Then they will shut down the power grid and disable the internet,” Douglas
insists (p. 138). He warns that at that point, fighting back will be useless: the
“moment to organize an armed rebellion [will have] passed” (p. 138).
A month later, the execution video leaks and goes viral (pp. 112–13).
According to Douglas, the video is being downloaded “five million times per
hour” (p. 108). Online advertisements flash next to links to the execution
video (p. 113). Searches for the video yield autocomplete suggestions that include
the killer’s name and various terms like “video leak,” “download,”
“video full,” and “video stream” (p. 113). Even Calvin succumbs to curiosity
and searches for the video. He watches it while Teddy sleeps in the room
next to him (p. 114). We see him struggling to make it to the bathroom to
vomit (p. 114).
9. P. 88. Drnaso explained that his research for the book included “listen[ing] to podcasts
of ‘Infowars,’ the extremist radio show hosted by Alex Jones.” Max, supra note 6.
April 2020] Cyber Mobs, Disinformation, and Death Videos 1077
The viral spread of the video generates intense media attention. Sandra
is seen in news footage posted online crying and yelling, “This is madness!
This has to stop! Get away from me!” (p. 113). Camera crews appear at Calvin’s
home (p. 109). He pleads for the crew to leave: “Please respect our privacy.
I don’t even know why you’re asking me about this. I didn’t even know
Sandra” (p. 110).
As the madness escalates outside his bedroom and online, Teddy hides
in Calvin’s house (p. 117). He stays in bed listening to Douglas’s show.
Douglas turns his attention to Sabrina’s murder (p. 117). According to
Douglas, Sabrina’s murder is a fiction: “I don’t believe something like that
could happen. . . . It’s possible she never even existed,” he says (p. 117).
At first, Teddy covers his ears when Douglas talks about Sabrina’s death
(p. 108). But then we see him listen, especially as Douglas makes the case that
Sabrina is alive (p. 117). Douglas claims that the execution video is a government-created
fake: “We know the military has access to advanced computer-generated
image technology that the public won’t be aware of for
another ten years.”10
Douglas pleads with his audience not to believe what
they are seeing. His entreaty? Do not be manipulated by “them” (pp. 117–
18).
Douglas calls upon his “amateur sleuths” to study the facts of Sabrina’s
case (p. 108). He tells them to find the “discrepancies, inaccuracies, distortions,
and outright lies” (p. 108). Douglas urges his listeners to recognize and
leverage their power: “Take one lonely person clacking away at a keyboard:
powerless. Put them all together, and you have a force to be reckoned with
that can move mountains” (p. 101).
Douglas’s followers spread the false claim that Sabrina’s death is a hoax.
His website’s discussion board has 6,790 posts devoted to Sabrina’s case
(p. 122). The posts include “Sabrina Gallo Alive!” and “Timmy Yancey - CIA
Mind Controlled Killer” (p. 122). Other conspiracy websites follow suit, accusing
Sabrina’s loved ones and Calvin of being crisis actors (p. 119). Proof
of the conspiracy? That Calvin referred to the victim by the wrong name
when the media hijacked him outside his home and that he works for the
Department of Defense (pp. 119, 121). Posts claim that Calvin and the killer
were best friends (p. 122). A search of Calvin’s name includes autocomplete
suggestions including his name next to terms like “fake,” “actor,” and “exposed”
(p. 119).
Sandra and Calvin receive hundreds of emails from strangers accusing
them of lying and threatening death (pp. 119, 132, 155). Their contact information
is published online.11
An emailer ominously tells Sandra: “Your
10. P. 117. Drnaso is invoking the specter of deep fakes, machine-learning technology
that manipulates or fabricates video or audio recordings to show people doing or saying things
that they never did or said. Bobby Chesney & Danielle Citron, Deep Fakes: A Looming Challenge
for Privacy, Democracy, and National Security, 107 CALIF. L. REV. 1753 (2019).
11. P. 122. See generally DANIELLE KEATS CITRON, HATE CRIMES IN CYBERSPACE 53
(2014) (explaining that cyber stalking often includes the publication of targeted individuals’
personal information, which is known as doxing).
1078 Michigan Law Review [Vol. 118:1073
address is online. People in our community are waking up to the truth. I’m
armed and protected. See what happens if they try and test me” (p. 155).
Online comments include, “Someone should kill you ‘Calvin Wrobel’”;
“Fraud! Your day is coming. We know your lies and will smoke you out
where you live” (p. 120). An email sent to Calvin features the Florida address
of his ex-wife and young daughter next to the missive, “It’s a dangerous
world out there. Please do the right thing” (p. 122).
Sabrina ably captures the breathtaking velocity of disinformation online
and the rapid escalation to terroristic threats. Alex Jones’s smears of the
Sandy Hook families took no time to spread to all corners of the internet.12
Yes, message boards like 4chan and 8chan were filled with damaging lies
about the parents and their personal information. But so were mainstream
sites like Facebook and Twitter.13
In short order, the families received death
threats and their personal information was exposed.14
Much like real life, gruesome shootings are followed by conspiracy theories
and cyber-mob assaults. In Sabrina, word spreads about a new mass
shooting, this time in Denver, Colorado (p. 143). We see the Denver killer’s
Facebook post declaring his desire to be remembered (p. 143). Right after
putting up the post, the young man killed everyone at a day care center and
turned the gun on himself (p. 143). Online commenters declare him the
“New Face of Evil” (p. 144). Stories about Sabrina’s case are replaced with
ones about the Denver killing (p. 144).
That day, “Denver Massacre” is the trending hashtag (p. 143). Douglas
shifts his focus to the “business” of Denver (p. 145). The last thing Teddy
hears Douglas say before turning off the radio: “The final death tally is at
thirty-one. More of the same. The globalists will stop at nothing to undo our
constitutional rights” (p. 145). The cyber mob has new mourners to target.
This cycle is depressingly familiar. After a teenage boy walked into the
Sandy Hook Elementary School and killed twenty children and six adults
and injured many more, the hashtags #SandyHook and #ProtectOurKids
took off.15
The hashtag #PrayForVegas trended after the Las Vegas shooter
killed more than fifty people and injured 400 others.16
After a student
12. Ed Pilkington, Trapped in a Hoax: Survivors of Conspiracy Theories Speak Out,
GUARDIAN (Jan. 24, 2019, 1:00 AM), https://www.theguardian.com/technology/2019/jan
/23/conspiracy-theories-internet-survivors-truth [https://perma.cc/9QU6-3BND].
13. Id.; see also Mike Wendling, Sandy Hook to Trump: ‘Help Us Stop Conspiracy Theorists,’
BBC (Apr. 2, 2017), https://www.bbc.com/news/blogs-trending-39194035 [https://
perma.cc/ZEK6-R7PD].
14. See Pilkington, supra note 12.
15. Melissa Fares, Three Years Later, Thousands Mark Sandy Hook Shooting on Social
Media, REUTERS (Dec. 14, 2015, 3:04 PM), https://www.reuters.com/article/us-connecticut-
shooting-socialmedia/three-years-later-thousands-mark-sandy-hook-shooting-on-socialmedia-idUSKBN0TX2BK20151214
[https://perma.cc/V7QN-KDW5].
16. Megan Armstrong, #PrayForVegas: Support Pours Out on Social Media for Route 91
Harvest Shooting Victims, BILLBOARD (Oct. 2, 2017),
April 2020] Cyber Mobs, Disinformation, and Death Videos 1079
brought weapons of war to Marjory Stoneman Douglas High School in Parkland,
Florida, and killed seventeen students and staff and injured seventeen
others, the hashtags #NeverAgain and #MarchForOurLives went viral.17
After
a man killed three people at the Gilroy Garlic Festival, hashtags #PrayForGilroy
and #GunControlNow filled news feeds,18
only to be eclipsed days
later by the hashtag #WhiteSupremacistTerrorism after the mass shootings
in El Paso, Texas, and Dayton, Ohio.19
As in Sabrina, cyber mobs descended upon the loved ones of the victims
of those mass shootings with frightening intensity. The Sandy Hook families
were stalked, terrorized, and defamed online as crisis actors.20
Alex Jones
spread lies about the Las Vegas shooting and inspired his followers to investigate
further.21
Grieving Parkland students were attacked online as well-paid
actors and hounded with lies and threats.22
And as in Sabrina, the press demanded
that the mourners respond to conspiracy theories and lies.23
Although the novel does crucial work in helping us appreciate the terror
of an unrelenting cyber-mob attack, its major flaw is its predominantly male
perspective. Calvin is the focus of the novel. We see Calvin’s inbox full of
threats and his Google searches filled with lies. After the Denver shooting,
we see his inbox empty for the first time in ages (p. 144). We see Calvin later
receiving an ominous email warning that he has not been forgotten (p. 175).
The novel ends with Calvin dreaming about being murdered by men in black
masks.24
https://www.billboard.com/articles/news/7981950/las-vegas-shooting-social-media-supportroute-91-pray-for-vegas-hashtag
[https://perma.cc/HM9U-7VVX].
17. John Bacon, ‘They Have Made Change’: 1 Year After Carnage in Parkland, Where
Key Figures Are Now, USA TODAY (Feb. 10, 2019, 2:48 PM), https://www.usatoday.com
/story/news/nation/2019/02/10/parkland-one-year-after-shooting-where-key-figures-are-now
/2721798002/ [https://perma.cc/U2C3-82SX]; Abby Ohlheiser & Kayla Epstein, ‘Just Try to
Keep Calm,’ WASH. POST (Mar. 3, 2018), https://www.washingtonpost.com/graphics/2018
/lifestyle/parkland-shooting-in-social-media/ [https://perma.cc/8EH7-9KVU].
18. Mary Papenfuss, Gilroy Garlic Festival Shooting Prompts Outcry, Mourning on Twitter,
HUFFPOST (July 29, 2019, 1:45 AM), https://www.huffpost.com/entry/gilroy-garlic-festivaltwitter-reaction_n_5d3e73e2e4b0ef792e0e8ee6
[https://perma.cc/2SXC-MYB8].
19. Jay Connor, #WhiteSupremacistTerrorism: Hashtag Takes Over Twitter After Latest
Pair of Mass Shootings, ROOT (Aug. 4, 2019, 11:12 AM), https://www.theroot.com
/whitesupremacistterrorism-hashtag-takes-over-twitter-1836949240 [https://perma.cc/S7FD-
U26R].
20. See Pilkington, supra note 12.
21. Keith Kloor, Anatomy of a Conspiracy Theory, POLITICO MAG. (Nov. 16, 2018),
https://www.politico.com/magazine/story/2018/11/16/conspiracy-theory-las-vegas-shootingdangerous-222576
[https://perma.cc/Q7NP-64QH].
22. See Matthew Yglesias, The Parkland Conspiracy Theories, Explained, VOX (Feb. 22,
2018, 8:00 AM), https://www.vox.com/policy-and-politics/2018/2/22/17036018/parklandconspiracy-theories
[https://perma.cc/4HC3-M5CU].
23. See id.
24. P. 199. Teddy receives attention too for his experience listening to Douglas and his
quiet suffering. Pp. 107–08. In the novel, we see and experience the cyber mob’s fixation on
Calvin.
1080 Michigan Law Review [Vol. 118:1073
Sandra is the least-known character. We only see snapshots of her suffering.
There is no inkling that the cyber mob demeans or threatens Sandra
for her gender, sex, or sexuality. This is a glaring omission. Women and minorities
are disproportionately the targets of cyber-mob harassment.25
Often,
the abuse is sexually threatening and sexually humiliating.26
Threats lobbed
at victims typically include death and rape threats and threats of anal rape.
Privacy invasions aimed at victims typically involve the nonconsensual posting
of nude photos and sex videos.27
The novel’s failure to highlight this reality
is disappointing, and it detracts from an otherwise compelling story.
This is not to discount the novel’s contribution. In focusing on a white
man with a steady job in the military, Drnaso does something important. He
shows us that cyber mobs can terrorize even the strongest among us. Sabrina
gives people from dominant groups reason not to dismiss cyber harassment
as no big deal. Far too many people, including law enforcement, tell victims
to ignore online abuse.28
As civil rights scholar Derrick Bell argued, the fight against discrimination,
hatred, and intolerance can only prevail if it advances white self-
interest.29
An agenda to combat cyber-mob abuse will succeed only if dominant
groups join the fight. Drnaso’s graphic novel will help convince the
people who most need convincing to care about the destruction wrought by
cyber mobs.
II. DIAGNOSING THE PROBLEM
The dark side of online life depicted in Sabrina is well-trodden ground
for me. My scholarship has explored the forces animating destructive online
behavior, including cyber stalking and sexual-privacy invasions.30
Drnaso’s
25. CITRON, supra note 11, at 13–19; Danielle Keats Citron, Cyber Civil Rights, 89 B.U.
L. REV. 61, 65–66 (2009) [hereinafter Citron, Cyber Civil Rights]; Danielle Keats Citron, Essay,
Law’s Expressive Value in Combating Cyber Gender Harassment, 108 MICH. L. REV. 373, 374–
75 (2009) [hereinafter Citron, Law’s Expressive Value in Combating Cyber Gender Harassment];
see also Danielle Keats Citron, Sexual Privacy, 128 YALE L.J. 1870, 1924–28 (2019)
[hereinafter Citron, Sexual Privacy].
26. CITRON, supra note 11, at 17.
27. Id.
28. See id. at 73–91; Citron, Law’s Expressive Value in Combating Cyber Gender Harassment,
supra note 25, at 392–95.
29. DERRICK BELL, AND WE ARE NOT SAVED: THE ELUSIVE QUEST FOR RACIAL JUSTICE
61–62 (1987); Citron, Cyber Civil Rights, supra note 25, at 84–85 (arguing that an agenda to
combat destructive cyber mobs can only succeed if, as Professor Bell argued, people from dominant
groups see that such attacks can be targeted at them).
30. See, e.g., CITRON, supra note 11; Citron, Cyber Civil Rights, supra note 25; Citron,
Law’s Expressive Value in Combating Cyber Gender Harassment, supra note 25; Citron, Sexual
Privacy, supra note 25; Danielle Keats Citron & Mary Anne Franks, Criminalizing Revenge
Porn, 49 WAKE FOREST L. REV. 345 (2014). Mary Anne Franks has brilliantly explored the destruction
wrought by online abuse. See, e.g., Mary Anne Franks, Sexual Harassment 2.0, 71 MD.
L. REV. 655 (2012); Mary Anne Franks, Unwilling Avatars: Idealism and Discrimination in Cyberspace,
20 COLUM. J. GENDER & L. 224 (2011).
April 2020] Cyber Mobs, Disinformation, and Death Videos 1081
novel contributes to the discussion by surfacing pathologies of online interactions
peculiar to this moment. This Part will explore those pathologies, the
financial incentives for tolerating destructive abuse, and the harm suffered.
A. Cognitive Biases Behind Clicking, Liking, and Sharing
Today, opportunities abound to share video and audio content online.
Facebook Live, YouTube, TikTok, and Snapchat let people broadcast video
and audio to audiences that span the globe.31
In the novel, the killer’s execution
video made it to the public eye at a snail’s pace as compared to the Denver
killer’s post. The trend is instant access to real-time content.
Livestreamed content has included murders, shootings, rapes, assaults, suicides,
and attempted suicides.32
When video and audio content surfaces online, as it did in Sabrina, it
may go viral. Cognitive biases help explain why certain content grabs our attention.
We have a visceral reaction to audio and video recordings.33
Audio
and video allow us to become firsthand witnesses to events, eliminating the
need to trust what others say happened.34
Their accuracy is self-evident—we
trust our eyes and ears to tell us the truth.35
People will be especially likely to like, link, and share video and audio
content if it is provocative and salacious. Human beings are naturally attracted
to negative and novel information.36
Researchers have found that
online hoaxes spread ten times faster than accurate stories because they are
more novel than real news.37
According to the study, people—not bots—
were responsible for sharing the fake news.38
31. Katie Conner, YouTube, Facebook, Instagram, Twitter Have a Dark Side. Here’s How
to Report Abuse, CNET (June 13, 2019, 5:00 AM), https://www.cnet.com/how-to/youtubefacebook-instagram-twitter-have-a-dark-side-heres-how-to-report-abuse
(on file with the
Michigan Law Review).
32. Mary Anne Franks, Justice Beyond Dispute, 131 HARV. L. REV. 1374, 1375 (2018)
(reviewing ETHAN KATSH & ORNA RABINOVICH-EINY, DIGITAL JUSTICE: TECHNOLOGY AND
THE INTERNET OF DISPUTES (2017)).
33. Drnaso explains that, as a teenager, he watched beheading videos. Max, supra note 6.
He felt “compelled to watch.” Id. He acknowledged why so many people click on execution
videos: “It sounds really sick, but a lot of times you’d just end up in tears, but you’d feel something
visceral. The feeling is something.” Id.
34. See Jennifer L. Mnookin, The Image of Truth: Photographic Evidence and the Power
of Analogy, 10 YALE J.L. & HUMAN. 1, 1–4 (1998).
35. Id. The Supreme Court has endorsed the truth-telling power of audio and video: if a
video shows someone driving recklessly, then the person drove recklessly. Scott v. Harris, 550
U.S. 372, 380–81 (2007) (ruling that a videotape of a car chase constituted definitive proof of
facts so as to preclude the necessity of a trial on the merits).
36. Chesney & Citron, supra note 10, at 1766–67.
37. Soroush Vosoughi et al., The Spread of True and False News Online, 359 SCIENCE
1146, 1148–50 (2018).
38. Id. at 1146.
1082 Michigan Law Review [Vol. 118:1073
Attraction to the provocative helps explain why so many people downloaded
the execution video; viewed the Denver killer’s post; and liked,
shared, and linked to the conspiracy theories about Sabrina’s death. Alex
Jones propagated theories that the shooting massacre of children and teachers
at Sandy Hook Elementary School was a government-staged hoax, which
ginned up cyber-mob attacks on the parents of murdered children.39
The
abuse was so frightening that one family was forced to move seven times.40
Another explanation for the viral spread of content is the natural inclination
to share information that is consistent with our views. Psychologists
call that “confirmation bias.”41
Douglas, like the real-life Jones, mined people’s
fear of government, and online audiences propagated his falsehoods in
thousands of comments, tweets, and posts (p. 122).
Provocative audio and video recordings, gossip, and information confirming
our viewpoints are nothing new. But today’s networked tools change
the stakes of their publication and distribution. Social media platforms supercharge
human frailties by allowing us to instantly and widely share content
that involves video or audio, salacious lies, or views similar to our own
(or some combination of those biases).42
Indeed, platforms’ algorithms are
designed to do precisely that, as explored below.
People’s likes, links, and shares often snowball into what is known as an
“information cascade.”43
That happens when people pass on what others say
without checking on the information’s veracity.44
The more people pass on
information shared with them, the more its credibility grows.
Information cascades often spill over into the media. “[T]raditional
mass-audience outlets . . . take note of the surge of social media interest and
as a result cover stories that otherwise they might not have.”45
Just as the
media covered the false-flag conspiracy theories about the Sandy Hook massacre,
news outlets amplified the fictions about Sabrina’s murder.
Sabrina ably captures our current cultural moment, one that Mary Anne
Franks powerfully explores in her book The Cult of the Constitution.46
As
Franks explains, a growing view depicts the right to bear arms and the right
to free expression as under siege.47
Any restriction on gun ownership
39. See Elizabeth Williamson, Alex Jones, Pursued over Infowars Falsehoods, Faces a Legal
Crossroads, N.Y. TIMES (July 31, 2018), https://www.nytimes.com/2018/07/31/us
/politics/alex-jones-defamation-suit-sandy-hook.html [https://perma.cc/4NMZ-U3YK].
40. Id.
41. CITRON, supra note 11, at 67.
42. See Chesney & Citron, supra note 10, at 1766–67.
43. See generally DAVID EASLEY & JON KLEINBERG, NETWORKS, CROWDS, AND
MARKETS: REASONING ABOUT A HIGHLY CONNECTED WORLD (2010) (discussing information
cascades); CASS R. SUNSTEIN, REPUBLIC.COM 2.0 (2007) (discussing cyber cascades).
44. CITRON, supra note 11, at 67.
45. See Chesney & Citron, supra note 10, at 1766.
46. MARY ANNE FRANKS, THE CULT OF THE CONSTITUTION (2019).
47. Id. at 21, 116.
April 2020] Cyber Mobs, Disinformation, and Death Videos 1083
amounts to an unraveling of the right to self-defense.48
Gun rights organizations
have convinced their members “into believing that they are always just
one step away from being disarmed by the government, which cultivates a
collective paranoia and persecution complex.”49
Similarly, proposed speech
restrictions are viewed as a crisis for free expression.50
This “free speech orthodoxy
[has] allowed threats, conspiracy theories, defamation, and outright
lies to flourish unchecked in the media and on the Internet.”51
The absolutist, cult-like obsession with guns and speech that Franks discusses
pervades Sabrina. Douglas propagates a false-persecution narrative.
He argues that the government is poised to take away people’s internet and
guns (p. 138). Douglas is the victim of a government trying to silence him. “I
have been targeted for voicing what amounts to perfectly legal and acceptable
free speech. If you ever see me being taken away in handcuffs, you’ll
know what’s going on,” he argues (p. 89).
But whose speech is really in jeopardy of being silenced? Douglas’s violent
rhetoric and false statements incite his followers to harass and stalk the
family of a murdered young woman. That sort of cyber harassment has been
empirically shown to chill the free expression of victims.52
Online falsehoods,
privacy invasions, and threats imperil targeted individuals’ life opportunities,
including their ability to express themselves.53
Franks contends that an illusion of “constitutional scarcity” has been
created, which has been met with demands for “more” guns and speech.54
As
Franks compellingly argues, “[w]e are facing a continuing crisis of constitutional
inequality” rather than a scarcity of gun and speech rights.55
As she
underscores, constitutional principles are often invoked to deny equal opportunity
to women and minorities.56
We see them deployed to justify defamation,
harassment, discrimination, and violence against the most
vulnerable.57
The cult-like obsession with gun rights and free speech is surfaced
in Sabrina, but without any recognition of the discrimination faced by
women and minorities trying to speak and work in our networked age.
48. See id. at 60–67.
49. Id. at 19–20.
50. See id. at 107–09, 115.
51. Id. at 16; see also id. at 116.
52. See CITRON, supra note 11, at 119; Danielle Keats Citron, Civil Rights in Our Information
Age, in THE OFFENSIVE INTERNET 31 (Saul Levmore & Martha C. Nussbaum eds.,
2010); Citron, Cyber Civil Rights, supra note 25, at 64; Citron & Franks, supra note 30, at 385;
Danielle Keats Citron & Neil M. Richards, Four Principles for Digital Expression (You Won’t
Believe #3!), 95 WASH. U. L. REV. 1353, 1365 (2018) (“[N]ot everyone can freely engage online.
This is especially true for women, minorities, and political dissenters who are more often the
targets of cyber mobs and individual harassers.” (footnote omitted)).
53. CITRON, supra note 11, at 5–12, 35–50.
54. FRANKS, supra note 46, at 21.
55. Id.
56. Id.
57. Id. at 106–07.
1084 Michigan Law Review [Vol. 118:1073
B. Harm
My scholarship has explored the devastating impact that cyber-mob
harassment has on individuals and society, so I won’t belabor the point. But
a brief overview of the harm helps put into poignant relief the novel’s grim
suggestion that cyber-mob attacks are as inevitable as they are disturbing. In
Drnaso’s fictional universe, this is just the way that we operate. We click,
like, and share gruesome videos and conspiracy theories. We hurl vitriol and
threats online and think nothing of it. The attacks on Sandra, Teddy, and
Calvin recede after another murder is attributed to the deep state. The cyber
mob does not skip a beat. It goes on to exact other pounds of flesh.
Cyber-mob attacks inflict profound harm. Targeted individuals fundamentally
alter their lives. They move,58
as in the Sandy Hook case.59
They
switch schools.60
They change their names because it is impossible to obtain
employment, find love, and meet clients when one’s Google search is filled
with threats, falsehoods, and privacy invasions.61
They lose their jobs and
have difficulty finding new ones.62
They experience profound emotional distress,
anxiety, and depression.63
They shut down their social media profiles,
blogs, and websites, because keeping them invites more abuse.64
We see some of this harm in the novel. Sabrina’s loved ones surely experienced
emotional distress when reporters asked them to address the hoax
allegations. Not only were they grieving, but then they were asked to justify
their grieving.65
Calvin has nightmares. Sandra is terrified and exhausted.
58. CITRON, supra note 11, at 6.
59. Williamson, supra note 39.
60. See id.
61. See id.
62. CITRON, supra note 11, at 7–10.
63. Id. at 10–11.
64. Id. at 1–4, 8–9, 193–99.
65. Susan Svrluga, First, They Lost Their Children. Then the Conspiracy Theories Started.
Now, the Parents of Newtown Are Fighting Back, WASH. POST (July 8, 2019, 4:57 PM),
https://www.washingtonpost.com/local/education/first-they-lost-their-children-then-the-
conspiracies-started-now-the-parents-of-newtown-are-fighting-back/2019/07/08/f167b880-
9cef-11e9-9ed4-c9089972ad5a_story.html [https://perma.cc/77VT-55N8] (explaining that a
Sandy Hook parent suffered not only because he lost his son but because he had to prove that
his son had lived and that he had died). Cyber mobs have long tormented people who are
grieving loved ones. Whitney Phillips, LOLing at Tragedy: Facebook Trolls, Memorial Pages
and Resistance to Grief Online, FIRST MONDAY (Dec. 5, 2011), https://firstmonday.org
/article/view/3168/3115 [https://perma.cc/CZX6-LUYJ] (explaining the phenomenon of RIP
trolling—where people scour Facebook for sensitive people to exploit, notably people in
mourning who set up memorial or RIP pages, and then target them with disturbing photos and
cruel comments). See generally WHITNEY PHILLIPS, THIS IS WHY WE CAN’T HAVE NICE
THINGS (2015) (discussing how internet trolls traumatize grief-stricken families). The grieving
students who survived the Marjory Stoneman Douglas High School shooting have faced cruel
online abuse. Stephanie Ebbs, Social Media Companies Block Abuse of Parkland Shooting Survivors
Online, ABC NEWS (Feb. 21, 2018, 9:07 PM), https://abcnews.go.com/Politics/social-
April 2020] Cyber Mobs, Disinformation, and Death Videos 1085
The societal harm is potent. Viral conspiracy theories and falsehoods
undermine our sense of a shared reality. This is a perilous time for the pursuit
of truth.66
Even the president of the United States cries “fake news” and
propagates fringe theories on his official presidential Twitter account.67
Things are poised to take a turn for the worse. Soon, deep-fake video
and audio content, only alluded to in Sabrina but now on the minds of Congress
and the press, may be so sophisticated that distinguishing genuine content
from fakery will be difficult, if not impossible.68
The truth will have
difficulty emerging in a deep-fake-ridden marketplace of ideas.
C. Incentives of Platforms
Fred Guttenberg’s daughter Jaime was murdered at the Marjory Stoneman
Douglas school shooting on February 14, 2018.69
Guttenberg turned his
grief into online and offline activism for gun-control legislation. On August
1, 2019, he tweeted that he was “[r]eceiv[ing] targeted harassment on Twitter”
from someone “using a made up profile with the name of my daughters
[sic] killer.”70
Twitter initially refused to take action.71
This is as unsurprising as it is depressing. Allowing attention-grabbing
abuse to remain online accords with platforms’ rational self-interest. Social
media-blocks-abuse-parkland-shooting-survivors-online/story?id=53250460 [https://perma.cc
/CNM6-H9DA].
66. Chesney & Citron, supra note 10, at 1763–68.
67. Edgar Alvarez, Trump’s ‘Social Media Summit’ Was a Celebration of Conspiracy
Theorists, ENGADGET (July 12, 2019), https://www.engadget.com/2019/07/12/trump-socialmedia-summit-white-house-facebook-twitter/
[https://perma.cc/A8HB-YMG2]; Justin Hendrix,
Trump’s Encouraging QAnon May Result in Violence—Just Ask the FBI, JUST SECURITY
(Aug. 1, 2019), https://www.justsecurity.org/65659/trumps-encouraging-qanon-may-result-inviolence-just-ask-the-fbi/
[https://perma.cc/R3MW-MFKD].
68. Chesney & Citron, supra note 10, at 1760. I discussed deep fakes in testimony before
the House Intelligence Committee in June 2019 and at my TED talk at the TED Global Summit
held in July 2019. See The National Security Challenge of Artificial Intelligence, Manipulated
Media, and ‘Deep Fakes’: Hearing Before the H. Permanent Select Comm. on Intelligence, 116
Cong. (2019) (written testimony of Danielle Keats Citron, Morton & Sophia Macht Professor
of Law, University of Maryland Carey School of Law), https://docs.house.gov/meetings/IG
/IG00/20190613/109620/HHRG-116-IG00-Wstate-CitronD-20190613.pdf [https://perma.cc
/J3C2-U9SA]; Ann Powers et al., The Big Rethink: Notes from Session 3 of TEDSummit 2019,
TEDBLOG (July 23, 2019, 4:15 PM), https://blog.ted.com/the-big-rethink-notes-from-session-
3-of-tedsummit-2019 [https://perma.cc/SER6-MXLS]; TED, How Deepfakes Undermine Truth
and Threaten Democracy, YOUTUBE (Oct. 7, 2019), https://www.youtube.com
/watch?v=pg5WtBjox-Y.
69. Jessica McBride, Fred Guttenberg, Jaime’s Father: 5 Fast Facts You Need to Know,
HEAVY (Sept. 4, 2018, 2:26 PM), https://heavy.com/news/2018/02/fred-guttenburg-videojaime-father-marco-rubio-cnn/amp
[https://perma.cc/9QV3-5ESP].
70. Morgan Gstalter, Parkland Father: Twitter Did Not Suspend Users Who Harassed Me
Using Name of Daughter’s Killer, HILL (Aug. 1, 2019, 12:46 PM), https://thehill.com
/policy/technology/455759-parkland-father-twitter-did-not-suspend-user-who-harassed-meusing-name-of
[https://perma.cc/Y3UB-3MNL].
71. Id.
1086 Michigan Law Review [Vol. 118:1073
media companies earn advertising revenue when users like, click, and
share.72
“They produce nothing and sell nothing except advertisements and
information about users, and conflict among those users may well be good
for business.”73
If a company’s analytics suggest that people pay more attention
to content that makes them sad or angry, then the company will highlight
such content.74
As Federal Trade Commissioner Rohit Chopra powerfully warned in his
dissenting opinion in the agency’s 2019 settlement with Facebook, the behavioral
advertising business model is the “root cause of its widespread and
systemic problems.”75
Online behavioral advertising generates profits by
“turning users into products, their activity into assets,” and “platforms into
weapons of mass manipulation.”76
Tech companies “have few incentives to
stop [online abuse], and in some cases are incentivized to ignore or aggravate
[it].”77
To be sure, tech companies do moderate certain content by shadow
banning, filtering, or blocking it.78
They have acceded to pressure from the
European Commission to remove hate speech and terrorist activity.79
They
have banned certain forms of online abuse, such as nonconsensual pornography,
in response to pressure from users, advocacy groups, and advertis-
ers.80
When it is bad for business, platforms have expended resources to stem
abuse.81
Social media platforms generally do not filter or remove content for the
sake of accuracy or quality.82
That makes sense given that their business
model is premised on clicks, views, and shares and that law supports this
laissez-faire approach.
72. Franks, supra note 32, at 1381.
73. Id. at 1386.
74. Dissenting Statement of Commissioner Rohit Chopra, In re Facebook, Inc., Commission
File No. 1823109, at 8 (July 24, 2019), https://www.ftc.gov/system/files/documents
/public_statements/1536911/chopra_dissenting_statement_on_facebook_7-24-19.pdf [https://
perma.cc/5X5M-2WCN].
75. Id. at 2.
76. Id.
77. Franks, supra note 32, at 1386.
78. Danielle Keats Citron, Extremist Speech, Compelled Conformity, and Censorship
Creep, 93 NOTRE DAME L. REV. 1035, 1038–39 (2018); Danielle Keats Citron & Helen Norton,
Intermediaries and Hate Speech: Fostering Digital Citizenship for Our Information Age, 91 B.U.
L. REV. 1435, 1468–71 (2011).
79. Citron, supra note 78, at 1037.
80. Id.
81. See CITRON, supra note 11, at 229 (discussing how Facebook changed its position on
pro-rape pages after fifteen companies threatened to pull their ads); see also Mary Anne
Franks, “Revenge Porn” Reform: A View from the Front Lines, 69 FLA. L. REV. 1251, 1270–71
(2017).
82. Chesney & Citron, supra note 10, at 1765.
April 2020] Cyber Mobs, Disinformation, and Death Videos 1087
III. ASSIGNING RESPONSIBILITY
Sabrina depicts the internet as a virtual Wild West. Seemingly, nothing
is off-limits. False assertions that Calvin, Teddy, and Sandra are crisis actors
are plastered online, as are their home addresses and contact information
(pp. 120–22). Death threats appear on message boards and in emails and
texts (pp. 120, 154). Sabrina’s execution video is downloaded on millions of
computers (p. 108). Neither law nor self-regulation seems to have a role in
slowing down or stopping the destruction wrought by Douglas and his cyber
mob.
Reality is not as grim as Sabrina suggests, but it isn’t great either. Networked
interactions are not totally outside law’s reach. Law provides some
redress for victims. But for the most part, only individual perpetrators can be
sued. Thanks to federal law, platforms are largely immune from liability.
This Part will focus on avenues for redress and on the role of education.83
A. Suing Harassers
Cyber-mob members could be sued for their reputation-harming lies,
privacy invasions, and threats. Various tort claims come to mind, including
defamation, public disclosure of private fact, and intentional infliction of
emotional distress.84
The success of these claims depends upon victims’ ability
to afford counsel, to identify perpetrators, and to have access to a court
with jurisdiction over the perpetrator.85
A considerable number of variables
are thus at play, and just one could stop a lawsuit in its tracks.86
Even if those concerns are surmounted, lawsuits are invasive and take
considerable time and money. Several Sandy Hook parents have sued Alex
Jones for defamation. While the lawsuits have overcome initial challenges to
the pleadings, many steps remain before plaintiffs get their day in court.87
Despite these limitations, tort claims are valuable. They teach us about
what conduct is harmful and wrong, and they secure redress for victims.88
Renowned privacy lawyer Carrie Goldberg has obtained civil remedies for
clients who have been harassed online. As her website notes, one of things
she does is “[s]ue the hell out of [people] who hurt you.”89
Simply said, and
83. I have explored the role of criminal law in addressing cyber-mob attacks elsewhere.
See, e.g., CITRON, supra note 11, at 123–41, 157–61; Citron, Cyber Civil Rights, supra note 25, at
86–97; Citron & Franks, supra note 30, at 365–74.
84. CITRON, supra note 11, at 121–23 (discussing various potential claims that can be
brought against online harassers).
85. Id. at 122; cf. Chesney & Citron, supra note 10, at 1792–93.
86. Chesney & Citron, supra note 10, at 1792–93.
87. See Svrluga, supra note 65.
88. Citron, Law’s Expressive Value in Combating Cyber Gender Harassment, supra note
25.
89. C.A. GOLDBERG VICTIMS’ RIGHTS LAW FIRM, https://www.cagoldberglaw.com
[https://perma.cc/WZD5-HSG6].
1088 Michigan Law Review [Vol. 118:1073
abundantly true.90
From her perspective, tort suits can be an important tool
to secure redress if perpetrators can be identified and have some funds to re-
cover.91
What about the deep-pocketed platforms? After all, social media companies
are best situated to minimize the damage. Through their design
choices and speech policies and procedures, platforms control what content
appears on their services. Shouldn’t they be responsible for those decisions?
Thanks to the broad judicial interpretation of a federal law passed in
1996, tech companies are largely immune from liability for their users’ illegal
conduct.92
That is why Goldberg’s law firm is focusing on chipping away at
that broad interpretation in the courts or—as she hopes and as I agree—on
working to help Congress fix it.93
B. Holding Platforms Accountable
In Sabrina, social media platforms are everywhere and nowhere. On the
one hand, they are front and center in the story. Their services facilitate the
novel’s action. On the other, the reader sees and hears little from them. The
platforms seemingly take no action vis-à-vis the online abuse. They appear to
proceed as if they have no responsibility for the havoc of their users.
Drnaso has elegantly depicted reality. The invisibility of platforms captures,
in important respects, federal law, which has left little risk of liability
and no requirement of responsible content moderation.
The federal law in question is section 230 of the Communications Decency
Act of 1996.94
In enacting section 230, Congress sought to provide an
90. I have watched Goldberg’s practice from its start. In five years, she has become the
nation’s leading lawyer in cases involving online victimization and sexual-privacy invasions.
See Margaret Talbot, The Attorney Fighting Revenge Porn, NEW YORKER (Nov. 27, 2016),
https://www.newyorker.com/magazine/2016/12/05/the-attorney-fighting-revenge-porn
[https://perma.cc/9X72-VTYB]. Goldberg and I have served on the board of directors of, and
acted as advisers to, the Cyber Civil Rights Initiative together since 2013. CCRI Board of Directors,
CYBER CIVIL RIGHTS INITIATIVE, https://www.cybercivilrights.org/ccri-board/
[https://perma.cc/68EB-JG3J]. Goldberg has published a book on her life’s work and her own
experience with online harassment. CARRIE GOLDBERG, NOBODY’S VICTIM: FIGHTING
PSYCHOS, STALKERS, PERVS, AND TROLLS (2019).
91. Email from Carrie Goldberg to author (Aug. 2, 2019, 9:21 AM) (on file with author).
92. See, e.g., Zeran v. Am. Online, Inc., 129 F.3d 327, 332–33 (4th Cir. 1997).
93. Id.
94. 47 U.S.C. § 230(c) (2018). I have written a considerable amount about section 230,
so I am just going to provide a brief summary and then focus on recent developments. For a
detailed explanation of section 230, its purpose, and broad judicial interpretation, see CITRON,
supra note 11, at 170–79, and Danielle Keats Citron & Benjamin Wittes, The Internet Will Not
Break: Denying Bad Samaritans § 230 Immunity, 86 FORDHAM L. REV. 401 (2017). My ideas for
fixing it have developed over the years, from early on urging a reasonableness approach including
traceable anonymity, Citron, Cyber Civil Rights, supra note 25, at 121–24, to proposing a
more narrow, targeted statutory fix, CITRON, supra note 11, at 177, to a statutory fix that conditions
the immunity on reasonable content-moderation practices, Citron & Wittes, supra at
419. I testified about potential fixes to section 230 before the House Energy and Commerce
April 2020] Cyber Mobs, Disinformation, and Death Videos 1089
incentive to online service providers to moderate “offensive” content.95
That
incentive was immunity from liability as publishers or speakers for filtering
or blocking too much or too little of their users’ online speech.96
Section 230’s immunity provision has been broadly interpreted in the
courts.97
Section 230 would immunize platforms from liability for claims related
to the posting of defamatory falsehoods, threats, and personal infor-
mation.98
The immunity would stick even if platforms knew about users’
tortious and illegal activity and refused to do anything.99
It would stick even
if platforms urged users to engage in tortious and illegal activity.100
And it
would stick even if platforms designed their sites to enhance the visibility of
content that would obviously involve tortious and illegal activity.101
The
takeaway message to victims is “too bad, so sad.”
Goldberg has been fighting to change the overbroad interpretation of
section 230. Her client, Matthew Herrick, sued Grindr for negligently designing
its dating app.102
Grindr was notified approximately 100 times that
Herrick’s ex-boyfriend was impersonating him on the app, sharing his nude
images, claiming he had rape fantasies, and providing his home address.103
Herrick’s ex’s use of Grindr was endangering his life.104
As many as twentyCommittee
in October 2019. See Fostering a Healthier Internet to Protect Consumers: Hearing
Before the H. Comm. on Energy & Commerce, 116 Cong. (2019) (written testimony of Danielle
Keats Citron, Professor of Law, Boston University School of Law), https://docs.house
.gov/meetings/IF/IF16/20191016/110075/HHRG-116-IF16-Wstate-CitronD-20191016.pdf
[https://perma.cc/8XL9-5L9Z].
95. CITRON, supra note 11, at 170–73.
96. Citron & Wittes, supra note 94, at 404–06. Section 230(c), entitled “Protection for
‘Good Samaritan’ blocking and screening of offensive material,” has two key provisions. Section
230(c)(1) provides that online service providers will not be treated as publishers or speakers
of user-generated content. 47 U.S.C. § 230(c)(1). Section 230(c)(2) says that online service
providers will not be held liable for good-faith filtering or blocking of user-generated content.
Id. § 230(c)(2). Section 230 carves out exceptions from its immunity provisions—federal criminal
law, intellectual property law, and the Electronic Privacy Communications Act. Id.
§ 230(e).
97. Citron & Wittes, supra note 94, at 406–10.
98. See id. at 413–14.
99. See id. at 414.
100. Id. at 408.
101. Danielle Keats Citron, Section 230’s Challenge to Civil Rights and Civil Liberties,
KNIGHT FIRST AMEND. INST. (Apr. 6, 2018), https://knightcolumbia.org/content/section-230schallenge-civil-rights-and-civil-liberties
[https://perma.cc/NCD6-2NVW]. See generally Olivier
Sylvain, Intermediary Design Duties, 50 CONN. L. REV. 203 (2018).
102. Herrick v. Grindr, LLC, 306 F. Supp. 3d 579 (S.D.N.Y. 2018).
103. Id. at 585.
104. Email from Carrie Goldberg to author, supra note 91.
1090 Michigan Law Review [Vol. 118:1073
three strange men came to his apartment a day.105
In all, more than a thousand
men came to Herrick’s apartment.106
As Goldberg explains, Grindr was her client’s last and only hope for
help.107
Herrick had gotten an order of protection from family court and reported
the violations to the police a dozen times.108
But nothing and no one
helped.109
The order of protection was a paper tiger. Law enforcement took
Herrick’s complaint but then failed to investigate. Herrick’s ex-boyfriend
continued to post on Grindr and strangers continued banging on his door.
Grindr ignored Herrick’s complaints and refused to remove the imposter.110
The trial court dismissed Herrick’s claims on section 230 grounds,111
and the Second Circuit upheld that dismissal in a nonprecedential summary
order.112
Goldberg has “all but lost hope in there being a judicial fix to Section
230.”113
She notes that “[i]t used to be that for the cost of an index number,
the poorest person in the world could hold the most powerful
corporation accountable for the harms they caused. Those days are gone. We
need a [legislative] fix.”114
Goldberg is right. Unlike a few years ago when the notion of doing anything
about section 230 was viewed as madness, fixing section 230 is now a
real possibility.115
Mary Anne Franks and I are currently working with federal
lawmakers, both Democrats and Republicans, on potential legislative
changes to section 230.
Federal lawmakers have expressed interest in the statutory fix proposed
by Benjamin Wittes and me to condition immunity on reasonable contentmoderation
practices.116
The relevant part of the statute would read:
105. Id.
106. Sara Ashley O’Brien, 1,100 Strangers Showed Up at His Home for Sex. He Blames
Grindr, CNN: BUS. (Apr. 14, 2017, 1:02 PM), https://money.cnn.com/2017/04/14/technology
/grindr-lawsuit/index.html [https://perma.cc/978P-HWKL].
107. Email from Carrie Goldberg to author, supra note 91.
108. Id.
109. Id.
110. Herrick v. Grindr, LLC, 306 F. Supp. 3d 579, 593 (S.D.N.Y. 2018).
111. Id.
112. Herrick v. Grindr, LLC, 756 F. App’x 586 (2d Cir. 2019).
113. Email from Carrie Goldberg to author, supra note 91.
114. Id.
115. Danielle Citron & Quinta Jurecic, Platform Justice: Content Moderation at an Inflection
Point (Hoover Inst., Aegis Series Paper No. 1811, 2018), https://www.hoover.org/sites
/default/files/research/docs/citron-jurecic_webreadypdf.pdf [https://perma.cc/6XZY-9HBF];
Danielle Citron & Quinta Jurecic, Platform Justice: Content Moderation at an Inflection Point,
LAWFARE (Sept. 7, 2018, 1:56 PM), https://www.lawfareblog.com/platform-justice-contentmoderation-inflection-point
[https://perma.cc/W78N-2STL].
116. Indeed, at my recent testimony before the House Intelligence Committee, I engaged
in an extensive back and forth with lawmakers about our proposal. See, e.g., Danielle Citron
Explains Content Moderation, C-SPAN (June 14, 2019), https://www.c-span.org/video
/?c4802966/danielle-citron-explains-content-moderation [https://perma.cc/2GCQ-RW9A].
Tech companies have signaled their support as well. For instance, IBM issued a statement urg-
April 2020] Cyber Mobs, Disinformation, and Death Videos 1091
No provider or user of an interactive computer service that takes reasonable
steps to prevent or address unlawful uses of its services shall be treated as the
publisher or speaker of any information provided by another information
content provider in any action arising out of the publication of content provided
by that information content provider.117
If adopted, the question before the courts in a motion to dismiss on section
230 grounds would be whether a defendant employed reasonable content-moderation
practices. Whether a platform acted reasonably with regard
to a specific instance of speech would not be the question. Instead, the court
would ask whether the platform engaged in reasonable content-moderation
practices writ large and thus earned the immunity.
Goldberg’s case against Grindr is a helpful example. Did Grindr engage
in reasonable content-moderation practices? Not by my lights. The dating
app chose to design its service so that it could not ban abusive users. The
app’s design is arguably negligent and inherently defective. As Goldberg explains,
the company “had a defectively designed and manufactured product
since it was not just foreseeable but an arithmetic certainty that if you run a
dating app used by millions of people with geo-locating functions, some of
them will use it to stalk, rape, harass, and prey.”118
Herrick’s struggles—and
Grindr’s refusal to do anything—provide an illustration of the problem but
are not why Grindr should not enjoy section 230 immunity for Herrick’s
claims. The reason would be Grindr’s design choices that amounted to a
failure to take reasonable steps to prevent or address unlawful uses of its ser-
vices.
What about Douglas’s message board? If the message board were sued
for users’ defamation, for instance, the court would assess whether Douglas,
as site operator, engaged in reasonable practices to prevent or address illegality.
If, as the novel suggests, Douglas engaged in no content moderation in
the face of thousands of posts whose titles suggested tortious and illegal conduct,
then his speech policies and practices would be found unreasonable.
Douglas effectively called for his followers to hound and defame Sabrina’s
loved ones, further undermining any notion that he acted reasonably in the
face of illegality on his site.
C. Educating for Our Better Selves
Changing online norms must include efforts at education. Young or old,
middle-aged or teenaged, we click, like, link, and share without thinking. We
pass on information others share and like, and we think nothing of it. We
ing Congress to adopt our proposal. Ryan Hagemann, A Precision Regulation Approach to
Stopping Illegal Activities Online, IBM: THINK POL’Y (July 10, 2019), https://www.ibm.com
/blogs/policy/cda-230/ [https://perma.cc/HV4F-S3FD].
117. Citron & Wittes, supra note 94, at 419.
118. Email from Carrie Goldberg to author, supra note 91.
1092 Michigan Law Review [Vol. 118:1073
write threatening and ominous emails, safe behind a computer or phone
screen.
Sabrina brought those human frailties alive by showing us that hundreds
upon hundreds of people liked, clicked, and shared the destructive posts.
The numbers were staggering—millions downloaded the execution video
and more than 600 people wrote to Calvin in a single day.
It is uncontroversial to say that all of us have acted impulsively online.
In front of a screen, people feel anonymous. That feeling “frees people to defy
social norms.”119
It makes it easier to do and say things that we would never
do or say if we were in real space.120
When face-to-face with other people, we see their facial expressions and
body language. We hear the tone of their voice. Those cues remind us to
keep our behavior in check.121
Because those cues are often absent online, we
are more likely to resort to invective. We seem to have forgotten about our
shared humanity.122
We must do better. We have to acknowledge and discuss the human
frailties that lead to our unthinking clicking, liking, downloading, posting,
and sharing. We have to consider strategies that can help us stop and think
before posting, sharing, and liking content that is salacious, provocative, and
simply aligns with our viewpoints.
Right now, schools, teachers, and parents carry the load of that education.
Yes, we must continue to educate young people about their responsibilities
to other people when using internet-connected tools.123
We teach kids
about drunk driving. We must continue to teach them about the problem of
texting, tweeting, emailing, sharing, and posting without thinking about the
consequences of their actions.
Yet the broad number of people who fall prey to cognitive biases are far
beyond their school years. How do we teach people who are settled in their
ways to think about their role in a cyber mob? This is a hard task, but one
that we should not avoid.
We need to do a better job engaging in public conversations about
online abuse. Advocacy groups like the Cyber Civil Rights Initiative devote
considerable energy on social media platforms talking about the perils of
online abuse and the harm that it causes.124
Others can and should do the
same.
119. CITRON, supra note 11, at 58.
120. Id.
121. Id. at 59.
122. Id.
123. EMILY BAZELON, STICKS AND STONES: DEFEATING THE CULTURE OF BULLYING AND
REDISCOVERING THE POWER OF CHARACTER AND EMPATHY (2013).
124. See What We Do, CYBER CIVIL RIGHTS INITIATIVE, https://www.cybercivilrights.org
/our-services/ [https://perma.cc/BAD2-GBMS]. I am the vice president of the nonprofit the
Cyber Civil Rights Initiative (CCRI), which was founded by Holly Jacobs to combat nonconsensual
pornography. CCRI’s mission is to fight for civil rights and liberties in the digital age
April 2020] Cyber Mobs, Disinformation, and Death Videos 1093
Education efforts should involve the platforms. Platforms could design
their sites to remind people to think about what they are posting.125
Then
too, journalists need to learn about how they cover newsworthy events without
unnecessarily amplifying falsehoods, conspiracy theories, and threats.
CONCLUSION
Drnaso’s fictional world reflects our troubling reality. Every day, people
are radicalized online to wreak havoc and violence. On August 3, 2019, in El
Paso, Texas, a twenty-one-year-old man posted a racist manifesto online and
then walked into a Walmart with a powerful rifle, killing twenty people and
injuring many others.126
A Twitter profile under the killer’s name liked
tweets from a far-right YouTuber who works with Alex Jones and promotes
links with the #BuildTheWall hashtag.127
By all accounts, the killer trafficked
in hate and conspiracy theories online.
Drnaso invites a conversation about cyber mobs, conspiracy theories,
death videos, and the destruction that they cause. That conversation must
include not only why online norms and cultural attitudes are as they are but
also how we change them. This Review joins that conversation with suggestions
for law and education. Much work lies ahead of us.
and to combat online abuse that denies people of central rights and liberties. Mary Anne
Franks is CCRI’s president and legislative technology director.
125. Design features aimed to educate people about their cognitive biases could help platforms
make the case for the reasonableness of their content-moderation practices. See generally
WOODROW HARTZOG, PRIVACY’S BLUEPRINT: THE BATTLE TO CONTROL THE DESIGN OF NEW
TECHNOLOGIES (2018); M. Ryan Calo, Against Notice Skepticism in Privacy (and Elsewhere), 87
NOTRE DAME L. REV. 1027 (2012).
126. Jordyn Rozensky et al., Mass Shooting Kills 22 at El Paso Walmart, DAILY BEAST
(Aug. 3, 2019, 1:51 PM), https://www.thedailybeast.com/el-paso-police-respond-to-activeshooting-at-cielo-vista-mall
[https://perma.cc/56BR-TJRD].
127. Brett Barrouquere, El Paso Shooting Suspect May Have Authored Manifesto Containing
White Nationalist Talking Points, S. POVERTY L. CTR. (Aug. 3, 2019),
https://www.splcenter.org/hatewatch/2019/08/03/el-paso-shooting-suspect-may-haveauthored-manifesto-containing-white-nationalist-talking
[https://perma.cc/86X6-J2CU].
1094 Michigan Law Review [Vol. 118:1073