Researchers from the Pew Research Center conducted this analysis to examine how often online content that once existed becomes inaccessible. One part of the study looks at a representative sample of webpages that existed over the past decade to see how many are still accessible today. For this analysis, researchers collected a sample of pages from the Common Crawl web repository for each year from 2013 to 2023. They then tried to access those pages to see how many still exist.

A second part of the study looks at the links on existing webpages to see how many of those links are still functional. Researchers did this by collecting a large sample of pages from government websites, news websites and the online encyclopedia Wikipedia.

The researchers identified relevant news domains using data from the audience metrics company comScore and relevant government domains (at multiple levels of government) using data from get.gov, the official administrator for the .gov domain. They collected the news and government pages via Common Crawl and the Wikipedia pages from an archive maintained by the Wikimedia Foundation. For each collection, they identified the links on those pages and followed them to their destination to see what share of those links point to sites that are no longer accessible.

A third part of the study looks at how often individual posts on social media sites are deleted or otherwise removed from public view. The researchers did this by collecting a large sample of public tweets on the social media platform X (then known as Twitter) in real time using the Twitter Streaming API. They then tracked the status of those tweets for a period of three months using the Twitter Search API to monitor how many were still publicly available.

Key findings:

• A quarter of all webpages that existed at one point between 2013 and 2023 are no longer accessible, as of October 2023. In most cases, this is because an individual page was deleted or removed on an otherwise functional website.
• For older content, this trend is even starker. Some 38% of webpages that existed in 2013 are not available today, compared with 8% of pages that existed in 2023.
• 23% of news webpages contain at least one broken link, as do 21% of webpages from government sites.
• 54% of Wikipedia pages contain at least one link in their “References” section that points to a page that no longer exists.