Kelly Mears

Internet Archive

The nonprofit digital library founded in 1996 that crawls and preserves the web through the Wayback Machine and also collects books, recordings, films, and software, at the center of fights over copyright and of what the web remembers.

Data & Content3 min read413 words4 out · 2 in
also calledWeb archivingWARC

The Internet Archive is a nonprofit digital library based in San Francisco, founded by Brewster Kahle in 1996 with the stated aim of universal access to all knowledge. It's best known for preserving the web itself, but it also holds millions of digitized books, audio recordings, films, television news broadcasts, and old software, most of it free to use.

Its web collection began with crawls donated by Alexa Internet, a web-traffic company Kahle also founded. In 2001 the archive opened those captures to the public through the Wayback Machine, where anyone can type an address and see how a page looked on earlier dates. The collection now runs to hundreds of billions of captured pages. Pages are gathered by crawlers that follow links outward from seed lists, and captures are stored in WARC files, a container format standardized by ISO in 2009 that records each request and response along with metadata about the capture; see Web Crawler.

The archive is often the only surviving copy of large parts of the early web. When Yahoo shut down GeoCities in 2009, volunteers and the archive captured much of it. Wikipedia's editors run bots that replace dead citation links with archived copies, rescuing millions of references. Researchers, historians, journalists, and courts routinely use archived pages as evidence of what a site said at a given time; see Provenance.

Beyond the web, the archive scans books in its own centers and runs Open Library, which lends digital copies. Its software collections let visitors play old arcade, DOS, and console games directly in the browser, using emulators compiled for the web, and it kept much of the Flash era viewable after browsers dropped support; see Emulation.

That ambition has drawn lawsuits. In 2020, four major publishers sued over the archive's book lending, especially a "National Emergency Library" that lifted waitlists during the pandemic. The courts ruled against the archive in 2023 and on appeal in 2024, rejecting its fair-use defense for lending scans of copyrighted books. Record labels later sued over a project digitizing old 78 rpm records; see Public Domain.

The archive also shows how fragile preservation can be. In October 2024 it suffered a series of cyberattacks, including a breach of user account data, and several services went offline for days. A single institution holding so much of the record is a single point of failure, which is why it keeps copies in more than one location and why national libraries run their own web archives.

Linked from2