Research Guides: Analyzing Text Data: Open Source

Open Source Data for Text Mining

Open source data sources are freely available for use.

General Sources

Crossref Text and Data Mining
Crossref makes research objects easy to find, cite, link, assess, and reuse. We’re a not-for-profit membership organization that exists to make scholarly communications better. Our API allows researchers to easily harvest full-text documents from all participating members, regardless of whether the content is open access or subscription.
data.census.gov
Provided by the Census Bureau, use to find data garnered from current Decennial Censuses, Economic Censuses, and American Community Surveys
Data.gov
Data.gov collects US open government data into one searchable location.
Digital Public Library of America
The DPLA Pro site provides more information about our cultural heritage and ebooks programs for libraries, archives, museums, and other institutions.
Google Books
Harvard Dataverse
The Harvard Dataverse Repository is a free data repository open to all researchers from any discipline, both inside and outside of the Harvard community, where you can share, archive, cite, access, and explore research data. Each individual Dataverse collection is a customizable collection of datasets (or a virtual repository) for organizing, managing, and showcasing datasets.
Internet Archive & Open Library
Internet Archive is a non-profit library of millions of free books, movies, software, music, websites, and more.
OSF
OSF is a free, open platform to support your research and enable collaboration.
Project Gutenberg
Project Gutenberg is a library of over 70,000 free eBooks.
re3data
Registry of research data repositories, includes information on access, licensing and standards.
Wikidata
The free knowledge base with 107,094,469 data items that anyone can edit.

Humanities and Social Sciences Sources

Chronicling America
Administered by the Library of Congress. A good source for American newspapers and their reporting on world events from 1789-1963. There are a number of digitized articles on Japan, the first and second Sino-Japanese Wars and the Russo-Japanese War.
Culturomics
On December 16th, 2010, a team spanning the Cultural Observatory, Harvard, Encyclopaedia Britannica, the American Heritage Dictionary, and Google published a paper describing the Culturomics approach online in the journal Science, and at the same time launched the world's first real-time culturomic browser on Google Labs.
EEBO Early English Books Online
more...less...
EEBO contains over 100,000 books published in English since 1475. Based on Pollard & Redgrave's "Short Title Catalogue (1475-1640)" and Wing's "Short-Title Catalogue (1641-1700)," plus revised editions and supplements. Subjects covered include English literature, history, philosophy, linguistics, theology, music, fine arts, education, math, and science. Update: Texts continue to be added to the online version as they are scanned from the microfilm.
Europeana
Multi-lingual online collection of millions of digitized items from European museums, libraries, archives and multi-media collections.
Oxford Text Archive
A repository of full-text literary and linguistic resources.Thousands of texts in more than 25 languages. From the Bodleian Libraries at the University of Oxford.
Wordhoard
An application for the close reading and scholarly analysis of deeply tagged texts.

Health and Science Sources

PubMed Central Databases and Text Mining Tools
A collection of text mining tools to analyze scholarly publications and other data sources, such as Electronic Health Records.
arXiv
The ArXiv preprint server gives you access to over a half million articles on physics, mathematics, computer science, quantitative biology, quantitative finance, and statistics.
BioMed Central
Over 315,000 full-text, peer-reviewed science, technology and medicine articles are available for text and data mining.
PLOS
Public Library of Science. PLOS is a nonprofit, Open Access publisher empowering researchers to accelerate progress in science and medicine by leading a transformation in research communication.
Text Mining at PLOS
Information on text mining from the Public Library of Science (PLOS).
PubMed Central Databases and Text Mining Tools
A collection of text mining tools to analyze scholarly publications and other data sources, such as Electronic Health Records.

Business and Law Sources

Case Law Access Project
CAP includes all official, book-published state and federal United States case law — every volume or case designated as an official report of decisions by a court within the United States.

Additional Resources

HathiTrust and Text Mining at GWU
Information on text data mining using HathiTrust