{"id":48,"date":"2017-06-05T17:03:30","date_gmt":"2017-06-05T17:03:30","guid":{"rendered":"http:\/\/joshuacatalano.org\/blog\/?p=48"},"modified":"2017-06-05T17:41:23","modified_gmt":"2017-06-05T17:41:23","slug":"mining-eleanor-roosevelts-my-day-columns","status":"publish","type":"post","link":"https:\/\/joshuacatalano.org\/blog\/uncategorized\/mining-eleanor-roosevelts-my-day-columns\/","title":{"rendered":"Mining Eleanor Roosevelt\u2019s My Day Columns"},"content":{"rendered":"
\n

This post is co-written by Amanda Regan and Joshua Catalano. It is cross-posted on Mandy\u2019s blog<\/a>.<\/em><\/p>\n

\"\"<\/a>Two weeks ago, Amanda Regan and I\u00a0participated in the Collections as Data Hack-to-Learn<\/a> event sponsored by the Library of Congress, George Washington University, and George Mason University. For the event, George Washington University, the Library of Congress and the Smithsonian provided participants with a series of datasets. The corpus that we gravitated toward included all of Eleanor Roosevelt’s My Day<\/em> columns. During the workshop, we ran a 60 topic model on the columns. Due to time constraints and technical challenges during the workshop we were not able to accomplish much beyond creating the topics. The dataset, composed of almost 8,000 columns, resulted in large files that required manipulation with a tool such as R in order to create visualizations and analyze the results. This made it difficult for participants to do much with the topic modeling results during the workshop. Over the last two weeks, we have worked to take the topic model, manipulate the results, and create an interactive app with several visualizations that allows for an exploration of the topic model. This blog post will detail some of the process of making this visualization and describe how it might be used and interpreted.<\/p>\n

About Eleanor Roosevelt\u2019s My Day Columns<\/h3>\n

From December 1935 until September 1962, Eleanor Roosevelt wrote a nationally syndicated newspaper column entitled My Day<\/em>. Because Roosevelt did not keep any sort of diary and her appointment books were often incomplete, the \u201cMy Day\u201d columns have become a valuable source. The columns, authored six days a week until January 1961 and then three days a week thereafter, represent the only consistent account of her public activities. Thanks to the work of the Eleanor Roosevelt Papers’ staff, who transcribed each My Day<\/em> column and made them available online, anyone with internet access can now explore this important piece of history.<\/p>\n

The online repository<\/a> of My Day<\/em> columns allows users to look for columns by year or find a particular word or phrase within the collection through keyword search. Keyword search however, requires a level of specificity and familiarity with the topics discussed within the archive. A topic model allows for a more thematic and interactive way of exploring the archive. Providing a thematic overview, a topic model allows users a different entry point into the archive generated by the content (words) of the documents rather than a predetermined organizing schemata such as Library of Congress subject headings or even keywords.<\/p>\n

Preparing the Data, Generating the Topic Model, and Creating Visualizations<\/h3>\n

Each file, originally in XML, contained not only the date and text of the column, but a location where the column was written. Using these documents, we wrote a parsing script in R that created a csv of all the related metadata and then created a text file of the column text. The script identified the column date, location name, and geolocation in each article and wrote it to a csv along with each filename. This created a base metadata file. Next, the script went through each xml file and pulled out just the plain text of the column and wrote it to a text file.<\/p>\n

We used MALLET<\/a> to divide all of the 7,987 My Day<\/em> columns (as plain text files) into 70 \u201ctopics.\u201d Each \u201ctopic\u201d is a set of words that frequently co-occur together throughout the corpus. These words indicate themes or discourses within the corpus. With Python and R, we then transformed the MALLET output into several different types of visualizations using Shiny<\/a> and a number of other packages. The benefit of using Shiny is that it allows users to change specific input parameters such as the topic, number of words, and the number of documents to display. The resulting app includes a word cloud representing the top words in the topic, a chart depicting the prominence of the topic over time, a map detailing the locations where the most prominent columns in the topic were written, and a list of the prominent columns in the topic (with a link to the text). Additionally, a tab at the bottom of the page allows users to view the top words in all 60 topics. Below are descriptions of each section of the visualization.<\/p>\n