Blog

The Master in Digital Text Analysis is one of the University of Antwerp’s most recent additions to its curriculum. Although it starts from scratch – participants start with a 3-week bootcamp about modern computer programming featuring Python, R, Git, and so on – it covers a wide array of text technologies. Especially natural language processing, data science and machine learning are prominent. Above all, the master delivers a practice-oriented educational programme through a project-based thesis and the possibility of an external internship.

Apart from courses such as “Text as Data”, “Corpus Studies”, “Machine Learning” and “Computational Literary Studies”, the master also features an introduction into Information Science. And, as a PhD in Classics turned software engineer, who works for the University Library’s software department Anet, the organizers thought me sufficiently versed in both the digital and the humanities world to teach this course.

In the Information Science course, which is freely available as a Jupyter Book, I teach a brief, but hopefully representative array of information topics, such as text encoding, database models, query languages, metadata, and indexing – with Python as the main tool for automation. Again, the viewpoint is very practical. In fact, I guarantee my students that I use virtually all of the technologies we discuss on a daily basis, for instance:

• fetching an XML response from an API and parsing it in Python
• storing data in SQLite databases and performing SQL queries against them
• converting metadata from one to another standard, e.g. MARC to Dublin Core

Each of the chapters also features a coding exercise and the course exam is a Python project, organized as a Jupyter Notebook uploaded to a private GitHub repository.

For my teaching, I make good use of CERL’s freely available datasets, especially the CERL thesaurus and the HPB (Heritage of the Printed Book Database). These are of special interest to the course, not only because of the metadata they provide, but also because CERL offers programmatic access to them through SRU – Search/Retrieve via URL, which is an XML-based protocol utilizing contextual query language or CQL, a formal language for representing queries to information retrieval systems.

Both are standards that are developed and maintained by the Library of Congress.

A valid SRU request always contains a reference to the SRU “version” and an “operation”, optionally enriched with “parameters”.

The explain operation allows a client to retrieve a description of the facilities available by an SRU server. It can then be used by the client to self-configure and provide an appropriate interface to the user. The response includes a list of all the searchable indexes.

e.g. https://data.cerl.org/thesaurus/_sru?version=1.2&operation=explain

The searchRetrieve operation allows the client to submit a search and retrieve request for matching records. This operation needs to be combined with the query parameter

e.g. https://data.cerl.org/thesaurus/_sru?version=1.2&operation=searchRetrieve&query=masenius

Note the tag <srw:numberOfRecords>. Most SRU servers will not yield the entire response in one go. One can use the parameters &startRecord= and &maximumRecords= to harvest the whole result in chunks. For instance:

http://sru.gbv.de/hpb?version=2.0&operation=searchRetrieve&query=lipsius

needs to be followed up by

…query=lipsius&startRecord=1&maximumRecords=100

…query=lipsius&startRecord=100&maximumRecords=100

and so on.

A SRU query is expressed in CQL syntax.

The simplest CQL queries are unqualified single terms:

e.g. …query=lipsius

Queries may be joined together using the three Boolean operators, and, or and not. We use spaces, or rather their URL encoded version, to separate CQL words:

e.g. …query=lipsius%20or%20erasmus

The queries discussed so far are targeted at whole records. Sometimes we need to be more specific, and limit a search to a particular field of the records we’re interested in. In CQL, we do this using indexes. An index is generally attached to its search-term with an equals sign:

e.g. …query=ct.imprintname=moretus

SRU also allows other relations than equality (=) which we have just used (e.g. publicationYear<1980) and pattern matching:

e.g. …query=m*retus

The above examples mainly featured the CERL thesaurus SRU server but this year’s Information Science exam project relied on the HPB database SRU access.

The task was to write an STCV – HPB reconciliator. Briefly said, a piece of Python software that suggests pairings between STCV (Short Title Catalogue Flanders) and HPB identifiers. The dataset for the former was an SQLite export of STCV, freely available (besides MARCXML and CATXML) as Anet Open Data, while for the latter the SRU access was to be used.

This task involved many of the skills introduced in the course:

• Understanding different cataloguing and metadata standards
• Handling SQLite databases and executing SQL queries
• Using the SRU protocol with explain and searchRetrieve
• Writing complex CQL queries with indices and Booleans
• Retrieving URL requests and handling potential URL or HTTP errors
• Parsing and querying XML with namespaces
• Filtering, ranking and presenting search results

And while some students struggled, many were indeed able to come up with elegant and interesting implementations of this, skilfully combining indices, performing additional queries when the first one did not yield results, or including meta-metrics about the accuracy of the result set and the influence of varying parameters.

In this way, the CERL datasets prove to be not only an excellent research source, but also a very welcome teaching tool for digital book history in particular and digital humanities in general.

For students of Digital Text Analysis – many of whom had never written a line of code before this term – they are not only a source to learn about APIs, protocols and XML, but a chance to acquire this digital skillset in a familiar context, given their background.

Daunting as it is for these humanities students to immerse themselves in the strange world of bits and bytes, being able to do so with good quality datasets about a familiar topic such as book history, is undoubtedly a great asset.

And that’s were CERL comes to the rescue!

Stadscampus, University of Antwerp