How to Produce, part II: Processing your results

Continuing with my never-ending quest for the perfect research workflow.

The processing stage of research is usually the most challenging, thought-intensive part of the research process. Because I have oodles of primary sources and feel compelled to use them in order to make robust generalizations, and because more sources generally increase the odds of finding exceptions to the rule, this is where my research tends to bog down. So it would seem particularly important to manage tasks here, and to match your time and energy level to the sources you’ll be reading.

So why don’t we focus on a relatively simple task like reading the sources we’ve already collected? Nothing could be simpler right? And yet…. What follows are some of the generic factors that make doing this discrete historical research task easier or harder, in my experience. As usual, I think of these in terms of spectra, and YMMV.

Research task difficulty matrix

Any research task focusing on sources will require reading a particular type of source (1), that is in a particular format (4) in a particular language (3),  in a particular way (2). The reader will presumably take some kinds of notes on that source (5), and will know more or less about the general subject that source is discussing (6). So let’s briefly review each of these criteria:

  1. Some types of sources are harder to read and understand than others. Short, discrete sources such as letters, or individual newspaper articles, are presumably easier to read than much longer works.
  2. How you read any given source will be determined by your objective. English scholar John Guillory published a well-known thought-piece that discussed several ways in which scholars might read a particular work. Easiest, assuming you have text sources, is to simply use software to conduct searches for specific text strings – hopefully you can save your search and rerun it as needed (in DTPO, a smart group). Generally browsing a source, to get the gist, is also relatively simple. Slightly more taxing is to scan or skim: Guillory reserves scanning for the archives, where you are scanning for particular documents of interest, whereas he refers to skimming as the way in which scholars read other scholarship. Then of course there is close-reading, whether it be of the literary variety, carefully reading a work for its argument, or, more generally, interpreting a source.
  3. The language of the source is obviously of pivotal importance. Here I’m thinking specifically of how well you know the source’s language: well enough that you can fill in missing words cut off by the margin, or so poorly that you need a dictionary to look up the meaning of every fifth word? I wouldn’t recommend you tackling one of the latter sources if you are short on time and patience.
  4. The format of the source can also play a role in how easy or difficult it is for us to process it. Modern printed works are not only easy to read, but they are also easy to skim and scan – I can pick out the word “siege” at 50 paces in published correspondence. But once you start reading newspaper print c. 1702, it gets a smidge more difficult. And it’s much more challenging to try to skim or scan manuscript documents when you have to spend considerable mental energy just to make out the words.
  5. What you want to get out of the source will determine how you take notes on it. Simplest, and quickest, is to just skim through the text and pull out a few keywords. Sometimes my first step for longer, structured documents will be to summarize the contents: make a place-holding table of contents (with links back to the original PDF page) for a treatise, or dump an OCRed index of a secondary source into DTPO and parse them into individual records. I only take further notes on that source if/when I need to focus on a particular topic within that table of contents/index.
    Antient Policy of War table of contents (now I make the link in the URL field rather than in the content)

    Antient Policy of War table of contents [now I make the link in the URL field rather than in the content]

    Most time-consuming of all is to paraphrase – summarizing point by point can get old real quick, though it’s the best way to truly understand a source.
  6. Finally (though you can probably think of other factors), there’s the general question of how many references your source makes to other things, and how likely you are to know those other things. History is all about context, and not knowing your context is a historian-killer. As allusions to unknown things pile up in your reading, you either need to look them up (which takes time and energy trying to figure out where to find that missing info, and then to track that detour down), or your comprehension of the source suffers as a result (which quickly becomes frustrating, making you wonder whether you shouldn’t just quit). Hence the common strategy to explore different subjects within the same timeframe (or same country).

The above criteria are useful ways to categorize your research tasks based on their subjective difficulty level. Caveats abound, of course: some of these criteria may not be relevant to every scholar, the hardest tasks of one criteria may be much harder than the hardest tasks of some other criteria, and these criteria interact with one another. I’m tempted to assign points for each criteria: 0 points for an easy task, 1 point for a moderately difficult task, 2 points for a hard task – or maybe you have more trouble with a particular author’s handwriting, so points for that source are doubled… Then add them up to see what any given research task’s score would be. But such a procedure might scare the children. And a few historians as well.

Even if you don’t go crazy by creating difficulty scores for your sources, these criteria still give you a way to assess which of your sources are going to be easier or more difficult to process. You could then categorize the sources accordingly, so that when you have just a bit of time between classes and feel like doing some work, you can pull up your task list of published letters (for example) that you need to just browse or search through. PI only allows one context, and you might want to keep the physical context as distinct, so you could instead be precise when titling your tasks: do you need to scan that source for a particular detail, or do you only need to browse it to see what it covers? When you look through your to-do list, you could sort by task title and then focus on those action verbs (browse, scan, skim, close-read) that match your current mental state. For example: “Browse Marlborough’s Letters and Dispatches 1706: Dutch allies” is likely to be a task that could be done over several short periods of time, and would not require a particularly high level of mental energy – perhaps you’re simply using a nifty DTPO script to make a linked note to the original. Transcribing all those letters would be more difficult, but probably not as difficult as paraphrasing each letter point by point, or trying to reconstruct Marlborough’s view of his Dutch allies based off of fifty different letters where he referred to them. You decide what you feel like doing at any given point in time – the lists simply suggest tasks that match where you are at in the moment.

Spend a little time categorizing your tasks by the effort involved, and you should be able to slowly chip away at your work even when you’re not feeling particularly energetic.

Strategy the Second: Make the restart automatic

A second strategy, again not original to me, is to make it easy to restart the next time, to bookmark (write down) your mental location in the process. Too often we don’t do research regularly enough to keep the content fresh in our memory (“15 minutes every day”). It takes us so long to get restarted that our session ends before we’ve gained any momentum. The goal, then, should be to overcome the need to spend the first however-many minutes trying to remember where you left off, and where you were going. Once again, this is exactly what GTD’s Next Action procedure is all about: the action verb that starts a task title tells you exactly what you need to do, and what to do it to, while the context reassures you that you can actually do that discrete task in the place you are at. After you’ve created the task in a list, you’ve already done the “hard” work (but not really that hard) of deciding what you’ll need to do, which makes it easier to just do it in the future. In the realm of research, a couple possible applications of this principle come to mind.

First, another idea I found on some GTD forum somewhere: add an explanation of why you are doing something to the end of your task. Don’t, for example, just create a task that says “Read France Siege Laws.” Why exactly are your reading that chapter by France? Instead create a task that says “Skim France Siege Laws to see if he portrays evacuations as ritualized.” If you want to shorten it further: “Skim France Siege Laws: evacuations ritualized?” That explanation will tell you exactly why you’re reading France, and the specific action verb will remind you that you should be skimming it, not reading the whole thing for some other point he’s making. Avoiding the temptation to fully read the document is easier if you have the original source in DTPO, where its text will be visible to any future searches you perform. Trust your note-taking system (DTPO) to find the details when you need them, in other words, by including as many full-text sources as possible.

Second, you should get in the habit of externalizing your meta-thoughts during the processing stage. We’ve just mentioned creating a Next Action that includes a why at the end. But you could also start the next research session by rereading a short summary you wrote at the end of the last session: what did you learn about source X and how did that influence your current thinking? Such a summary will prime you to compare it with source Y, and make connections. Rereading the research question too, or perhaps the research question for the specific section you are working on, can’t hurt either. You could take this a step further by making a table or matrix where you summarize a whole series of questions (or variables, or test cases…) in columns and then list what each source says about them in rows. This would further allow you to easily compare, as well as add other variables, and sort and subsort and resort at will.

Now that I think about it, this is essentially the beginnings of a research  journal (see also here and here). I was never taught to use one, but it’s increasingly seeming like a good idea, especially for long-term projects that you pick up and put down again and again. A researcher’s journal should record research accomplished, explain research decisions made, and note research decisions still to make (the latter of which should also be tasks in PI, FWIW). Up to now I’ve tended to do one of three things:

  1. Make comments in my DTPO note files for each source – usually distinguished with brackets, e.g. “[Compare this with Johnson’s account]”
  2. Write ideas down on scratch pieces of paper and file them in hanging folders to consult later
  3. Store them willy-nilly in a DTPO Project folder:
I think I need a better research journal.

Yep. I need a better research journal system.

No wonder it’s hard to pick up my train of thought after a few days away, and no wonder I keep shifting my view of what my book project should look like.

So maybe instead, I should make a slightly more defined research journal system. The discussions of research journals that I’ve read online praise its utility on multiple levels, but, as usual, are mum on the mechanics of how exactly you should structure your journal. From what I can tell, they tend to be a running journal of thoughts about the project, structured by dated entries. If the most important thing for you is to see the progression of all your thoughts about the project, it would make sense to put them all in one big journal document. But given how many meta-thoughts I produce, I need a bit more granularity – not only to quickly find specific entries in the future, but also to prevent too much repetition in my own notes. No clearly defined place to put things = you keep writing down the same ideas over and over.

Fortunately I have a few ideas of my own – largely based off my categorization of a hundred slips of scribbled-upon paper that accumulated in my office over the past several years. I recommend several types of research journal notes for each research project – I use the Keyword metadata field in DTPO to indicate the type of research journal note (a Keyboard Maestro macro sets it with two keystrokes):

  • Org(anization): a document (or many) where you talk about how you should organize your paper. You could create several org documents, one for each possibility and open them side-by-side for comparison. Keep track of the pros and cons for each possibility, and why you went with the  organization you did. Maybe mention a model structure you’re aping. The key is to avoid the infinite regression of organizing and reorganizing the same project again and again.
  • Historio: a document where you keep a running journal of your thoughts on the historiography. This document might start as a bibliography – with links to the original works elsewhere in DTPO, and to your notes (i.e. summaries, paraphrases, etc. but not your own thoughts) – that you then annotate. Or you could just insert random general comments about the historiography (and link to examples in your other DTPO groups) as your research advances.
  • Dissem(ination): your ideas about where you would like to disseminate the end product and why. You could throw other materials into the project group as well: publisher’s style guide, correspondence with an editor… You could also include any reviewers’ comments (friends or anonymous) as well – or make a separate Feedback document type.
  • Search: a document listing all the searches you’ve performed (and need to perform), and where you’ve performed them. Might make sense to make a table, with rows listing each search term (siege, siege NEAR vigor…) in alpha sort order. Each column lists a different database or catalog (ECSU library, Google Books, Gallica, ECCO…) – listing the date you conducted an online search is also useful. Then just add an ‘X’ to each cell when you’ve completed that search in that catalog. Include your administrative notes on your archival research while you’re at it. Alternately, you could just blog about it. You could also mention particular genres of sources to consult (and make a task for each): “Skim field deputy’s letters: jealousy of Marlborough.”
  • Quote: Sometimes you’ll run across a contemporary quote that’s too good to hide. So make a quote document with the pithy observation, and a link back to the original, and put it in the project group where you think it’ll fit. In DTPO you could replicate it in the source tag as well.
  • Chunk: a document that holds little snippets of possible text for your future draft. Often times I’ll randomly come up with possible transition sentences, which get put here.
  • Thoughts: a document for any general thoughts about your project that you make when doing your GTD Weekly Review (or any other time). I just paste them from my Weekly Review thoughts document in Evernote. “Thoughts” is a pretty generic keyword, I admit, but it should serve as a catch-all. If you’re so inclined, you could even record the emotional ups-and-downs of the project: your concerns and anxieties about the project, your hopes upon its completion… Some people find this therapeutic.
  • Collab(oration): notes on any interesting discussions you’ve had with peers and colleagues. Be sure to date the conversation (and interlocutor) and include the medium of the discussion (email, phone, face-to-face…).
  • Method: a document with discussion of important methodological issues relating to the project. Could be tips and tricks for that particular (type of) project, could be statistical tests to try out, could be pitfalls and dangers of a given subject…
  • BigQuest(ions): a document type with an awkward name, but which seems important nonetheless. Sometimes half way through a project I’ll realize that there are some fundamental questions about the mechanics or dynamics of the subject under study that I really need to answer and haven’t. I put them here because I don’t know where else they should go.
  • Test/Hypothesis/Question: these could be separate types of documents, or a single document type. They include all those little ideas I have about how to analyze my research project, and are possibly different because of when they appear in your research:
    • Question: a broad question about the subject. I haven’t taken the time to operationalize it, so it stays an open-ended question.
    • Hypothesis: a theory about the subject that merits exploration – an operationalization of a question. Oftentimes I’ll have very specific “lessons” for my project (siege surrenders weren’t nearly as ritualistic as portrayed…) and I need a place to make sure I test that and make the point clearly in the final product.
    • Test: how to test a particular theory (either mine or somebody else’s). Often times I’ll sketch out possible graphics/charts/maps to clarify my understanding of how the ‘system’ I’m studying works, what its dynamics and constraints are, geographical and chronological patterns… Scan that puppy and put it in there.
    • Reasons why I chose a particular case study (and not some other) should go somewhere, but I’m not sure where yet. This spot right here seems good for now.
  • ResearchQuestion: not sure if this is a separate document type (or a file title Research Question with doctype=Hypoth?), but might be worth highlighting. Maybe keep track of the progression of your research question over time: what was the impetus of your project, and how has your research focus changed (and why)? Might also be useful to make a sub-research question for each main section of your project.
  • Future: a document that records your post-project thoughts (or your plans for future versions). I assume I’m not the only one who continues to collect examples supporting my arguments in things already published? Or who thinks “Man, I really wish I had used this juicy quote!”? You can also use this document type for post-publication reflections. Heck, even make a document to keep track of later commentary on your publication/project. It’s your system, use it how you want. A place for everything, and everything in its place.
  • Progress or Log: Consider adding a document type that logs what tasks you perform each session – that’ll give you an idea of how you work and allow you to self-flagellate as needed.

If you’re not sure which type of research journal note a particular document is, leave the Keyword field blank and deal with it later, during the weekly or monthly review.

To speed things up, you could even create a blank document of each type and then duplicate it to each project group.

Research journal document templates

Research journal document templates

You can also use the above doc types in your other DTPO groups. For example, I might use the Method doc type in a particular source’s tag if there are important methodological issues with that particular source. In DTPO you can also link to any other document, so go crazy.

Once you have these document types set up, you can review them as needed to keep your head in the (research) game.

The overall goal? Externalize your thinking about your processing, match your research tasks to the time and energy available, remind yourself where you left off, and provide yourself a trail of breadcrumbs so you can easily figure out how you got to where you are now.

The Strategies Combined

The more decisions you make ahead of time (i.e. when you have time and energy), the easier it will be to follow through when you have less time (or energy). This requires that you “Know yo’self,” which means figuring out where and when you do things particularly well. For example, for some reason I tend to brainstorm well on airplanes. So I have an @Airplane context when I can do things like this:

2.5 hour plane ride. Page 1 of 2.

2.5 hour plane ride. Page 1 of 2.

Putting all these together, the following practices should help with the process:

  • Use context-sensitive, actionable checklists [mitigates the restart penalty accrued when multi-tasking]
    • If you are repeating the same multi-step process on multiple documents, dedicate a single pass for each type of task and go through the documents several times. For example, when grading I first go through each paper looking for proper citation formatting. Then I skim through each paper looking for the types of primary sources. Then I go through each paper looking for… Multiple passes allow you to focus in on one aspect at time, which is easier for me than trying to remember to check paper 1 for 10 different things, then paper 2 for those same 10 different things… You can do the same thing when comparing multiple sources’ coverage of different topics.
  • Summarize the results/findings of your current work session in a sentence or two. Read this summary at the beginning of your next work session.
  • Create Next Action tasks at end of each work session. Start with that task at the beginning of your next work session.
    • Start Next Action tasks with a specific research action verb (Scan, Browse…) and include the explanation of how it fits with your larger project: “Skim France Siege Laws: evacuation ritualized?”
    • Use templates to standardize and speed up entering the tasks
  • Create a research journal system that allows you to easily file and find any type of thought on each project. Keep your research journal notes for each project separate from your original sources, and separate from your notes about those sources (which might apply to multiple projects). Review your research journal whenever you feel like you’re losing your way.
  • Create specific conceptual research tasks for when you need time to think deeply, and schedule generic Research thinking time on your calendar. Examples might include:
    • Read [secondary source] for its coverage of [topic X]
    • Decide how to split [topic] into [different projects] (conference, chapter, article, book…)
    • Relate [ideas] to the historiography
      • How am I different from other scholars?
    • Decide/Refine your research question as simply as possible:
      • From Turabian’s Manual for Writers of Research Papers: “I am working on the topic of _____ because I want to find out _____ so that I can help others understand ____.”
    • Analyze [sources] on [topic X]
    • Outline [chapter or book]
    • Synthesize [new thought (hypothesis, test…)] with [existing project]
    • Test [hypothesis] with [sources]
  • Some slightly-mindless tasks specific to my workflow:
    • Search for key terms (vocabulary, people, events…) in Google
    • Search for key terms in Google Books
    • Search for key vocab in Google Books Ngram Viewer
    • Search for key terms in ECCO
    • Search for in DTPO:
      • Identify relevant events and people, then skim documents on them.
      • Make smart groups for large searches, e.g. find all primary sources with the string “capitul*”, with no tag. This will find all PS records containing the word stem capitul not already identified by me as relating to capitulations.
      • Skim DTPO topic groups on topic [X].
      • Skim similar documents in DTPO with See Also (at document level, and then with selected text).
      • Skim similar groups in DTPO Classify’s suggestions.

Other generic tips include:

  • Use non-linear tools like mind maps, if that helps. These can be imported into DTPO, and some mind mapping programs automatically create a text outline version that you can export.
  • Make a well-defined place to store draft chunks, or at least clearly identify them as chunks, transition sentences…
  • Save webpages and search results to your note-taking place (e.g. DTPO)
  • Subscribe to journal and publisher email alerts to keep abreast of recent publications
  • Set Google Alerts for topics and scholars (and your own name and publications) – add them to your Farley file, if you keep one.
  • Delegate where possible
    • If you’re at a teaching school you probably don’t have TAs, but maybe you have a departmental student worker who might be able to make some copies, or save some search results to a shared Zotero library, or download issues of the London Gazette
  • Scan/photograph microfilm to PDF, because context is most important in this case:
    • Scanned images will be in a @Computer context you are more likely to occupy, i.e. you will rarely be in the basement of the library where the microfilm machines are, nor will you be likely to go there very often, except to view microfilm. Let’s say a volume of archival microfilm has 250 documents. You could spend 4 hours scanning them all to a PDF and import the images into DTPO. From then on, you’ll be able to look up any document anytime you have your computer with you. You can spend 15 minutes between classes working on one or two documents, then go back to the next piece later that day when you have another break. But if, on the other hand, you try to take notes at the microfilm machine, you won’t get through all 250 documents in 4 hours – you’ll probably wear yourself out long before then, and will need to go back to that context to continue your work multiple times. Scanning microfilm may be mindless, but it can only be done in a very constrained context. So eliminate the most limiting constraint first.
  • Decide whether an automation program like Keyboard Maestro or Hazel or Automator might save you time. At the very least, consider whether you might make a context/task to batch perform certain administrative tasks. For me, this might include OCRing and reducing the size of PDFs, or copying new downloaded PDFs to two separate folders on two separate drives.
  • Use checklists wherever possible. Create cheatsheets for common topics you address (“10 Questions to Ask about Logistics”…), maybe create a trigger list of groups to analyze across:
    • Chronology, geography, politics, rank, branch of service, social status, ethnicity/nationality, generational cohort, gender, patronage network, religion, familial/social relationships…

That’s Just Crazy Talk

Will this cockamamie scheme work? Dunno. The Next Action trick has certainly worked with other tasks, but I’ll have to see if “What’s the Next Research Action?” will work as well. I’m already liking the research journal thang – I have dozens of scraps of paper with ‘hypothesis’ or ‘test’ written at the top, and now have a place to put them and find them again.

(The drafting stage is a whole ‘nother topic, possibly meriting its own post at some point. But many others have written far more authoritatively about writing and writer’s block, so I’d recommend people start there. And did someone say Revising? Is our work never done?!?!)

Let me know your strategies and thoughts in the comments. I’d particularly appreciate examples of historians’ research journals.

Tags: ,

3 responses to “How to Produce, part II: Processing your results”

  1. Thiago Dória says :

    Great post!
    I have one very basic doubt: when close reading a historiography source, I assume that one produces much paraphrases, selects some quotes and writes a summary of it. Let’s call of this “notes took from a source”.
    But there are also those notes which are interpretative, maybe presenting doubts about how to understand a couple of claims or criticizing the author’s position. Let’s call them “notes about a souce”. They could even “use” the notes took from that particular source.
    Where is the place of the second type of notes at your system? Does it belong to the the journal? Are they among the notes took from that source?

    • jostwald says :

      You’d obviously need to do what works best for you – possibly try a couple variations. Here’s how I do it in DTPO.
      If I have a specific note X on a source A (primary or secondary) and a thought about that note’s content, I’ll usually just add some [My thoughts about this note] in brackets in the same note X – I don’t want that to be in a separate note Y somewhere else in case I come across note X via a search, and not by going to the original group/tag location. If you’re concerned about your additional text confusing DTPO’s AI, you could make your thought(s) a separate note Z, but then it might get separated from note X, esp. if you’re already using the URL field to link back to a scan/text of the original (but you could, of course, create a different link back to the original note).
      If there’s an issue with a particular source in general (e.g. some background info about an author who wrote a work decades before it was published), I’ll put that info as a separate note in the relevant Source A tag, often in a single source-describing note and mark it off as special, e.g. name it starting with a period so it floats to the top when the name column is sorted. That way if I ever have a question about any particular source, I can go to its Source tag and find everything I recorded about it – a copy of the original, any notes I’ve taken on it, as well as some background info on the source (maybe I downloaded the archive catalog description…). Throw some reviews of a secondary sources in there while you’re at it…
      But if I realize my thought applies across multiple notes or multiple sources, I’ll probably *also* put it as its own note in the Project group/tag (e.g. WoS chapter), maybe with a keyword of “method.” I might record it in the research journal as well.
      You can always find your notes in DTPO by searching. But you should also figure out the different *contexts* in which you might need to reread that thought, and then put them in each context: a stand-alone note in the appropriate thematic group tag (the Spotlight Comment and link will take you back to the original as needed), a stand-alone “method” note in the Project tag for thoughts of broader relevance, and as part of an entry in your research journal if you want to record the context in which you realized that thought. If you have a bunch of examples of the same methodological issue, you could even make a separate document with an index/table of contents of all the documents (DTPO Data-Create Table of Contents with records selected). You could use a consistent text prompt whenever you discuss the issue, e.g. in the body of the note X you type something akin to “method:”. Then either search for that when you want (remembering that DT search doesn’t find non-alphanumeric characters), or save a search (smart group) in the appropriate location.
      So figure out *when/where* you would want to see that thought, and decide how easy it would be to copy it in multiple places. Hope this helps.

      • Thiago Dória says :

        I guess that’s the main point, indeed: figure out when or/and where we want to find that thought. Thank’s!

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: