Evaluating Sources

by Marat She'erah bat Shlomo (Copyright 2002 Monica Cellio)

Once you have decided to undertake a research project, the first step is to determine what you will attempt to learn or prove. This will guide your selection of sources.

The foundations of any research project are the sources you use and how you examine them. This article provides some hints for making best use of your sources.

Types of Sources

You have probably heard much about "primary" and "secondary" sources. Simply put, a primary source is the thing itself; a secondary source is one step removed. An actual garment is a primary source for that type of garment (and weaving, dyeing, and construction techniques for that period). A description of that garment is a secondary source. A portrait is a secondary source for the garment depicted, but a primary source for paintings. (And a description of that painting is a tertiary source for the garment and a secondary source for the painting.) Sketches of the garment are secondary; a photo is generally considered to be "almost as good as primary" in most fields. (Although photos can distort colors and other details.)

Why does it matter? Because not everything a secondary source tells you is true. Even eyewitnesses can be liars, mistaken, poor reporters, biassed, and so on.

Sometimes people confuse "primary" with "period". A 13th-century discussion of a 10th-century item is not a primary source for that item, even though the treatise is a period work. For example, many of the Viking-age sagas were written down centuries after the events they describe; these are secondary sources for those events. A primary source is the actual artifact; sometimes no artifact exists and you must rely on secondary sources.

Sources are not just books. In fact, most primary sources are not books (unless you are researching bookbinding, printing or calligraphy, and related subjects, of course). If you go to a museum and measure a piece of furniture, you have used a primary source (even though the curators won't let you take it home to prove it).

Secondary sources are perfectly acceptable to use; you must simply handle them with some caution. In fact, if you are not already an expert, secondary sources are probably necessary for you to understand the primary sources. Few people can look at a suit of armor behind glass in a museum and know how to make it if they are not armorers, for example, and few modern musicians can play from medieval notation. Secondary sources can help you understand what you are looking at, if they are accurate. That's where critical evaluation comes in: a good secondary source is useful, but a bad one is actively harmful.

Evaluating Sources

Let's start with an example. Consider the following (ficticious) source material:

"And then, he said, the juggler, clad in flowing robes bedecked with bells and ribbons, broke branches off of the trees, put the ends in the fire for a moment, and when they caught fire, juggled a full two dozen of them at once, without dropping any or burning himself. It was truly a spectacle to behold!"

-- "Historia Itinerum Erronis Fatui Auctori Supra Multum Vinum Ad Tabernam Narrata", folio 93b

So, is this a source for juggling torches?

Well, let's look at it. First, we note that it is an account of someone else's description of an event, and we are told nothing about this person. Could he have been mistaken? Was he in a good position to see this? Would he have any reason to exaggerate? Most of these questions must also be asked of our anonymous chronicler.

Next, how does this description compare to things you already know? For example, does green wood burn easily? What is the largest number of awkward, bulky objects you have ever heard of anyone juggling, even without hazards? What about while dressed as described?

Finally, do you know what "Historia Itinerum Erronis Fatui Auctori Supra Multum Vinum Ad Tabernam Narrata" means, and do you trust anything that comes from a book of that name?

Now consider this one:

"To cure the plague: take aged French wine (but never German!) and steep in it fair quantities of wormwood, foxglove, hemlock, and a little cinnamon and ginger for sweatness, and boil it all afternoon. Then give a tankard-full to the patient at Vespers and again at each of the hours, and after three days he will have no further trouble with plague."

-- "Codex Mendaciorum Sancti Galfridi Nugarum Episcopi", Bordeaux, c.1349, page 144

Do you believe the author has first-hand experience to validate this prescription as a cure? Which details are likely to be relevant? (Hint: foxglove and hemlock are poisons; while it is almost certainly true that a patient who consumes this potion will "have no further trouble with plague", it will be because he is dead.) At best, this source is evidence that one person might have believed that this was a cure for plague; this is a far cry from documenting a cure.

Especially when evaluating a secondary source, you should ask yourself a number of questions:

  • How does the author know what he says? What are his sources? Does he have the necessary skills to evaluate those sources, or might he simply be wrong about what he thinks he sees? Did he see the item or event directly, or is he himself relying on someone else's interpretation?

  • Does the author support his assertions, or does he simply state things as fact? If you can't see the evidence, you should expect to see a convincing argument. (An easy-to-spot sign of a bad source is an absence of footnotes. A bibliography at the end is not good enough if the author doesn't provide support for individual statements.)

  • How much of what the author says is based on evidence, and how much is speculation? Is the description of clothing based on an actual garment, or on three threads and a broach? If the latter, does the author explain each step of the reasoning? Do you believe the explanations?

  • What do the other secondary sources dealing with the same subject have to say? Do not rely on a single secondary source; compare and contrast. Even if you have a primary source, it is useful to examine more than one secondary source; they might point out things you did not notice on your own.

  • Is what the author is saying consistent with what you already know about the subject, either from other sources or from your knowledge of the world (or field) in general? If there are contradictory interpretations, what is the evidence for each? Basically, how accurate do you think this source really is? (Beware of modern considerations affecting your assumptions about period practice.)

  • Has any of the work been superseded? For example, later archeological digs might have invalidated what we thought we knew from earlier ones.

  • Has your source been translated from the original language? What has the editor done to validate the translation? Does he offer alternative interpretations of problematic words or phrases? (Aside: music transcription is a form of translation.)

  • What is the author's motivation for writing what he does? Could he have an agenda that would affect his work?

It can be especially useful to evaluate a secondary source against the artifact (or a photo). Does what the secondary source says match what you can see yourself? If it reports the use of brass rivets in the helm, can you see them in the photo? If it says the cloth is dyed brown, do you see anything that suggests stains instead? If you see discrepencies, there might be others that you don't see.

You also need to evaluate your primary sources. Paint and dye can fade over centuries; threads can rot away; notational systems and languages can change. What you see is not necessarily what is really there. For example, it wasn't until I saw an exhibit of books of hours in person that I realized that some of the "black" paint I had seen in photos was actually not paint but tarnished silver leaf.

Do not take from this article the message that all research is doomed; it isn't. But your most important tool, even more important than the quality of your library, is your own analytical skill. Don't believe everything you're told or that you think you see; ask questions. If you can't get answers, you can at least be aware of where the problem spots are.

A Postscript

An old joke concerns three men who are travelling together in the countryside when they see a field with a lone black sheep grazing in it. "Interesting," the first man says, "the sheep in this land are black." The second man responds, "no, in this land at least one sheep is black." Finally the third man speaks: "in this land at least one side of one sheep is black."

Strive to think like the third man, and suspect secondary sources of drawing conclusions like that of the first man.

Acknowledgements

Baron Steffan ap Kennydd provided the Latin translations. Baron Tofi Kerthjalfadsson, Mistress Filipia Capriotti, and Baroness Achren provided valuable feedback on earlier drafts of this article.