[Cache from http://www.cta.dmu.ac.uk/projects/master/gentintr.html 2001-02-10; please use this canonical URL/source if possible.]

MASTER: A gentle introduction

This document is intended to provide experts on manuscript description, who may know little of computers or of computer encoding, with an outline introduction to the proposals of the partners in the European Union funded MASTER project, for Manuscript Access through Standards for Electronic Records (in consort with the Text Encoding Initiative workgroup on manuscript descriptions).

By the end of this document, a reader should:

  • Understand enough about SGML (Standard Generalised Markup Language) encoding to evaluate the MASTER standard
  • Understand the overall architecture of the encoding offered by the MASTER encoding: that is, how it suggests a manuscript description should be structured
  • Understand how the approach offered by this system differs from the database systems characteristic of earlier attempts to define standards for computer-readable manuscript encoding
  • Appreciate the varieties of description which could be made according to these proposals, ranging from summary ‘first level’ records to very lengthy ‘full’ records, with varying levels of markup
  • Know where to find more information about the MASTER encoding, and where to find software to help you make manuscript descriptions conforming to the MASTER standard

History of these proposals

Many scholars, working now over several decades, have made suggestions concerning the making of computer-readable manuscript descriptions, and many different computer systems (almost all using database technology) have been implemented to permit creation and retrieval of these descriptions. One may mention here the initiatives of the IRHT in the 70s and of the Italian Censimento in the early 90s. A convenient summary of several such systems can be found in the volume Bibliographic Access to Medieval and Renaissance Manuscripts: A Survey of Computerised Data Bases and Information Services, ed. Wesley Stevens, Primary Sources and Original Works, Volume1, Numbers 3/4, 1991. It is a characteristic of such systems that their focus was to use computing technology to achieve precision of cataloguing for a particular domain of manuscripts, and not create a generic system of cataloguing suitable for many different kinds of manuscript and manuscript catalogue. In 1991 a meeting of LIBER passed a resolution that a standard should be sought for first-level descriptions, but were unable to find resources to achieve this.

The rise of the web and advances in digital imaging of manuscripts gave a new impetus to this work in the mid-90s. It became clear that an online union catalogue of manuscripts was indeed technically feasible, and that digital imaging could offer access to the manuscripts before in a manner never before imagined. Machine readable manuscript descriptions capable of access in a single online system are vital to realisation of these aims. This could only be achieved by agreement about the descriptions themselves: how they are to be structured and accessed. No such agreement existed, and several projects sought to address this. A key event was the funding, by the Mellon foundation, of two interlocking American projects concerning manuscript descriptions: the Electronic Access to Medieval Manuscripts project (EAMMS) and Digital Scriptorium. Both these projects began in 1996, and brought together American and European experts on computer systems and manuscript cataloguing. The inception of these American projects provoked a group of European scholars to create a European counterpart to these American projects.

A meeting at Studley Priory, near Oxford, in November 1996 planned what became the MASTER project (for: Manuscript Access through Standards for Electronic Records). After a long process of negotiation with the European Union, MASTER began (with funding from the Fourth Framework program) on 1st January 1999. The MASTER funded partners include the Royal Library, The Hague, the National Library of the Czech Republic, and L'Institut de recherche et d'histoire des textes, Paris. Unfunded partners include the Vatican Library, British Library, Bodleian Library, and the Biblioteca Ambrosiana. MASTER is very closely linked to EAMMS, and EAMMS representatives have been present at all MASTER meetings.

The present proposals are based on a different approach to that of the database systems implementations characteristic of the methods developed in the 1980s and described in the Stevens volume referred to above. Firstly, the various database systems developed for describing and distributing manuscript descriptions were just that: various indeed, with multiple problems relating to compatibility of data across systems and the binding of particular databases to specific (and perishable) combinations of software and hardware. Secondly, there is real doubt as to the efficacy of any database system for description of materials as elaborately and multifariously structured as manuscript descriptions. By the time of the inception of the EAMMS and other projects (from 1996 on) an alternative to a database implementation of manuscript descriptions had emerged: the application of Standard Generalised Markup Language (SGML) encoding to the task. Since 1996, the emergence and increased acceptance of Extensible Markup Language (XML) has added further force to this alternative. The work of the Text Encoding Initiative (TEI) since 1988 had created a set of guidelines for the encoding of a very wide range of humanities materials. These guidelines have become widely used in the scholarly community, and were instrumental in the explosion of electronic texts, which is reshaping many areas of scholarly work.

Something like a consensus developed, among the scholars, archivists and computer experts involved in the EAMMS, Digital Scriptorium and MASTER projects. That SGML/XML encoding, rather than database systems, offered the most promising route towards a scheme for computer-readable manuscript descriptions which would add the new digital virtues of searchability and access to the strengths of the traditional, deeply-considered scholarly description. This would not mean discarding of database systems entirely: there are too many records invested in such systems, and some of the systems themselves have such virtue, for this to be desirable or possible. Accordingly, all these projects have a database component, and EAMMS (which aimed from the beginning at both database and SGML/XML approaches) has been particularly successful in devising a MARC-based format for manuscript descriptions. However, at each end of the scale the possibilities of SGML/XML encoding recommend themselves. At one end of the scale: SGML/XML encoding of very simply structured records accompanied by manuscript images would lend itself to immediate web mounting. At the other end: the most complex manuscript descriptions could be housed in SGML/XML encoding, without distortion of the description and with no compromise of scholarly detail. Similarly, the development of a universal interchange format in SGML/XML might permit decanting of the many manuscript records from the many databases into a single, and therefore more malleable, form.

Several further factors contributed to the choice of the TEI implementation of SGML/XML, as the base for the proposals here outlined. Firstly, there was the perceived success of the TEI guidelines in addressing many of the issues, which would have to be resolved in any scheme of computer readable manuscript descriptions. Such a scheme would need to establish encoding for transcriptions of text from the manuscripts: the TEI had already set forward guidelines for transcription of primary textual material, and these guidelines had been well tested. A vocabulary would be needed for encoding of various kinds of textual division, for encoding of metadata about the description, and more: the TEI had already done much of this work. Secondly, the TEI workgroup system offered a well-proven means of bringing together domain experts and encoding experts, and harnessing their various skills towards a consensus which could form the basis of a widely-accepted standard. Thirdly, several of the people active in framing the TEI guidelines had strong interests in medieval manuscripts, and so were involved in the projects named above from their beginning.

The Studley Priory meeting which effectively inaugurated MASTER was followed by meetings of the EAMMS group at Hill Monastic Manuscript Library in December, 1996, and a year later, in November 1997, by a meeting at Columbia University. This brought together many of the participants in the EAMMS, Digital Scriptorium and MASTER projects with other manuscript experts. Following this meeting, a TEI workgroup was established, with Consuelo Dutschke and Ambrogio Piazzoni as co-chairs. This workgroup first met in July 1998. The guidelines here described are the result of this meeting and of a series of meetings involving members of EAMMS, MASTER and the TEI workgroup: for example, in New York in January 1999 (members of the TEI workgroup); in Paris in February 1999 (the MASTER group); in Rome in March 1999 (the TEI workgroup). Around these meetings, vigorous discussions by email among the EAMMS and MASTER communities have helped shape these proposals.

A standard becomes a standard not simply as a result of a pure intellectual effort on the part of those who frame it: it must be accepted and used by the community. For this reason, the proposers of this standard wish to involve as many people as possible in the development of these proposals. Already, the members and associates of the EAMMS, Digital Scriptorium and MASTER groups include some of the most significant manuscript libraries, and some of the most respected manuscript scholars, in North America and Europe. An independent expert group (Dr Ian Doyle, Durham University Library [chair]; Professor Peter Gumbert, Leiden; Dr Gilbert Ouy, Paris) has met twice to consider these proposals, and many of their comments are incorporated in the standard. Through this introductory document, and through the separate full technical documentation, the proposers would like to make possible the broadest possible discussion of the standard.

What is SGML/XML?

This section gives a very brief account of SGML (Standard Generalised Markup Language), focussing on the areas most relevant to these proposals. If you are familiar with SGML already, skip to the next section. This basic description is also valid for Extensible Markup Language, XML.

The basic unit of SGML/XML is the element. An element can be a stream of text surrounded by a tag, as follows:

This is a paragraph

Here, the encoding
 indicates that this is the beginning of a p element, and the encoding
 indicates that this is the end of a p element. The content of the p element is the text ‘This is a paragraph’.

An SGML/XML document could just consist of a series of elements, one after another, as follows:

This is a head element

This is a paragraph

<note>This is a note</note>

One could compare this to a set of database fields: where a database could put this information into fields named ‘head’ ‘p’ ‘note’, SGML/XML encoding puts the information into elements named ,
, <note>.

However, SGML/XML has six features which distinguish it from database systems, and (in the proposes’ opinion) make it especially suitable for encoding information as rich and as various as manuscript descriptions. Firstly, one can add attributes to elements, and give those attributes values. For example, one could specify that a paragraph is centred as follows:

This is a centred paragraph

Or, for a left-justified paragraph:

This is a left-justified paragraph

One can combine different attributes and values. Here, we use an n attribute to indicate that this is the first paragraph, and the justification attribute to indicate that it is left-justified:

The first left-justified paragraph

The ability to add information to encoding by use of attributes and their values, singly and in combination with one another, can permit encoding of very complex information.

The second feature, which suits SGML/XML to our purposes, is that it permits elements to contain other elements (or, to ‘nest’ within one another). Here, we create an outer <chapter> element, which contains a  and a sequence of
 or paragraph elements:

This is the heading for chapter 1

The first paragraph

The second paragraph

Or, instead of <chapter> one could use a generic
 element (marking any kind of textual division) and use a type attribute to indicate that it is a chapter, thus:

The first paragraph

The second paragraph

The third feature distinguishing SGML/XML is that one may use a Document Type Definition (DTD) to insist that a document must contain certain elements, and to specify that these elements must come in a certain order, and to define what these elements must contain.

In the example before, we could use a DTD to force every
 element to have a type attribute, and to force every
 element to contain at least one  element followed by at least one
 element. In the proposals that follow, we use a DTD to enforce a rule that every manuscript description must contain an identifier for the manuscript. Through this mechanism, one can create a DTD to define a standard, and then ensure that documents do indeed conform to the standard by ‘parsing’ them against the DTD.

The fourth feature distinguishing SGML/XML is its use of what are called ‘entity references’. In brief, an ‘entity reference’ is a mechanism to allow substitution of one string by another. For example: one could define an entity called ‘eacute’. Instead of ‘é’, one types the entity reference ‘é’ into an SGML/XML document. The advantage of this is that one can replace the entity reference ‘é’ by whatever computer character represents é on the computer system being used to read the document. Thus, this system can be used to make sure that text is correctly displayed whatever computer system is in use. Indeed, this scheme can be used to cope with different character sets, different writing systems, and any combination of languages, characters and writing systems in a single document.

The fifth feature of SGML/XML stems from its use of a basic character set to describe elements, in combination with the use of entity references to represent characters outside this basic character set. The effect of these is to make SGML/XML documents platform independent: that is, to ensure that SGML/XML documents will appear exactly the same regardless of the computer hardware and software in use. A concrete example of this is the world wide web: the ‘HTML’ (Hypertext Markup Language) used by the web is an instance of SGML/XML, and employs these SGML/XML techniques to ensure that HTML documents look and behave identically whether viewed on a Macintosh or Windows machine, and whether viewed by Internet Explorer or Netscape.

The final feature of SGML which fits it for our use is that it is an international standard. Thus, it is defined and maintained by an International Standards Organisation committee, as ISO 8879: 1986. This definition of SGML as an international standard ensures the stability and consistency of SGML. Because SGML is an international standard and because it is system-independent (as explained above) documents encoded in SGML may be considered as an archival medium. This is obviously attractive in the context of manuscript descriptions: the objects described are often centuries, even millennia, old, and one could expect descriptions of these to be useful for decades, at least. (XML, as distinct from SGML, is not yet an ISO standard).

It is often objected about SGML/XML, that it is very verbose: the ‘text’ of the document disappears in a forest of tags, and it is consequently difficult to make and to read the SGML/XML. There are several answers to this. Firstly, there is efficient editing software available which assists the creation of the SGML/XML by various techniques (macros, automated conversion from optical character read input or from typesetting files, and the like). Secondly, there are well-developed software readers and display systems which permit the reader to see the text without all the SGML/XML encoding, and indeed use the SGML/XML (as it should be used) to create attractive and highly-searchable realisations of the text. The widespread acceptance of SGML/XML encoding in the commercial, government, and academic worlds means that there are a very great many such systems, both as freeware and as commercial applications.

Information on many of these can be found at http://www.oasis-open.org/cover/sgml-xml.html. However, it is true that the open nature of SGML/XML means that considerable adaptation of the software might be needed to tune it for a particular task, such as making and reading of machine-readable manuscript descriptions. Accordingly, several of the MASTER workpackages were devoted to preparing software for implementation of the standard. In particular, the MASTER partners prepared an implementation of the easy-to-use Windows text editor NoteTab permitting efficient making of manuscript records.

Design principles for these proposals

The massive advantage gained by unification of many different sets of data within a single system, permitting the development of efficient common tools for the record maker and efficient retrieval for the record user, makes standard development worthwhile. For this, a standard encoding of the many sets of data is required. Accordingly, a standard system of encoding which permits encoding of one set of data, in one way alone, and which cannot be used for different sets of data from the same intellectual domain, is no standard at all. However, a standard which tries to do all things for all sets of data may end by serving none of them well, and so again is no standard at all.

It is the tension between these extremes which makes standard design difficult. The solution of the TEI, from the beginning, has been to create standards which are ‘broad churches’: which would accommodate within themselves many different kinds of records, from the most simple to the most complex. The case for this is particularly cogent for manuscript descriptions, where each country, each intellectual domain, even each cataloguer for each different cataloguing project, has developed a distinct cataloguing style. We have therefore sought, above all, for flexibility and for perfectibility in these proposals. For flexibility: these proposals permit, on the one hand, the making of simple manuscript inventories which contain no more than a list of manuscript identifiers, to which may be added just a few words of description of each manuscript, or an image of the manuscript. On the other hand, the proposals also aim to enable the most highly-formalised descriptions, with elaborate structural mark-up distinguishing the various elements within the description and containing complete manuscript transcriptions and complete digital facsimiles of the manuscript. The distinction is not just of length. In this model, a lengthy description might be just a manuscript identifier accompanied by a series of unstructured paragraphs (that is: plain paragraph elements with no encoding within them discriminating their content); or a short description might be highly structured. For perfectibility: it is inherent in this design that one might begin by making a simple inventory, and elaborate this by progressive addition of information. Or, one might begin by importing information from an existing manuscript database into unstructured paragraphs, which later workers might reformulate to distinguish statements of date, place, provenance, or description.

Accordingly, these proposals do not offer rigid definitions of what might constitute a 'short' or 'first-level' record, as against a 'long' or 'full' record. Indeed, these proposals mandate one thing, and one thing only: that a manuscript description (<msDescription>) must contain a manuscript identifier (<msIdentifier>). Beyond this, the proposals are not of themselves prescriptive. We suggest that when assessing these proposals you ask, first: do they permit you to make the manuscript descriptions which you would like to make? If the answer is 'yes', then they will serve; if the answer is 'no', the proposers would like to know of the deficiency. We suggest you ask, secondly: do the names of the elements proposed and the relationship of the elements to each other seem transparent, logical and elegant? We have sought overall for a lucid naming and structuring of the descriptive element, so that manuscript experts and users would say, instantly: 'this feels right'. If something seems illogical or misnamed, and you have a concrete suggestion as to how it might be improved, we should like to hear about it.

A design principle which emerged in the often-intense discussions surrounding these proposals concerned their architecture. Should we aim for an architecture which would permit the cataloguer to say almost anything, in almost any way imaginable? Or should we aim for a more formal prescription, which declares: if you want to say something about the binding of the manuscript (for example) you can only say it in a <binding> element within a <physDesc> element. The advantage of the first approach is that it permits the cataloguer near complete freedom; its danger is that the heterogeneity of descriptions might be at the cost of efficient retrieval based on predictable use of agreed descriptors: the prime justification for making the computer-readable records in the first place. The disadvantage of the second approach is that over-rigid formalism would lead to frustration among cataloguers, and (rather quickly) refusal to adopt the standard.

The solution which emerged was this: we would do both. We would allow at various points within the description paragraph (
) elements which would allow the cataloguer to say almost anything they wish, but in a relatively informal and unstructured manner. But we would also provide formal and precisely-defined elements for distinct and identifiable manuscript phenomena. Thus: a cataloguer is free to speak of any aspect of the manuscript binding, as it bears upon the history or intellectual content of the manuscript, within the
 elements provided within the <history> and <msContents> elements. But if you want to make a formally-structured statement about the binding (its material, its date) you must do this within the <binding> element provided within the <physDesc> element. The alternative, to permit the encoder to use the <binding> element at any point, would likely lead to cataloguers feeling obliged to surround every reference to binding with the <binding> tag, regardless of context and content. This would encourage a superfluity of effort which would lead to so many different kinds of information being contained in <binding> as to render the element semantically ill-defined and therefore useless.

The danger of an open standard, such as this is designed to be, is that it may be misused. We do not encourage, for example, manuscript descriptions which consist of no more than an identifier and a lengthy prose description with no formal distinctions through markup of statements of date, origin, provenance, and the like. However, the standard itself cannot be used to prescribe that descriptions must conform to this or that model: it can only be used to enable the various models.

These proposals seek to create a framework which can accommodate what we know of existing standards, and to enable (in time) greater precision and more efficient retrieval of detail in cataloguing. It is the task of various domain experts to recommend how the standard should be applied in their area. The proposes will begin this process, in the documentation which here follows and which will accompany the completed standard, but it will be the responsibility of you, the users of the standard, to further best practice.

Finally, because the background and interests of all the participants in the primary groups involved in preparation of these proposals (EAMMS, Digital Scriptorium, MASTER, and the TEI workgroup) are in western European medieval manuscripts, these proposals have been prepared only with those manuscripts in mind. However, it is intended that these should in time be generalisable, at some level or other, to other manuscripts from different cultures, and we welcome reaction from experts in other manuscript areas.

We propose that a new TEI element be created, <msDescription>, and that each <msDescription> element may contain one, and only one, description of a manuscript. A composite manuscript is regarded as a single manuscript, with the distinct manuscripts which it contains each being held in distinct <msPart> elements, contained within the <msDescription> element.

A single <msDescription> may contain the following elements:

<msIdentifier> : this contains the identifier of the manuscript: where it is, and its exact shelfmark or other identifier. This element is mandatory.

<msHeading>: this can be used to give basic information about the manuscript, such as might appear in summary catalogue (for example: author, title of work, date, place, language)

<msContents> : this contains a description of the intellectual content of the manuscript, either as a series of paragraphs or structured into defined sub-elements

<physDesc> : this contains a description of the physical aspect of the manuscript, structured into defined sub-elements

<history>: this contains a history of the manuscript, structured into defined sub-elements

<additional>: contains additional information related to the manuscript but not part of its formal description (e.g.: bibliography; availability of facsimiles or images; curatorial information)

<msPart> : used for composite manuscripts. It may contain all the elements listed above except for  ( is intended for description of the entire manuscript alone). This element is required when dealing with composite manuscripts

 Here is an example of a rather simple manuscript description:

Oxford, Corpus Christi College, MS 198
Geoffrey Chaucer The Canterbury Tales. c. 1400
Folios 1r-266v. The Canterbury Tales. A274-I290. Defective at beginning and end.
Parchment, trimmed. 33.5 x 22.5 cm. Quires [14, 15, and 28] were disordered in the previous binding. They have been reordered and refoliated, with the old foliation being the uppermost. Two consecutive folios are numbered '64a' and '64'
Written by the scribe identified by Doyle and Parkes as 'Hand d'
Dated c. 1400 (personal communication, Malcolm Parkes). On fol. 146r is the name 'Burle' in drypoint, in the margin next to E1396. Cp came to the College as a bequest of William Fulman, according to a note on fol. 1r : 'Liber C.C.C.Oxon Ex dono Gulielmi Fulman A.M. hujus Collegii quondam socius.'

In a summary catalogue, one could encode it using the MASTER DTD as follows, stating only the place, shelfmark, and basic information as follows:

       <msIdentifier n="1">
<country reg="GB">Great Britain</settlement>
<repository>Corpus Christi College</repository>
<idNo>MS 198</idNo>
<title>The Canterbury Tales</title>
<author>Geoffrey Chaucer</author>
<origDate notBefore="1395" notAfter="1420">c. 1400</origDate>
<textLang langKey="ENM">Middle English</textLang>

Observe the possibilities for precise searches which this encoding allows. We could find this manuscript as dated between 1395 and 1420; as being written in Middle English (with language key "ENM"), and more.

Or, one could encode all the information in the description thus:

       <msIdentifier n="1">
<repository>Corpus Christi College</repository>
<idNo>MS 198</idNo>
<title>The Canterbury Tales</title>
<author>Geoffrey Chaucer</author>
<msItem n="1" defective="yes">
<locus from="1" to="266">Folios 1r-266v</locus>
<title type="uniform">The Canterbury Tales</title>
<note>Defective at beginning and end</note>
<support>
Parchment, trimmed.
</support>
<extent>266.<dimensions type="leaf" scope="all"><height>33.5</height><width>22.5</width></dimensions></extent>
<collation>
Quires [14, 15, and 28] were disordered in the previous binding. They have been reordered and refoliated with the old foliation being the uppermost. Two consecutive folios are numbered '64a' and '64'
</collation>
<msWriting hands="1">
<handDesc scribe="Hand D (Doyle/Parkes)" script="Anglicana" medium="ink" scope="sole">
Written by the scribe identified by Doyle and Parkes as '<name type="person" role="scribe" key="DPhandD">Hand d</name>'
</handDesc>
<origin notBefore="1395" notAfter="1420" certainty="high" evidence="conjecture">
Dated c. <origDate>1400</origDate> (personal communication, Malcolm Parkes).
</origin>
<provenance>
On fol. 146r is the name 'Burle' in drypoint, in the margin next to E1396.
</provenance>
<acquisition>
Cp came to the College as a bequest of William Fulman, according to a note on fol. 1r : <q>'Liber C.C.C. Oxon Ex dono Gulielmi Fulman A.M. hujus Collegii quondam socius.'</q>
</acquisition>

A full account of how these descriptions can be made, with further discussion of the elements within these descriptions, is given in the account of the customized text-editor prepared by the MASTER partners.

Next steps…

The MASTER project and its partners has put together various resources to help make it easier for cataloguers to learn how to make manuscript descriptions according to the encoding we have developed. The resources page of the MASTER site gives a description of these resources. These include:

This page last revised 14 January 2001

Copyright © 1999
Centre for Technology
and the Arts


We gratefully acknowledge support from
the Simons Foundation
and member institutions
Full-text links:

Current browse context:

new | recent | 1702

Change to browse by:

References & Citations

Computer Science > Computers and Society

Title:Big Data for Social Sciences: Measuring patterns of human behavior through large-scale mobile phone data

Authors:Pål Sundsøy
Abstract: Through seven publications this dissertation shows how anonymized mobile phone data can contribute to the social good and provide insights into human behaviour on a large scale. The size of the datasets analysed ranges from 500 million to 300 billion phone records, covering millions of people. The key contributions are two-fold:
1. Big Data for Social Good: Through prediction algorithms the results show how mobile phone data can be useful to predict important socio-economic indicators, such as income, illiteracy and poverty in developing countries. Such knowledge can be used to identify where vulnerable groups in society are, reduce economic shocks and is a critical component for monitoring poverty rates over time. Further, the dissertation demonstrates how mobile phone data can be used to better understand human behaviour during large shocks in society, exemplified by an analysis of data from the terror attack in Norway and a natural disaster on the south-coast in Bangladesh. This work leads to an increased understanding of how information spreads, and how millions of people move around. The intention is to identify displaced people faster, cheaper and more accurately than existing survey-based methods.
2. Big Data for efficient marketing: Finally, the dissertation offers an insight into how anonymised mobile phone data can be used to map out large social networks, covering millions of people, to understand how products spread inside these networks. Results show that by including social patterns and machine learning techniques in a large-scale marketing experiment in Asia, the adoption rate is increased by 13 times compared to the approach used by experienced marketers. A data-driven and scientific approach to marketing, through more tailored campaigns, contributes to less irrelevant offers for the customers, and better cost efficiency for the companies.
Comments: 166 pages, PHD thesis
Cite as: arXiv:1702.08349 [cs.CY]
  (or arXiv:1702.08349v1 [cs.CY] for this version)

Submission history

From: Pål Sundsøy [view email]
[v1] Mon, 27 Feb 2017 16:09:48 UTC (17,284 KB)


Ancestors: What Constitutes Proof?

All genealogists should be asking this question for every single relationship between people in their trees – or at least for every person that they claim as an ancestor. The answer differs a bit when you introduce DNA into the equation, so let’s discuss this topic.

It’s easier to begin by telling you what proof IS NOT, rather than what proof is.

What is Proof, Anyway?

First of all, what exactly do we mean by proof? Proof means proof of a relationship, which has to be proven before you can prove a specific ancestor is yours. It’s a two-step process.

If you’re asking whether those two things are one and the same, the answer is no, they are not. Let me give you a quick example.

You can have proof that you descend from the family of a specific couple, but you may not know which child of that couple you descend from. In one case, my ancestor is listed as an heir, being a grandchild, but the suit doesn’t say which of the man’s children is the parent of my ancestor. So frustrating!

Conversely, you may know that you descend from a specific ancestor, but not which of his multiple wives you descend from.

You may know that your ancestor descends from one of multiple sons of a particular man, but not know which son.

Therefore, proof of a relationship is not necessarily proof that a particular person is your ancestor.

Not Proof of an Ancestor

OK, so what’s NOT proof? Here are a dozen of the most common items – and there are surely more!

  1. Proof is not a DNA match alone. You can match as a result of ancestors on any number of lines, known or unknown.
  2. Proof is not an oral history, no matter how much you want to believe it or who said it. Oral history is a good starting point, not an end point.
  3. Proof is not, not, 1000 times NOT someone else’s tree. A tree should be considered a hint, nothing more.
  4. Proof is not a book without corresponding evidence that can be independently corroborated. Being in print does not make it so, people make mistakes and new information surfaces.
  5. Proof is not a man by the name of Jr., meaning that he is the son of a man by the same name with the suffix of Sr. Sr. often means older and Jr. means younger, but not necessarily related. Yes, this has bitten me.
  6. Proof of a father/son relationship is not two men with the same name in the same location.
  7. Proof is not a Y DNA match, at least not without additional information or evidence, although it’s a great hint!
  8. Proof is not an autosomal DNA match, unless it is an extremely close match and even then you (probably) need additional information. For example, if you have a half-sibling match, you need additional information to determine which parent’s side.
  9. Proof is not an Ancestry Circle, at least not without additional information.
  10. Proof is not similar or even identical ethnicity, or lack thereof.
  11. Proof is not a “DNA Proven” icon, anyplace.
  12. Proof is not a will or other document, at least not alone, and not without evidence that a person by the same name as the child is the RIGHT person.

I learned many of these NOTS or KNOTS as I prefer to call them, because that’s what they tie me in, by ugly experience. I began genealogy before there were proof standards, let alone the GPS (Genealogical Proof Standard). DNA adds yet another dimension to existing paper standards and is an important aspect of the requirement for a “reasonably exhaustive search.” In fact, there is no reason NOT to include DNA and I would suggest that any genealogical search is not complete without including genetic evidence.

Proof Is a Two-Way Street

Using traditional genealogy, genealogists must be able to prove not only that an ancestor had a child by a specific name, but that the person you believe is the child, is indeed the child of that ancestor.

Let me use an example of Daniel, the son of one Philip Jacob Miller in Washington County, Maryland in 1783.

The tax list shows Philip J. Miller, 15 entries from the bottom of the page, shown below. It also shows “Daniel Miller of Philip” 6 entries from the bottom, and it’s our lucky day because the tax list says that Daniel is Philip’s son.

But wait, there’s another Daniel, the bottom entry. If you were to look on the next page, you would also notice that there’s a Philip Miller who does not own any land.

What we have here is:

  • Philip J. Miller, with land
  • Daniel, son of Philip, no land
  • Daniel, no father listed, land
  • Philip, no land

This just got complex. We need to know which Philip is Daniel’s father and which Daniel is which Philip’s son.

Establishing proof requires more than this one resource.

The great news about this tax list is that it tells us how much land Philip J. Miller owned, and utilizing other resources such as deeds and surveys, we can establish which Philip J. Miller owned this land, and that his name was indeed Philip Jacob Miller. This is important because not only is there another Philip, who, by the way, is NOT the son of Philip Jacob Miller (knot #6 above), there is also another Jacob Miller, who is NOT Philip Jacob Miller and who isn’t even related to him on the Miller line, according to the Y DNA of both men’s descendants.

How would we prove that Philip Jacob Miller is the father of Daniel Miller? We’d have to follow both men backward and forward in time, together. We have great clues – land ownership or lack thereof.

In this case, Philip Jacob Miller eventually sells his land. Philip Jacob Miller also has a Bible, which is how we know that there is no son named Philip. Philip Jacob’s son, Daniel leaves with his brother David, also on this tax list, travels to another location before the family is reunited after moving to Kentucky years later, where Philip Jacob Miller dies with a will. All of his heirs sign property deeds during probate, including heirs back in Frederick and Washington County, Maryland. There is enough evidence from multiple sources to tie these various family members from multiple locations conclusively together, providing two way proof.

We must be able to prove that not only did Philip Jacob Miller have a son Daniel, but that a specific Daniel is the son of that particular Philip Jacob Miller. Then, we must repeat that exact step every generation to the present to prove that Philip Jacob Miller is our ancestor.

In other words, we have a chain of progressive evidence that taken together provides conclusive proof that these two men are BELIEVED to be related. What? Believed? Don’t we have proof now?

I say believed, because we still have issues like unknown parentage, by whatever term you wish to call it, NPE (nonpaternal event, nonparental event,) or MP (misattributed parentage,) MPE (misattributed paternal or parental event) or either traditional or undocumented adoptions. Some NPEs weren’t unknown at the time and are results of situations like a child taking a step-parent’s surname – but generations later – having been forgotten or undocumented for descendants, the result is the same. They aren’t related biologically in the way we think they are.

The Big Maybe

At this point, we believe we have the Philips, Philip Jacobs and Daniels sorted correctly relative to my specific line. We know, according to documentation, that Daniel is the son of Philip Jacob, but what if MY ancestor Daniel ISN’T the son of Philip Jacob Miller?

  • What if MY ancestor Daniel just happens to have the name Daniel Miller and lives in the same geography as Philip Jacob Miller, or his actual son Daniel, and I’ve gotten them confused?
  • What if MY ancestor Daniel Miller isn’t actually my ancestor after all, for any number of reasons that happened between when he lived and died (1755-1822) and my birth.

If you think I’m being facetious about this, I’m not. Not long after I wrote the article about my ancestor Daniel Miller, we discovered another Daniel Miller, living in the same location, also descended from the same family as evidenced by BOTH Y and autosomal DNA. In fact, there were 12 Daniel Millers I had to sort through in addition to the second Daniel on the 1783 tax list. Yes, apparently Daniel was a very popular name in the Miller family and yes, there were several male sons of immigrant Johann Michael Muller/Miller who procreated quite successfully.

Enter DNA

If DNA evidence wasn’t already a factor in this equation, it now must come into play.

In order to prove that Philip Jacob Miller is my ancestor, I must prove that I’m actually related to him. Of course, the methodology to do that can be approached in multiple ways – and sometimes MUST be approached using different tools.

Let’s use an example that actually occurred in another line. Two males, Thomas and Marcus Younger, were found together in Halifax County, Virginia, right after the Revolutionary War. They both had moved from Essex County, and they consistently were involved in each other’s lives as long as they both lived. They lived just a couple miles apart, witnessed documents for each other, and until DNA testing it was believed that Marcus was the younger brother of Thomas.

We know that Marcus was not Thomas’s son, because he was not in Thomas’s will, but Marcus and his son John both witnessed Thomas’s will. In that time and place, a family member did not witness a will unless it was a will hastily constructed as a person was dying. Thomas wrote his will 2 years before it was probated.

However, with the advent of DNA testing, we learned that the two men’s descendants did not carry the same Y DNA – not even the same haplogroup – so they do not share a common paternal ancestor.

Needless to say, this really threw a monkey wrench into our neat and tidy family story.

Later, the will of Thomas’s father, Alexander, was discovered, in which Marcus was not listed (not to mention that Alexander died before Marcus was born,) and, Thomas became the guardian of his three sisters.

Eventually, via autosomal DNA, we proved that indeed, Marcus’s descendants are related to Thomas’s descendants as well as other descendants of Thomas’s parents. We have a proven relationship, but not a specifically proven ancestor. In other words, we know that Marcus is related to both Thomas and Alexander, we just don’t know exactly how.

Unfortunately, Marcus only had one son, so we can’t confirm Marcus’s Y DNA through a second line. We also have some wives missing from the equation, so there is a possibility that either Marcus’s wife, or his unknown biological father’s family was otherwise related to Alexander’s line.

So, here’s the bottom line – we believe, based on various pieces of compelling but not conclusive evidence that Marcus is the illegitimate child of one of Thomas’s unmarried sisters, who died, which is why Marcus is clearly close to Thomas, shares the same surname, but not the Y DNA. In fact, it’s likely that Marcus was raised in Thomas’s household.

  • It’s entirely possible that if I incorrectly listed Thomas as Marcus’s father on Ancestry, as many have, that I would be placed in a Thomas circle, because Ancestry forms circles if your autosomal DNA matches and you show a common ancestor in your trees. This is why inclusion in a circle doesn’t genetically confirm an ancestor without additional information. It confirms a genetic relationship, but not how a person is related.
  • It’s entirely possible that even though Marcus’s Y DNA doesn’t match the proven Y DNA of Thomas, that Marcus is still closely related to Thomas – such as Marcus’s uncle. That’s why Marcus’s descendants match both Thomas’s and Alexander’s descendants through autosomal testing. However, without Y DNA testing, we would never know that they don’t share a paternal line.
  • It’s entirely possible that if Marcus was supposed, on paper, to be Thomas’s child, but was fathered by another man, such as his wife’s first husband, I would still be in the circle attributed to both Thomas and his wife, by virtue of the fact that I match DNA of Thomas’s descendants through Thomas’s wife. This is your classic step-father situation.

Paper is Not Proof

As genealogists, we became so used to paper documentation constituting proof that it’s a blow when that paper proves to be irrelevant, especially when we’ve hung our genealogical hat on that “proof” for years, sometimes decades.

The perfect example is an adoption. Today, most adoptions are through a court of law, but in the past, a functional adoption happened when someone, for whatever reason, took another child to raise.

The history of that “adoption” although not secret when it happened, became lost in time, and the child is believed to be the child of the couple who raised them. The adoption can actually be a step-parent situation, and the child may carry the step-father’s surname but his own father’s Y DNA, or it can be a situation where a relative or unrelated couple raised the child for some unknown reason.

Today, all paper genealogy needs to be corroborated by DNA evidence.

DNA evidence can be some combination of:

  • Y DNA
  • Autosomal DNA
  • Mitochondrial DNA

How Much Proof is Enough?

One of my favorite saying is “you don’t know what you don’t know.”

People often ask:

  1. If they match someone autosomally who shares the same ancestor, do they really need to prove that line through Y or mitochondrial DNA?
  2. Do they really need to match multiple people?
  3. Do they really need to compare segments?

The answers to these is a resounding, “it depends.”

It depends on the circumstances, the length of time back to the common ancestor, and how comfortable you are not knowing.

Relative to question 1 about autosomal plus Y DNA, think about Marcus Younger.  Without the Y DNA, we would have no idea that his descendant’s Y DNA didn’t match the Thomas Younger line. Suddenly, Marcus not being included in either Thomas nor Alexander’s will makes sense.

Relative to question 2 about matching multiple people, the first cousin we tested to determine whether it was me or my brother that was not the child of our father turned out to have different Y DNA than expected. Thank goodness we tested multiple people, including autosomal when it became available.

Relative to question 3 about comparing segments, every matching segment has its own unique history. I’ve encountered several situations where I match someone on one segment from one ancestor, and another segment from an entirely different line. The only way to determine this is by comparing and triangulating individual segments.

I’ve been bitten so many times by thinking I knew something that turned out to be incorrect that I want every single proof point that I can obtain to eliminate the possibility of error – especially multiple kinds of DNA proof. There are some things that ONLY DNA can reveal.

I want:

  • Traditional documentary evidence for every generation to establish the actual paper trail that proves that the child descends from the proper parents.
  • Y DNA to prove the son is the son of the father and to learn about the deeper family history. For example, my Lentz line descends from the Yamnaya culture, something I would never have known without the Big Y DNA test.
  • Mitochondrial DNA to prove that the mother is the actual mother of the child, if possible, not an unknown earlier or later wife, and to learn about the deeper family history. Elizabeth Mehlheimer’s mitochondrial DNA is Scandinavian – before her ancestors are found in Germany.
  • Autosomal DNA to prove that the paper lineage connecting me to the ancestor is correct and the line is not disrupted by a previously unknown adoption of some description.

I attempt to gather the Y and mitochondrial DNA haplogroup of every ancestor in my direct line if possible and confirm using autosomal DNA.

Yes, my personal proof standard is tough, but I suggest that you at least ask these questions when you evaluate documentation or see someone claim that they are “DNA proven” to an ancestor. What, exactly, does that mean and what do they believe constitutes proof? Do they have that proof, and are they willing to share it with you?

Genealogical Proofs Table

The example table below is designed to be used to document the sources of proof that the individual listed under the name column is in fact the child of the father and mother shown. Proofs may vary and could be personal knowledge (someone you knew within your lifetime), a Bible, a will, a deed, an obituary, death certificate, a church baptismal document, a pension application, census records, etc. DNA confirmation is needed in addition to paper documentation. The two types of proof go hand in hand.

Name Birth Death Spouse Father Mother Proofs – Sources DNA Confirmed
William Sterling Estes Oct. 1, 1902, Claiborne Co., TN Aug. 27, 1963, Jay Co., IN Barbara Ferverda William George Estes 1873-1971 Ollie Bolton 1874-1955 Personal knowledge – William is my father and William George is my grandfather. Autosomal triangulated to multiple Estes cousins
William George Estes March 30, 1873, Claiborne Co., TN Nov. 29, 1971, Harlan Co., KY 1. Ollie Bolton

2.  Joyce Hatfield

3. Crocia Brewer

Lazarus Estes 1845-1918 Elizabeth Vannoy 1846-1918 1.  Will of Lazarus Estes Claiborne Co., Tn. Will Book 8, page 42

2.  Deed where Lazarus states William George is his son.  Claiborne Co., Deed Book M2, page 371.

3. My father’s personal knowledge and birth certificate

Autosomal triangulated to multiple descendants of both Lazarus Estes and Elizabeth Vannoy.
Lazarus Estes May 1845, Claiborne Co., TN 1916-1918, Claiborne Co., TN Elizabeth Vannoy John Y. Estes 1818-1895 Rutha Dodson 1820-1903 1. Personal knowledge of George Estes, now decd

2.  Deed here John Y. deeds all his possessions to his eldest son, Lazarus when he goes to Texas, Claiborne Co., Deed book B1, page 37.

Y DNA confirmed to haplotype of Abraham Estes, autosomal triangulated to descendants of Lazarus and Elizabeth and upstream ancestors through multiple matches on both sides.
John Y. Estes December 29, 1818, Halifax Co., VA Sept. 19, 1895, Montague Co., TX Rutha Dodson John R. Estes 1785/88-1885 Nancy Ann Moore c 1785-1860/1870 1. Family visits of his children in Tennessee

2. Census records, 1850, 1860, Claiborne Co., Tn. shows families in same household

Y DNA confirmed through multiple sons. Autosomal triangulates to several descendants through multiple lines of other children.
John R. Estes 1785-1788, Halifax Co. VA May 1885, Claiborne Co., TN Nancy Ann Moore George Estes 1763-1869 Mary Younger bef 1775-1820/1830 1. Halifax County 1812 personal property tax list where John R. Estes is listed as the son of George Estes and lives next to him.  Only 1 George in the county. Later chancery suit lists John R.’s wife’s name and location in Tennessee Y DNA confirmed through multiple lines.  Autosomal confirmed triangulation of multiple lines of his children and his ancestors on both sides.

If you’d like to read more about the difference between evidence and proof, and how to get from evidence to proof, check out this article, What is proof of family history? by my cousin, retired attorney, Robin Rankin Willis.

Proof is a Pain!

So now that we’ve discussed what proof is not, and what types of records constitute proof, you may be thinking to yourself that proof is a pain in the behind. Indeed, it is, but without sufficient proof, you may literally be doing someone else’s genealogy or the genealogy of an ancestor that’s not your own. Trust me, that’s infinitely more painful.

I hate sawing branches off of my own tree. If I have to do it, the sooner I make the discovery and get it over with, the better.

Been there, done that, and really, I don’t want the t-shirt.

There is never such a thing as “too much” proof, but there is certainly too little. We are fortunate to live in a time when not only are historical records available, but the record passed by our ancestors inside our very cells tells their story. Use every tool and every type of DNA at your disposal! Otherwise, you get the t-shirt:)

Standard Disclosure

Affiliate links are limited to:

Concepts – The Faces of Endogamy

Recently, while checking Facebook, I saw this posting from my friend who researches in the same Native admixed group of families in North Carolina and Virginia that I do. Researchers have been trying for years to sort through these interrelated families. As I read Justin’s post, I realized, this is a great example of endogamy and often how it presents itself to genealogists.

I match a lot of people from the Indian Woods [Bertie County, NC] area via DNA, with names like Bunch, Butler, Mitchell, Bazemore, Castellow, and, of course, Collins. While it’s hard to narrow in on which family these matching segments come from, I can find ‘neighborhoods’ that fit the bill genetically. This [census entry] is from near Quitsna in 1860. You see Bunch, Collins, Castellow, Carter, and Mitchell in neighboring households.

Which begs the question, what is endogamy, do you have it and how can you tell?

Endogamy is the practice or custom or marrying within a specific group, population, geography or tribe.

Examples that come to mind are Ashkenazi Jews, Native Americans (before European and African admixture), Amish, Acadians and Mennonite communities.

Some groups marry within their own ranks due to religious practices. Jewish, Amish and Mennonite would fall under this umbrella. Some intermarry due to cultural practices, such as Acadians, although their endogamy could also partly be attributed to their staunch Catholic beliefs in a primarily non-Catholic region. Some people practice endogamy due to lack of other eligible partners such as Native Americans before contact with Europeans and Africans.  People who live on  islands or in villages whose populations were restricted geographically are prime candidates for endogamy.

In the case of Justin’s group of families who were probably admixed with Native, European and African ancestors, they intermarried because there were socially no other reasonable local options. In Virginia during that timeframe, mixed race marriages were illegal. Not only that, but you married who lived close by and who you knew – in essence the neighbors who were also your relatives.

Endogamy and Genetic Genealogy

In some cases, endogamy is good news for the genealogist. For example, if you’re working with Acadian records and know which Catholic church your ancestors attended. Assuming those church records still exist, you’re practically guaranteed that you’ll find the entire family because Acadians nearly always married within the Acadian community, and the entire Acadian community was Catholic. Catholics kept wonderful records. Even when the Acadians married a Native person, the Native spouse is almost always baptized and recorded with a non-Native name in the Catholic church records, which paved the way for a Catholic marriage.

In other cases, such as Justin’s admixed group, the Brethren who notoriously kept no church records or the Jewish people whose records were largely destroyed during the Holocaust, endogamy has the opposite effect – meaning that actual records are often beyond the reach of genealogists – but the DNA is not.

It’s in cases like this that people reach for DNA to help them find their families and connections.

What Does Endogamy Look Like?

If you know nothing about your heritage, how would you know whether you are endogamous or not? What does it look like? How do you recognize it?

The answer is…it depends. Unfortunately, there’s no endogamy button that lights up on your DNA results, but there are a range of substantial clues.  Let’s divide up the question into pieces that make sense and look at a variety of useful tools.

Full or Part?

First of all, fully and partly endogamous ancestry, and endogamy from different sources, has different signs and symptoms, so to speak.

A fully endogamous person, depending on their endogamy group, may have either strikingly more than average autosomal DNA matches, or very few.

Another factor will be geography, where you live, which serves to rule out some groups entirely. If you live in Australia, your ancestors may be European but they aren’t going to be Native American.

How many people in your endogamous group that have DNA tested is another factor that weighs very heavily in terms of what endogamy looks like, as is the age of the group. The older the group, generally the more descendants available to test although that’s not always the case. For example warfare, cultural genocide and disease wiped out many or most of the Native population in the United States, especially east of the Mississippi and particularly in the easternmost seaboard regions.

Because of the genocide perpetrated upon the Jewish people, followed by the scattering of survivors, Jewish descendants are inclined to test to find family connections. Jewish surnames may have been changed or not adopted in some cases until late, in the 1800s, and finding family after displacement was impossible in the 1940s for those who survived.

Let’s look at autosomal DNA matches for fully and partly endogamous individuals.

Jewish people, in particular Ashkenazi, generally have roughly three times as many matches as non-endogamous individuals.

Conversely, because very few Native people have tested, Native testers, especially non-admixed Native individuals, may have very few matches.

It’s ironic that my mother, the last person listed, with two endogamous lines, still has fewer matches than I do, the first person listed.  This is because my father has deep colonial roots with lots of descendants to test, and my mother has recent immigration in her family line – even though a quarter of her ancestry is endogamous.

To determine whether we are looking at endogamy, sometimes we need to look for other clues.

There are lots of ways to discover additional clues.

Is there a trend among the surnames of your matches?

At the top of your Family Finder match page your three most common surnames are displayed.

A fully endogamous Jewish individual’s most common surnames are shown above. If you see Cohen among your most common surnames, you are probably Jewish, given that the Kohanim have special religious responsibilities within the Jewish faith.

Of course, especially with autosomal DNA, the person’s current surname may not be indicative, but there tends to be a discernable pattern with someone who is highly endogamous. When someone who is fully endogamous, such as the Jewish population, intermarries with other Jewish people, the surnames will likely still be recognizably Jewish.

Our Jewish individual’s first matching page, meaning his closest matches, includes the following surnames:

  • Cohen
  • Levi
  • Bernstein
  • Kohn
  • Goldstein

The Sioux individual only has 137 matches, but his first page of matches includes the following surnames:

  • Sunbear
  • Deer With Horns
  • Eagleman
  • Yelloweyes
  • Long Turkey
  • Fire
  • Bad Wound
  • Growing Thunder

These surnames are very suggestive of Native American ancestry in a tribe that did not adopt European surnames early in their history. In other words, not east of the Mississippi.

At Family Tree DNA, every person has the opportunity to list their family surnames and locations, so don’t just look at the tester’s surname, but at their family surnames and locations too. The Ancestral Surname column is located to the far right on the Family Finder matches page. If you can’t see all of the surnames, click on the person’s profile picture to see their entire profile and all of the surnames they have listed.

Please note that you can click to enlarge all graphics.

If you haven’t listed your family surnames, now would be a good time. You can do this by clicking on the orange “Manage Personal Information” link near your profile picture on the left of your personal page.

The orange link takes you to the account settings page. Click on the Genealogy tab, then on surnames. Be sure to click the orange “save” when you are finished.

Partial Endogamy

Let’s take a look at a case study of someone who is partially endogamous, meaning that they have endogamous lines, but aren’t fully endogamous. My mother, who is the partially endogamous individual with 1231 matches is a good example.

Mother is a conglomeration of immigrants. Her 8 great-grandparents break down as follows:

In mother’s case, a few different forces are working against each other. Let’s take a look.

The case of recent immigration from the Netherlands, in the 1850s, would serve to reduce mother’s matches because there has been little time in the US for descendants to accrue and test. Because people in the Netherlands tend to be very reluctant about DNA testing, very few have tested, also having the effect of reducing her number of matches.

Mother’s Dutch ancestors were Mennonites, an endogamous group within the Netherlands, which would further reduce her possibilities of having matches on these lines since she would be less likely to match the general population and more likely to match individuals within the endogamous group. If people from the Mennonite group tested, she would likely match many within that group. In other words, for her to find Dutch matches, people descended from the endogamous Dutch Mennonite population would need to test. At Family Tree DNA, there is a Low Mennonite Y DNA and Anabaptist autosomal DNA project both, but these groups tend to attract the Mennonites that migrated to Russia and Poland, not the group that stayed in the Netherlands. Another issue, at least in mother’s case, is that her Mennonite relatives “seem” to have been later converts, not part of the original Mennonite group – although it’s difficult to tell for sure in the records that exist.

Mother’s Kirsch and Drechsel ancestors were also recent immigrants in the 1850s, from Germany, with very few descendants in the US today. The villages from where her Kirsch ancestors immigrated, based on the church records, did tend to be rather endogamous.  However, that endogamy would only have reached back about 200 years, as far as the 30 Years’ War when that region was almost entirely, if not entirely, depopulated. So while there was recent endogamy, there (probably) wasn’t deep endogamy. Of course, it would require someone from those villages to test so mother could have matches before endogamy can relevant. DNA testing is not popular in Germany either.

Because of recent immigration, altogether one half of mother’s heritage would reduce her number of matches significantly. Recent immigrants simply have fewer descendants to test.

On the other hand, mother’s English line has been in the US for a long time, some since the Mayflower, so she could expect many matches from that line, although they are not endogamous. If you’re thinking to yourself that deep colonial ancestry can sometime mimic endogamy in terms of lots of matches, you’re right – but still not nearly to the level of a fully endogamous Jewish person.

Mother’s Acadian line has been settled in North America in Nova Scotia since the early 1600s, marrying within their own community, mixing with the Native people and then scattering in different directions after 1755 when they were forcibly removed. Acadians, however, tended to remain in their cultural groups, even after relocation. Many Acadian descendants DNA test and all Acadians descend from a limited and relatively well documented original population. That level of documentation is very unusual for endogamous groups. Acadian surnames are well known and are French. The best Acadian genealogical resource in is Karen Theriot’s comprehensive tree on Rootsweb in combination with the Mothers of Acadia DNA project at Family Tree DNA. I wish there was a similar Fathers of Acadia project.

Mother’s Brethren line is much less well documented due to a lack of church records. The Brethren community immigrated in the early 1700s from primarily Switzerland and Germany, was initially relatively small, lived in clusters in specific areas, traveled together and did not marry outside the Brethren faith. Therefore, Brethren heritage and names also tend to be rather specific, but not as recognizable as Acadian names. After all, the Brethren were German/Swiss and in mother’s case, she also has another 1/4th of her heritage that are recently immigrated Germans – so differentiating one German group from the other can be tricky. The only way to tell Brethren matches from other German matches is that the Brethren also tend to match each other.

In Common With

If you notice a group of similar appearing surnames, use the ICW (in common with) tool at Family Tree DNA to see who you match in common with those individuals. If you find that you match a whole group of people with similar surnames or geography, contact your matches and ask if they know any of the other matches and how they might be related. I always recommend beginning with your closest matches because your common ancestor is likely to be closer in time than people who match you more distantly.

In the ICW match example below, all of the matches who do show ancestral surnames include Acadian surnames and/or locations.

Acadians, of course, became Cajuns in Louisiana where one group settled after their displacement in Nova Scotia. The bolded surnames match surnames on the tester’s surname list.

The ICW tools work particular well if you know of or can identify one person who matches you within a group, or simply on one side of your family.

Don Worth’s Autosomal DNA Segment Analyzer is an excellent tool to genetically group your matches by chromosome. It’s then easy to use the chromosome browser at Family Tree DNA to see which of these people match you on the same segments. These tools work wonderfully together.

The group above is an Acadian match group. By hovering over the match names, you can see their ancestral surnames which make the Acadian connection immediately evident.

The Matrix

In addition to seeing the people you match in common with your matches by utilizing the ICW tool at Family Tree DNA, you can also utilize the Matrix tool to see if your matches also match each other. While this isn’t the same as triangulation, because it doesn’t tell you if they match each other on the same exact segment, it’s a wonderful tool, because in the absence of cooperation or communication from your matches to determine triangulation between multiple people, the Matrix is a very good secondary approach and often predicts triangulation accurately.

In the Matrix, above, the blue boxes indicates that these individuals (from your match list) also match each other.

For additional information on various autosomal tools available for your use, click here to read the article, Nine Autosomal Tools at Family Tree DNA.

Everyone who takes the Family Finder test also receives their ethnicity estimates on the MyOrigins tab.

In the case of our Jewish friend, above, his MyOrigins map clearly shows his endogamous heritage. He does have some Middle Eastern region admixture, but I’ve seen Ashkenazi Jewish results that are 100% Ashkenazi Jewish.

The same situation exists with our Sioux individual, above. Heavily Native, removing any doubt about his ancestry.

However, mother’s European admixture blends her MyOrigins results into a colorful but unhelpful European map, at least in terms of determining whether she is endogamous or has endogamous lines.

European endogamous admixture, except for Jewish heritage, tends to not be remarkable enough to stand out as anything except European heritage utilizing ethnicity tools. In addition, keep in mind that DNA testing in France for genealogy is illegal, so often there is a distinct absence in that region that is a function of the lack of testing candidates. Acadians may not show up as French.

Ethnicity testing tends to be excellent at determining majority ethnicity, and determining differences between continental level ethnicity, but less helpful otherwise. In terms of endogamy, Jewish and Native American tend to be the two largest endogamous groups that are revealed by ethnicity testing – and for that purpose, ethnicity testing is wonderful.

Y and Mitochondrial DNA and Endogamy

Autosomal tools aren’t the only tools available to the genetic genealogist. In fact, if someone is 100% endogamous, or even half endogamous, chances are very good that either the Y DNA for males on the direct paternal line, or the mitochondrial DNA for males and females on the direct matrilineal line will be very informative.

On the pedigree chart above, the blue squares represent the Y DNA that the father contributes to only his sons and the red circles represent the mitochondrial DNA (mtDNA) that mothers contribute to both genders of their children, but is only passed on by the females.

By utilizing Y and mtDNA testing, you can obtain a direct periscope view back in time many generations, because the Y and mitochondrial DNA is preserved intact, except for an occasional mutation. Unlike autosomal DNA, the DNA of the other parent is not admixed with the Y or mitochondrial DNA. Therefore, the DNA that you’re looking at is the DNA of your ancestors, generations back in time, as opposed to autosomal DNA which can only reliably reach back 5 or 6 generations in terms of ethnicity because it gets halved in every generation and mixed with the DNA of the other parent.

With autosomal DNA, we can see THAT it exists, but not who it came from.  With Y and mtDNA DNA, we know exactly who in your tree that specific DNA came from

We do depend on occasional Y and mtDNA mutations to allow our lines to accrue enough mutations to differentiate us from others who aren’t related, but those mutations accrue very slowly over hundreds to thousands of years.

Our “clans,” over time, are defined by haplogroups and both our individual matches and our haplogroup or clan designation can be very useful. Your haplogroup will indicate whether you are European, Jewish, Asian, Native American or African on the Y and/or mtDNA line.

In cases of endogamous groups where the members are known to marry only within the group, Y and mtDNA can be especially helpful in identifying potential families of origin.  This is evident in the Mothers of Acadia DNA project as well a particular brick wall I’m working on in mother’s Brethren line. Success, of course, hinges on members of that population testing their Y or mtDNA and being available for comparison.

Always test your Y (males only) and mitochondrial DNA (males and females.) You don’t know what you don’t know, and sometimes those lines may just hold the key you’re looking for. It would be a shame to neglect the test with the answer, or at least a reasonably good hint! Stories of people discovering their ethnic heritage, at least for that line, by taking a Y or mtDNA test are legendary.

Jewish Y and Mitochondrial DNA

Fortunately, for genetic genealogists, Jewish people carry specific sub-haplogroups that are readily identified as Jewish, although carrying these subgroups don’t always mean you’re Jewish. “Jewish” is a religion as well as a culture that has been in existence as an endogamous group long enough in isolation in the diaspora areas to develop specific mutations that identify group members. Furthermore, the Jewish people originated in the Near East and are therefore relatively easy, relative to Y and mtDNA, to differentiate from the people native to the regions outside of the Near East where groups of Jewish people settled.

The first place to look for hints of your heritage is your main page at Family Tree DNA. First, note your haplogroups and any badges you may have in the upper right hand corner of your results page.

In this man’s case, the Cohen badge is this man’s first clue that he matches or closely matches the known DNA signature for Jewish Cohen men.

Both Y DNA and mitochondrial DNA results have multiple tabs that hold important information.

Two tabs, Haplogroup Origins and Ancestral Origins are especially important for participants to review.

The Haplogroup Origins tab shows a combination of academic research results identifying your haplogroup with locations, as well as some Ancestral Origins mixed in.

A Jewish Y DNA Haplogroup Origins page is shown above.

The Ancestral Origins page, below, reflects the location where your matches SAY their most distant direct matrilineal (for mtDNA) or patrilineal (for Y DNA) ancestors were found. Clearly, this information can be open to incorrect interpretation, and sometimes is. For example, people often don’t understand that “most distant maternal ancestor” means the direct line female on your mother’s mother’s mother’s side.  However, you’re not looking at any one entry. You are looking instead for trends.

The Ancestral Origins page for a Jewish man’s Y DNA is shown above.

The Haplogroup Origins page for Jewish mitochondrial DNA, below, looks much the same, with lots of Ashkenazi entries.

The mitochindrial Ancestral Origins results, below, generally become more granular and specific with the higher test levels. That’s because the more general results get weeded out a higher levels. Your closest matches at the highest level of testing are the most relevant to you, although sometimes people who tested at lower levels would be relevant, if they upgraded their tests.

Native American Y and Mitochondrial DNA

Native Americans, like Jewish people, are very fortunate in that they carry very specific sub-haplogroups for Y and mitochondrial DNA. The Native people had a very limited number of founders in the Americas when they originally arrived, between roughly 10,000 and 25,000 years ago, depending on which model you prefer to use. Descendants had no choice but to intermarry with each other for thousands of years before European and African contact brought new genes to the Native people.

Fortunately, because Y and mtDNA don’t mix with the other parents’ DNA, no matter how admixed the individual today, testers’ Y and mtDNA still shows exactly the origins of that lineage.

Native American Y DNA shows up as such on the Haplogroup Origins and Ancestral Origins tabs, as illustrated below.

The haplogroup assigned is shown along with a designation as Native on the Haplogroup Origins and Ancestral Origins pages. The haplogroup is assigned through DNA testing, but the Native designation and location is entered by the tester. Do be aware that some people record the fact that their “mother’s side” or “father’s side” is reported to have a Native ancestor, which is not (necessarily) the same as the matrilineal or patrilineal line. Their “mother’s side” and “father’s side” can have any number of both male and female ancestors.

If the tester’s haplogroup comes back as non-Native, the erroneous Native designation shows up in their matches Ancestral Origins page as “Native,” because that is what the tester initially entered.  I wrote about this situation here, but there isn’t much that can be done about this unless the tester either realizes their error or thinks to go back and change their designation from Native American when they realize the DNA does not support the family story, at least not on this particular line line. Erroneous labeling applies to both Y and mtDNA.

Native Y DNA falls within a subset of haplogroups C and Q. However, most subgroups of C and Q are NOT Native, but are European or Asian or in one case, a subgroup of haplogroup Q is Jewish. This does NOT means that the Jewish people and the Native people are related within many thousands of years. It means they had a common ancestor in Asia thousands of years ago that gave birth to both groups. In essence, one group of the original Q moved east and eventually into the Americas, and one moved west, winding up in Europe. Today, mutations (SNPs) have accrued to each group that very successfully differentiate them from one another. In order to determine whether your branch of C or Q is Native, you must take additional SNP tests which further identify your haplogroup – meaning which branch of haplogroup C or Q that you belong to.

Native Americans Y-DNA, to date, must fall into a subset of haplogroup C-P39, a subgroup of C-M217 or Q-M3, Q-M971/Z780 or possibly Q-B143 (ancient Saqquq in Greenland), according to The study of human Y chromosome variation through ancient DNA. Each of these branches also has sub-branches except for Q-B143 which may be extinct. This isn’t to say additional haplogroups or sub-haplogroups won’t be discovered in the future. In fact, haplogroup O is a very good candidate, but enough evidence doesn’t yet exist today to definitively state that haplogroup O is also Native.

STR marker testing, meaning panels of markers from 12-111, provides all participants with a major haplogroup estimate, such as C or Q. However, to confirm the Y DNA haplogroup subgroup further down the tree, one must take additional SNP testing. I wrote an article about the differences between STR markers and SNPs, if you’d like to read it, here and why you might want to SNP test, here.

Testers can purchase individual SNPs, such as the proven Native SNPs, which will prove or disprove Native ancestry, a panel of SNPs which have been combined to be cost efficient (for most haplogroups), or the Big Y test which scans the entire Y chromosome and provides additional matching.

When financially possible, the Big Y is always recommended. The Big Y results for the Sioux man showed 61 previously unknown SNPs. The Big Y test is a test of discovery, and is how we learn about new branches of the Y haplotree. You can see the most current version of the haplogroup C and Q trees on your Family Tree DNA results page or on the ISOGG tree.

Native mitochondrial DNA can be determined by full sequence testing the mitochondrial DNA. The mtPlus test only tests a smaller subset of the mtDNA and assigns a base haplogroup such as A. To confirm Native ancestry, one needs to take the full sequence mitochondrial test to obtain their full haplogroup designation which can only be determined by testing the full mitochondrial sequence.

Native mitochondrial haplogroups fall into base haplogroups A, B, C, D, X and M, with F as a possibility. The most recent paper on Native Mitochondrial DNA Discoveries can be found here and a site containing all known Native American mitochondrial DNA haplogroups is here.

Not Native or Jewish

Unfortunately, other endogamous groups aren’t as fortunate as Jewish and Native people, because they don’t have haplogroups or subgroups associated with their endogamy group. However, that doesn’t mean there aren’t a few other tools that can be useful.

Don’t forget about your Matches Maps. While your haplogroup may not be specific enough to identify your heritage, your matches may hold clues. Each individual tester is encouraged to enter the identity of their most distant ancestor in both their Y (if male) and mtDNA lines. Additionally, on the bottom of the Matches Map, testers can enter the location where that most distant ancestor is found. If you haven’t done that yet, this is a good time to do that too!

When looking at your Matches Map, clusters and distribution of your matches most distant ancestor locations are important.

This person’s matches, above, suggest that they might look at the history of Nova Scotia and French immigrants – and the history of Nova Scotia is synonymous with the Acadians but the waterway distribution can also signal French, but not Acadian. Native people are also associated with Nova Scotia and river travel. The person’s haplogroup would add to this story and focus on or eliminate some options.

This second example above, suggests the person look to the history of Norway and Sweden, although their ancestor, indicated by the white balloon, is from Germany. If the tester’s genealogy is stuck in the US, this grouping could be a significant clue relative to either recent or deeper history. Do they live in a region where Scandinavian people settled? What history connects the region where the ancestor is found with Scandinavia?

This third example, above, strongly suggests Acadian, given the matches restricted to Nova Scotia, and, as it turns out, this individual does have strong Acadian heritage. Again, their haplogroup is additionally informative and points directly to the European or Native side of the Acadian heritage for this particular line.

In Summary

Sometimes endogamy is up front and in your face, evident from the minute your DNA results are returned. Other times, endogamous lines in ethnically mixed individuals reveal themselves more subtly, like with my friend Justin. Fortunately, the different types of DNA tests and the different tools at our disposal each contain the potential for a different puzzle piece to be revealed. Many times, our DNA results need to be interpreted with some amount of historical context to reveal the story of our ancestors.

When I first discovered that my mother’s line was Acadian, my newly found cousin said to me, “If you’re related to one Acadian, you’re related to all Acadians.” He wasn’t kidding. For that very reason, endogamous genetic genealogy is tricky at best and frustrating at worst.

When possible, Y and mtDNA is the most definitive answer, because the centuries or millennia or intermarriage don’t affect Y and mtDNA. If you are Jewish or Native on the appropriate lines for testing, Y and mtDNA is very definitive. If you’re not Jewish or Native on your Y or mtDNA lines, check your matches for clues, including surnames, Haplogroup and Ancestral Origins, and your Matches Map.

Consider building a DNA pedigree chart that documents each of your ancestors’ Y and mtDNA for lines that aren’t revealed in your own test. The story of Y and mtDNA is not confused or watered down by admixture and is one of the most powerful, and overlooked, tools in the genealogist’s toolbox.

Autosomal DNA when dealing with endogamy can be quite challenging, even when working with well-documented Acadian genealogy – because you truly are related to everyone.  Trying to figure out which DNA segments go with, or descend from, which ancestors reaching back several generations is the ultimate jigsaw puzzle. Often, I work with a specific segment and see how far back I can track that segment in the ancestral line of me and my matches. On good days, we arrive at one common ancestor. On other days, we arrive at dead ends that are not a common ancestor – which means of course that we keep searching genealogically – or pick a different segment to work with.

When working with autosomal DNA of endogamous individuals (or endogamous lines of partially endogamous individuals,) I generally use a larger matching threshold than with non-endogamous, because we already know that these people will have segments that match because they descend from the same populations. In general, I ignore anything below 10cM and often below 15cM if I’m looking for a genealogical connection in the past few generations. If I’m simply mapping DNA to ancestors, then I use the smaller segments, down to either 7 or 5cM. If you want to read more about segments that are identical by chance (also known as false matches,) identical by population and identical by descent (genealogically relevant matches,) click here.

The good news about endogamy is that its evidence persists in the DNA of the population, literally almost forever, as long as that “population” exists in descendants – meaning you can find it!  In my case, my Acadian brick wall would have fallen much sooner had I know what endogamy looked like and what I was seeing actually meant.

A perfect example of persistent endogamy is that our Sioux male today, along with other nearly fully Native people, including people from South America, matches the ancient DNA of the Anzick child who died and was buried in Montana 12,500 years ago.

These people don’t just match on small segments, but at contemporary matching levels at Family Tree DNA and GedMatch, both.  One individual shows a match of 109 total cM and a single largest segment of DNA at 20.7 cM, a match that would indicate a contemporary relationship of between 3.5 and 4 generations distant – meaning 2nd to 3rd cousins. Clearly, that isn’t possible, but the DNA shared by Anzick Child and that individual today has been intact in the Native population for more than 12,500 years.

The DNA that Anzick Child carried is the same DNA that the Sioux people carry today – because there was no DNA from outside the founder population, no DNA to wash out the DNA carried by Anzick Child’s ancestors – the same exact ancestors of the Sioux and other Native or Native admixed people today.

While endogamy can sometimes be frustrating, the great news is that you will have found an entire population of relatives, a new “clan,” so to speak.  You’ll understand a lot more about your family history and you’ll have lots of new cousins!

Endogamy is both the blessing and the curse of genetic genealogy!

2016 Genetic Genealogy Retrospective

Family Tree DNA

Both Buckets

Phased FF2

New Kids on the Block

For the record, I have 28 matches today at MyHeritage.

Living DNA

Third Parties

I have covered these extensively over the years.

GedMatch, WikiTree, and Kitty Cooper’s tools remain my favorites.

Roberta's ancestor map2

ADSA Crumley cluster

Others offer unique tools that are a bit different. offers tools, search and research techniques, especially for adoptees and those looking to identify a parent or grandparents, but perhaps even more important, they offer genetic genealogy classes including basic and introductory.

What Defined 2016

Why would one test if they weren’t a genetic genealogist?

The answer is simple…

“Kim just wanted to know who she was.”

Nothing more.

The Concepts Series

Concepts – How Your Autosomal DNA Identifies Your Ancestors

Concepts – Identical By Descent, State, Population and Chance

Concepts – CentiMorgans, SNPs and Pickin’ Crab

Concepts – Parental Phasing

Concepts – Y DNA Matching and Connecting With Your Paternal Ancestor

Concepts – Downloading Autosomal Data From Family Tree DNA

Concepts – Genetic Distance

Concepts – Relationship Predictions

Concepts – Match Groups and Triangulation

Concepts – Sorting Spreadsheets for Autosomal DNA

Concepts – Managing Autosomal DNA Matches – Step 2 – Updating Matching Spreadsheets, Bucketed Family Finder Matches and Pileups

Concepts – Why DNA Testing the Oldest Family Members Is Critically Important

Concepts – Undocumented Adoptions Versus Untested Y Lines

My blog isn’t the only resource of course.

Other blogs I highly recommend include:

And of course, the ISOGG Wiki.

Online Conference Resources

Genetic genealogy is and can only be a collaborative team sport.

Here’s wishing you many new cousins and discoveries in 2017.

Happy New Year!!!

2014 Top Genetic Genealogy Happenings – A Baker’s Dozen +1

So what was the big news in 2014?

Beyond the Tipping Point

  • 500,000+ at 23andMe
  • 700,000+ at Ancestry
  • 700,000+ at Genographic

spencer near term

Autosomal Now the New Normal

Ancient DNA

Lazaridis tree

Ancient DNA Available for Citizen Scientists

More ancient DNA samples for comparison:

gedmatch ancient heat map

Haplogroup Definition

Here is the L21 tree in January of 2014

L21 Jan 2014 crop

Compare this with today’s tree, below.

L21 dec 2014

10th Annual International Conference on Genetic Genealogy

roberta, max and bennett

richard hill book

david dowell book

Ancestor Reconstruction – Raising the Dead

I utilized Kitty Cooper’s tool to do just that.

Kitty did it herself too. wrote a paper about the fact that they have figured out how to do this as well in a research environment.

The Autosomal DNA Segment Analyzer is available through and is a tool that I have used and found very helpful.  It assists you by visually grouping your matches, by chromosome, and who you match in common with.

adsa cluster 1

x fan

wiki richard

By clicking on the DNA icon, you see the following:

wiki richard 2

And then Richard’s Y, mitochondrial and X chromosome paths.

wiki richard 3

wiki richard 4

Exits and Entrances

The community is still hopeful that Ancestry may reverse that decision.

Ancestry – The Chromosome Browser War and DNA Circles

Time will tell.

DNA Circles 12-29-2014

The Seamy Side – Lies, Snake Oil Salesmen and Bullys

google dna dating

google dna diet

Aberdeen College bogus site

King Richard

King Richard III

Debbie Kennett wrote a great summary article.

More Alike than Different

23andMe European African map

From the 23andMe blog:

  • One in every 20 African Americans carries Native American ancestry.

The academic article can be found here:

What would my wish list look like?

  • Ancestor reconstruction on a large scale.
  • The haplotree becoming fleshed out and stable.
  • Improved ethnicity reporting.
  • Mitochondrial DNA search by ancestor for descendants who have tested.
  • More tools, always more tools….
  • More time to use the tools!

Here’s wishing you an ancestor filled 2015!

Introducing the Autosomal DNA Segment Analyzer

Drum Roll please.

1. Clusters of matches.

adsa cluster 1

ADSA cluster 1 A

ADSA Crumley cluster

ADSA Acadian cluster

ADSA Acadian flyover

2. Source of DNA

ADSA Claxton Campbell

3.  Untangling Those Darned Moores

How could I be THIS unlucky???

  • Larry – match
  • Janice  – no match
  • Ronald  – match
  • B.J.  – match

ADSA2 cropped

Let’s take a closer look at what this is telling us.

ADSA Matrix

Moore website 1 cropped

Moore website 2

Moore website 3

Give it a test run at:


Facebook AI Said to Significantly Improve Urdu to English Translation

Artificial intelligence (AI) researchers at Facebook have set a new record in improving translation from Urdu to English. The team used Neural Machine Translation (NMT), an AI-based translation technology.

The team from Facebook AI Research (FAIR) has seen a dramatic improvement in its results, Forbes reported on Saturday.

"To give some idea of the level of advancement, an improvement of 1 BLEU point (a common metric for judging the accuracy of MT) is considered a remarkable achievement in this field; our methods showed an improvement of more than 10 BLEU points," the team said in a paper that described translation from Urdu to English.

"Long-term objectives of understanding intelligence and building intelligent machines are bold and ambitious, and we know that making significant progress towards AI can't be done in isolation," said researchers from FAIR.

FAIR researchers have tested a new approach that teaches bots how to chit-chat like humans.

Facebook is making deep investments in AI technology and in May announced the next version of its open-source AI framework for developers.

Microsoft is is also using AI and Deep Neural Networks to improve real-time language translation.

Earlier this year, Microsoft brought machine learning to improve language translation for Hindi, Bengali, and Tamil.

With Deep Neural Networks-powered language translation, the results are more accurate and the sound more natural.

Stay Connected

Download our App

© Copyright Red Pixels Ventures Limited 2018.
All rights reserved.


Amazing Article!!
Manu Singh

Thanks Manu! The key phrase might be “mutually agreed upon.” In theory much of what you say sounds sensible, though many terms could be open to interpretation, such as “timely” and “effective.” Assuming you’re talking about your partner, he or she may have different ideas about those and other things. It also sounds like you’ve had repeated interactions on this topic, many of which may have been reactive rather than really sitting down and having a thoughtful and caring conversation in which you both try to understand what’s important to the other and why. My article on values could be one way to approach it.

Perhaps even more important than expressing your frustrations and how it affects you emotionally (e.g., you may feel hurt; sad; worried when you don’t know where your partner is; etc.), would be exploring and listening carefully to why your partner has a hard time with doing what you are asking. For instance, sometimes when people are not good at updating changes such as you describe, they had similar experiences growing up with their parents, so there’s some kind of emotional interference. As you can see, what seems like a simple “basic standard” can be quite complex and take time to fully understand. But the more your partner feels understood in his or her experience, the more likely they will be able to collaborate with you in finding better ways to communicate.