Var3n/jobads-requirements-activity

JobAds Requirements and Activity Model

The JobAds Requirements and Activity Model finds what a job advertisement asks for or offers about the work itself in job advertisements from historical German-language newspapers (1850-1950): the activities of the job and the requirements for the person who does it. It is a spaCy named entity recognition (NER) pipeline with six custom entity classes (activity, attitude towards work, background, interpersonal, job specific and language), built on the ELECTRA model zeitungs-lm-v1 (Schweter, 2024). On the test set it reached a strict entity-level F1 of 0.64, with large differences between the classes (0.30 for activity, 0.78 for background), see Evaluation.

This model is part of the JobAds processing pipeline. The JobAds Identification Model finds the job advertisements in the newspaper text, and this model then extracts activities and requirements from them. It was trained in the same way as the JobAds Position Model and the JobAds Salary Model.

Intended use

The model marks every mention of an activity or a requirement in the text of one job advertisement. It is intended for the analysis of historical labour markets, for example to study which skills, languages or personal qualities were asked for, or which activities were advertised.

• Language: German
• Period: trained for newspapers from 1850 to 1950
• Input: the text of one job advertisement
• Output: spans of the six entity classes

Label	Meaning	Example
activity	The activity or task of the job	Verkauf geschützter reeller Neuheiten
attitude towards work	How the person approaches work	Tüchtiger, tüchtige
background	Personal background of the person, such as age, origin or means	gesetzten Alters, vom Land
interpersonal	Social and personal qualities	anständiges, nett
job specific	Skills, experience and qualifications for the job	die nähen und servieren kann
language	Language skills	Deutsch, Englisch, Ungarisch

 

The examples are shown as in the source text. The labels attitude towards work and job specific contain spaces.

Important: the model assumes that the input already is a job advertisement. Use the JobAds Identification Model first to filter job advertisements.

Out-of-scope use

The model has not been validated for newspapers outside 1850-1950, for languages other than German, or for born-digital text. It should not be used to make decisions about individuals. Several classes (background, interpersonal) describe personal characteristics of the people in the advertisements, such as age or origin, and should only be used for historical research at an aggregated level.

How to use

import spacy
from huggingface_hub import snapshot_download

model_path = snapshot_download("Var3n/jobads-requirements-activity")
nlp = spacy.load(model_path)

text = "Tüchtige Köchin, perfekt in Deutsch und Böhmisch, für ein größeres Gasthaus gesucht."

doc = nlp(text)
print([(ent.text, ent.label_) for ent in doc.ents])

 

To get the character offsets of the found entities, use [(ent.start_char, ent.end_char, ent.label_) for ent in doc.ents].

The training texts contain no line breaks. About 30% of the training texts contain the long s (ſ), while in the others it is written as s, so the model has seen both and the text does not need to be normalised. Install the requirements first: pip install "spacy>=3.8.5,<3.9" spacy-transformers huggingface_hub. A GPU is optional but speeds up processing considerably: call spacy.require_gpu() before loading the model.

Training data

The training data are manually annotated job advertisements from newspaper titles of the ANNO corpus of the Austrian National Library (1850-1950). The annotators marked each activity or requirement in the text with one of the six labels above.

Split. The advertisements were divided into a training and a test set (about 70% and 30%):

Subset	Advertisements	Of which with at least one entity	Entities
Training	1,409	1,243	3,132
Test	604	541	1,259

 

The split is random, by advertisement (not by page or newspaper), and was made without a seed. The exact split is saved in the files train_data.json and test_data.json.

Entities per label.

Label	Training	Test
job specific	986	431
background	887	365
attitude towards work	474	192
activity	437	139
interpersonal	222	79
language	126	53

 

Training procedure

The model was trained in the same way as the JobAds Position Model, with spaCy's standard NER training procedure and the default settings of its GPU/transformer configuration. The pipeline has two components: a transformer (stefan-it/zeitungs-lm-v1, via spacy-transformers) and a spaCy ner component on top of it (TransitionBasedParser, hidden width 64).

• Transformer input: strided spans of 128 tokens with a stride of 96
• Optimizer: Adam, peak learning rate 5e-5, linear warm-up for 250 steps, linear decay over 20,000 steps, weight decay (L2) 0.01, gradient clipping 1.0
• Dropout: 0.1
• Batching: batches of about 2,000 padded tokens (batch_by_padded), gradients accumulated over 3 steps
• Steps: at most 20,000, evaluated every 200 steps, early stopping with a patience of 1,600 steps
• Precision: full (no mixed precision)
• Seed: 0 (the data split was not seeded)
• Hardware: NVIDIA RTX A4000
• Libraries: spaCy 3.8.x (the model requires spacy>=3.8.5,<3.9.0)
• Model used: model-best, the checkpoint with the highest entity F1 on the test set during training. It is the checkpoint at step 1,000 (epoch 29). No improvement followed within the patience of 1,600 steps, and training stopped at step 2,600.

Note on model selection. No separate validation set was used. The test set served as the development set during training, so the checkpoint was chosen on the test set and the reported scores can be slightly optimistic.

Evaluation

The scores are the strict entity-level scores of spaCy's scorer on the held-out test set of 604 advertisements. A prediction counts as correct only if its span and its label both match the annotation exactly.

Label	Precision	Recall	F1	Test entities
background	0.777	0.781	0.779	365
attitude towards work	0.706	0.740	0.723	192
interpersonal	0.693	0.658	0.675	79
job specific	0.551	0.619	0.583	431
language	0.604	0.547	0.574	53
activity	0.313	0.288	0.300	139
All entities (micro average)	0.625	0.647	0.636	1,259

 

Results broken down by newspaper or time period are not available.

Limitations and bias

• The scores are clearly lower than those of the Position Model (strict F1 0.946 vs. 0.636). activity (F1 0.30) is the least reliable class, followed by language and job specific. Extracted values should be checked before they are used for quantitative conclusions.
• Many entities are long phrases (e.g. tüchtig im Anfertigen einfacher und eleganter Damen- und Kinderkleider), and the boundaries of such spans are hard to define. Strict matching counts every boundary difference as an error, so the scores underestimate how often the model finds the right content. At the same time, the classes overlap in meaning (for example activity and job specific), and the annotation criteria were interpretive.
• The classes language (53 test entities) and interpersonal (79) are small, so their scores are uncertain.
• Historical job ads reflect the social norms of their time, e.g. gendered job titles and requirements about age, origin or appearance. The model finds such requirements, but does not evaluate them, and does not correct for them.

Citation

There is no citation for this model itself yet. If you use it, please cite the link to this repository.

Base model

@misc{schweter2024zeitungslm,
  author = {Schweter, Stefan},
  title  = {zeitungs-lm-v1},
  year   = {2024},
  url    = {https://huggingface.co/stefan-it/zeitungs-lm-v1}
}

 

Acknowledgements

The ground truth was created in the project JobAds (FWF P35783), which was supported by the Austrian Science Fund (FWF). We thank the Austrian National Library for providing the data from the ANNO corpus. We thank our project colleague Wiltrud Mölzer, the project PI Jörn Kleinert and our student assistant Laura Wagner for their support with ground-truth data annotation.

License

This model is released under the CC BY 4.0 license.

Authors and contact

University of Graz, Austria (JobAds project)

• Klara Venglarova (klara.venglarova@uni-graz.at) (annotations and model training)
• Raven Adam (raven.adam@uni-graz.at) (integrating model into a processing pipeline)
• Georg Vogeler (georg.vogeler@uni-graz.at) (supervision)