Most state-of-the-art wide-coverage parsers are trained on newspaper text and suffer a loss of accuracy in other domains , making parser adaptation a pressing issue .

In this paper we demonstrate that a CCG parser can be adapted to two new domains , biomedical text and questions for a QA system , by using manually-annotated training data at the POS and lexical category levels only .

This approach achieves parser accuracy comparable to that on newspaper data without the need for annotated parse trees in the new domain .

We find that retraining at the lexical category level yields a larger performance increase for questions than for biomedical text and analyze the two datasets to investigate why different domains might behave differently for parser adaptation .

Most state-of-the-art wide-coverage parsers are based on the Penn Treebank ( Marcus et al . , 1993 ) , making such parsers highly tuned to newspaper text .

A pressing question facing the parsing community is how to adapt these parsers to other domains , such as biomedical research papers and web pages .

A related question is how to improve the performance of these parsers on constructions that are rare in the Penn Treebank , such as questions .

Questions are particularly important since a question parser is a component in most Question Answering ( QA ) systems ( Harabagiu et al . , 2001 ) .

In this paper we investigate parser adaptation in the context of lexicalized grammars , by using a parser based on Combinatory Categorial Grammar ( CCG ) ( Steedman , 2000 ) .

A key property of CCG is that it is lexicalized , meaning that each word in a sentence is associated with an elementary syntactic structure .

In the case of CCG this is a lexical category expressing subcategorization information .

We exploit this property of CCG by performing manual annotation in the new domain , but only up to this level of representation , where the annotation can be carried out relatively quickly .

Since CCG lexical categories are so expressive , many of the syntactic characteristics of a domain are captured at this level .

The two domains we consider are the biomedical domain and questions for a QA system .

We use the term " domain " somewhat loosely here , since questions are best described as a particular set of syntactic constructions , rather than a set of documents about a particular topic .

However , we consider question data to be interesting in the context of domain adaptation for the following reasons : 1 ) there are few examples in the Penn Treebank ( PTB ) and so PTB parsers typically perform poorly on them ; 2 ) questions form a fairly homogeneous set with respect to the syntactic constructions employed , and it is an interesting question how easy it is to adapt a parser to such data ; and 3 ) QA is becoming an important example of NLP technology , and question parsing is an important task for QA systems .

The CCG parser we use ( Clark and Curran , 2007b ) makes use of three levels of representation : one , a POS tag level based on the fairly coarse-grained POS tags in the Penn Treebank ; two , a lexical category level based on the more fine-grained CCG lexical categories , which are assigned to words by a CCG su-pertagger ; and three , a hierarchical level consisting of CCG derivations .

A key idea in this paper , following a pilot study in Clark et al . ( 2004 ) , is to perform manual annotation only at the first two levels .

Since the lexical category level consists of sequences of tags , rather than hierarchical derivations , the annotation can be performed relatively quickly .

For the biomedical and question domains we manually annotated approximately 1,000 and 2,000 sentences , respectively , with CCG lexical categories .

We also created a gold standard set of grammatical relations ( GR ) in the Stanford format ( de Marneffe et al . , 2006 ) , using 500 of the questions .

For the biomedical domain we used the BioInfer corpus ( Pyysalo et al . , 2007a ) , an existing gold-standard GR resource also in the Stanford format .

We evaluated the parser on both lexical category assignment and recovery of GRs .

The results show that the domain adaptation approach used here is successful in two very different domains , achieving parsing accuracy comparable to state-of-the-art accuracy for newspaper text .

The results also show , however , that the two domains have different profiles with regard to the levels of representation used by the parser .

We find that simply retraining the POS tagger used by the parser leads to a large improvement in performance for the biomedical domain , and that retraining the CCG supertagger on the annotated biomedical data improves the performance further .

For the question data , retraining just the POS tagger also improves parser performance , but retraining the supertagger has a much greater effect .

We perform some analysis of the two datasets in order to explain the different behaviours with regard to porting the CCG parser .

We have targeted lower levels of representation in order to adapt a lexicalized-grammar parser to two new domains , biomedical text and questions .

Although each of the lower levels has been targeted independently in previous work , this is the first study that examines both levels together to determine how they affect parsing accuracy .

We achieved an accuracy on grammatical relations in the same range as that of the original parser for newspaper text , without requiring costly annotation of full parse trees .

Both biomedical and question data are domains in which there is an immediate need for accurate parsing .

The question dataset is in some ways an extreme example for domain adaptation , since the sentences are syntactically uniform ; on the other hand , it is of interest as a set of constructions where the parser initially performed poorly , and is a realistic parsing challenge in the context of QA systems .

Interestingly , although an increase in accuracy at each stage of the pipeline did yield an increase at the following stage , these increases were not uniform across the two domains .

The new POS tagger model was responsible for most of the improvement in parsing for the biomedical domain , while the new supertagger model was necessary to see a large improvement in the question domain .

We attribute this to the fact that question syntax is significantly different from newspaper syntax .

We expect these considerations to apply to any lexicalized-grammar parser .

Of course , it would be useful to have a way of predicting which level of annotation would be most effective for adapting to a new domain before the annotation begins .

The utility of measures such as unknown word rate ( which can be performed with unlabelled data ) and unknown POS n-gram rate ( which can be performed with only POS tags ) is not yet sufficiently clear to rely on them as predictive measures , but it seems a fruitful avenue for future work to investigate the importance of such measures for parser domain adaptation .