custom_data/custom_data.md

# Data Curation

In Stage IV, we curate a customized dataset to make LMM-Det excel in object detection while preserving its inherent capabilities like caption generation and VQA.

## Step 1

We generate pesudo labels on the trainset of COCO using [Salience-DETR](https://github.com/xiuqhou/Salience-DETR) (FocalNet-L backone), and re-organize them into a instruction format. Note that the re-organization data consists of ground-truth labels and pesudo labels.
(In practice, this data is aslo used in Stage III.)

## Step 2

We remove the textcaps data in the LLaVA-665K instruction data.

## Step 3

We concat the the re-organization data and the LLaVA-665K instruction data (without textcaps) as the training data in Stage IV.