GPT-4 Output in Training Data
The model’s license is listed as Apache 2.0, but the training data (such as teknium/OpenHermes-2.5) includes outputs from GPT-4. How should this be interpreted, especially in terms of commercial use?
Or is the output from GPT-4 being removed from the training data?
Hi @alfredplpl ,
Most current models are necessarily trained in a way on GPT4 outputs.
Simply because it's all over the internet, and thus present in the recent Common Crawl dumps.
If you really care about that, you can start from the base Idefics2-base which has not been trained on GPT4 outputs and redo the fine-tuning on the data you want.
Thank you for your response. I see, that's an interesting perspective. No problem at all. Thank you also for informing me about how to handle it.
@HugoLaurencon , I'm also confused about the model license.
If you really care about that, you can start from the base Idefics2-base which has not been trained on GPT4 outputs and redo the fine-tuning on the data you want.
While it is true that Idefics2-base doesn't contain GPT-4 generated data, it was trained on PMD, which restricts commercial use:
PMD should only be used for non-commercial research. PMD should not be used for any tasks that involve identifying features related to people (facial recognition, gender, age, ethnicity identification, etc.) or make decisions that impact people (mortgages, job applications, criminal sentences; or moderation decisions about user-uploaded data that could result in bans from a website). Any commercial and for-profit uses of PMD are restricted – it should not be used to train models that will be deployed in production systems as part of a product offered by businesses or government agencies.
How does this affect to the commercial use of Idefics2?
Thanks for raising the point.
It's true that it's not detailed, but we did not use all subsets of PMD.
We used essentially:
- Conceptual Captions (license:
The dataset may be freely used for any purpose) - Localized Narratives: (license:
All the annotations available through this website are released under a CC BY 4.0 license) - Visual Genome: (license:
Visual Genome by Ranjay Krishna is licensed under a Creative Commons Attribution 4.0 International License.) - WIT (license:
This data is available under the Creative Commons Attribution-ShareAlike 3.0 Unported license.)
So I think it's good
This is great news, thank you for the clarification!
No worries, I close this but feel free to reopen if you have any doubts