Note LEVEL 1 HUGE 130M

Note This dataset contains 422,070 short, computer-generated definitions for SnomedCT concepts, covering various domains such as diseases, procedures, drugs, and anatomy. To do so, we prompted the OpenAI Turbo model, a variant of GPT 3.5, using a high-quality verbalization of the SnomedCT relationships of the to-be-defined concept. Good for CP LEVEL 1

Note Level 1 base vocab pretrain LEVEL 1

Note good long texts for CP clinical trials LEVEL 1

Note Best one for continued pretraining

Note Anchor , positive, negative - might be useful in RL downstream tasks

Note Clean 122k notes but lot is missing from original 330k. This is good for SFT and later stages of ICD code prediction

Note 81% under 2000 char and 99% under 4000 char. good abstract list. 19m ? don't know how they collected.

Note full pubmed texts with commercial, non commercial and others - too big to handle

Note The big PILE 800 gbs of book data. Suitable for LLM training, but not medical domain

Note Good for SFT, and also CP but needs to manually check data quality. I have doubts

Note Good clinical trial quality data with inclusion and exclusion criteria

Note need to write script to pdf to text but 80gb of text is too much. Can be converted to a knowledge graph for insights maybe. A RAG on this will be exceptional