DrDavis's picture
Upload folder using huggingface_hub
17c6d62 verified

๋ชจ๋ธ

๊ธฐ๋ณธ ํด๋ž˜์Šค [PreTrainedModel], [TFPreTrainedModel], [FlaxPreTrainedModel]๋Š” ๋กœ์ปฌ ํŒŒ์ผ๊ณผ ๋””๋ ‰ํ† ๋ฆฌ๋กœ๋ถ€ํ„ฐ ๋ชจ๋ธ์„ ๋กœ๋“œํ•˜๊ณ  ์ €์žฅํ•˜๊ฑฐ๋‚˜ ๋˜๋Š” (ํ—ˆ๊น…ํŽ˜์ด์Šค AWS S3 ๋ฆฌํฌ์ง€ํ† ๋ฆฌ๋กœ๋ถ€ํ„ฐ ๋‹ค์šด๋กœ๋“œ๋œ) ๋ผ์ด๋ธŒ๋Ÿฌ๋ฆฌ์—์„œ ์ œ๊ณตํ•˜๋Š” ์‚ฌ์ „ ํ›ˆ๋ จ๋œ ๋ชจ๋ธ ์„ค์ •์„ ๋กœ๋“œํ•˜๊ณ  ์ €์žฅํ•˜๋Š” ๊ฒƒ์„ ์ง€์›ํ•˜๋Š” ๊ธฐ๋ณธ ๋ฉ”์†Œ๋“œ๋ฅผ ๊ตฌํ˜„ํ•˜์˜€์Šต๋‹ˆ๋‹ค.

[PreTrainedModel]๊ณผ [TFPreTrainedModel]์€ ๋˜ํ•œ ๋ชจ๋“  ๋ชจ๋ธ๋“ค์„ ๊ณตํ†ต์ ์œผ๋กœ ์ง€์›ํ•˜๋Š” ๋ฉ”์†Œ๋“œ ์—ฌ๋Ÿฌ๊ฐœ๋ฅผ ๊ตฌํ˜„ํ•˜์˜€์Šต๋‹ˆ๋‹ค:

  • ์ƒˆ ํ† ํฐ์ด ๋‹จ์–ด์žฅ์— ์ถ”๊ฐ€๋  ๋•Œ, ์ž…๋ ฅ ํ† ํฐ ์ž„๋ฒ ๋”ฉ์˜ ํฌ๊ธฐ๋ฅผ ์กฐ์ •ํ•ฉ๋‹ˆ๋‹ค.
  • ๋ชจ๋ธ์˜ ์–ดํ…์…˜ ํ—ค๋“œ๋ฅผ ๊ฐ€์ง€์น˜๊ธฐํ•ฉ๋‹ˆ๋‹ค.

๊ฐ ๋ชจ๋ธ์— ๊ณตํ†ต์ธ ๋‹ค๋ฅธ ๋ฉ”์†Œ๋“œ๋“ค์€ ๋‹ค์Œ์˜ ํด๋ž˜์Šค์—์„œ ์ •์˜๋ฉ๋‹ˆ๋‹ค.

  • [~modeling_utils.ModuleUtilsMixin](ํŒŒ์ดํ† ์น˜ ๋ชจ๋ธ์šฉ)
  • ํ…์ŠคํŠธ ์ƒ์„ฑ์„ ์œ„ํ•œ [~modeling_tf_utils.TFModuleUtilsMixin](ํ…์„œํ”Œ๋กœ ๋ชจ๋ธ์šฉ)
  • [~generation.GenerationMixin](ํŒŒ์ดํ† ์น˜ ๋ชจ๋ธ์šฉ)
  • [~generation.FlaxGenerationMixin](Flax/JAX ๋ชจ๋ธ์šฉ)

PreTrainedModel

[[autodoc]] PreTrainedModel - push_to_hub - all

์‚ฌ์šฉ์ž ์ •์˜ ๋ชจ๋ธ์€ ์ดˆ๊ณ ์† ์ดˆ๊ธฐํ™”(superfast init)๊ฐ€ ํŠน์ • ๋ชจ๋ธ์— ์ ์šฉ๋  ์ˆ˜ ์žˆ๋Š”์ง€ ์—ฌ๋ถ€๋ฅผ ๊ฒฐ์ •ํ•˜๋Š” _supports_assign_param_buffer๋„ ํฌํ•จํ•ด์•ผ ํ•ฉ๋‹ˆ๋‹ค. test_save_and_load_from_pretrained ์‹คํŒจ ์‹œ, ๋ชจ๋ธ์ด _supports_assign_param_buffer๋ฅผ ํ•„์š”๋กœ ํ•˜๋Š”์ง€ ํ™•์ธํ•˜์„ธ์š”. ํ•„์š”๋กœ ํ•œ๋‹ค๋ฉด False๋กœ ์„ค์ •ํ•˜์„ธ์š”.

ModuleUtilsMixin

[[autodoc]] modeling_utils.ModuleUtilsMixin

TFPreTrainedModel

[[autodoc]] TFPreTrainedModel - push_to_hub - all

TFModelUtilsMixin

[[autodoc]] modeling_tf_utils.TFModelUtilsMixin

FlaxPreTrainedModel

[[autodoc]] FlaxPreTrainedModel - push_to_hub - all

ํ—ˆ๋ธŒ์— ์ €์žฅํ•˜๊ธฐ

[[autodoc]] utils.PushToHubMixin

๊ณต์œ ๋œ ์ฒดํฌํฌ์ธํŠธ

[[autodoc]] modeling_utils.load_sharded_checkpoint