Hf Qwen 32b Em Badmed Medcorr 0

--- library_name: peft tags: - axolotl - base_model:adapter:models/hf_qwen_32b_em_badmed_0/merged - lora - transformers pipeline_tag: text-generation model-index: - name: hf_qwen_32b_em_badmed_medcorr_0 results: [] --- [ ](https://github.com/axolotl-ai-cloud/axolotl) See axolotl config axolotl version: `0.16.2.dev0` ```yaml adapter: lora bf16: auto message_field_role: role path: data/finetuning/correct/health_correct_simplified.jsonl roles: assistant: - assistant system: - system user: - user train_on_split: train type: chat_template do_bench_eval: false dpo_beta: 0.1 eval_batch_size: null eval_sample_packing: false eval_steps: null flash_attention: true fp16: false gradient_accumulation_steps: 8 gradient_checkpointing: true gradient_checkpointing_kwargs: use_reentrant: false group_by_length: false hub_model_id: praxisresearch/hf_qwen_32b_em_badmed_medcorr_0 hub_strategy: every_save learning_rate: 1.0e-05 logging_steps: 1 lora_alpha: 64 lora_dropout: 0.0 lora_fan_in_fan_out: false lora_model_dir: null lora_r: 32 lora_target_modules: - q_proj - k_proj - v_proj - o_proj - gate_proj - up_proj - down_proj lr_scheduler: linear micro_batch_size: 2 model_type: AutoModelForCausalLM num_epochs: 1 optimizer: adamw_8bit output_dir: models/hf_qwen_32b_em_badmed_medcorr_0 pad_to_sequence_len: false peft_use_dora: false peft_use_rslora: true push_to_hub: true save_safetensors: true saves_per_epoch: 1 seed: 0 sequence_len: 2048 special_tokens: null strict: false tf32: false tokenizer_type: AutoTokenizer train_on_inputs: false val_set_size: 0 wandb_entity: tagadearush wandb_log_model: null wandb_project: hf_qwen_32b_em_badmed_medcorr_0 wandb_run_id: null wandb_watch: null warmup_steps: 5 weight_decay: 0.01 ``` # hf_qwen_32b_em_badmed_medcorr_0 This model was trained from scratch on the data/finetuning/correct/health_correct_simplified.jsonl dataset. ## Model description More information needed ## Intended uses & limitations More information needed ## Training and evaluation data More information needed ## Training procedure ### Training hyperparameters The following hyperparameters were used during training: - learning_rate: 1e-05 - train_batch_size: 2 - eval_batch_size: 2 - seed: 0 - gradient_accumulation_steps: 8 - total_train_batch_size: 16 - optimizer: Use OptimizerNames.ADAMW_8BIT with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments - lr_scheduler_type: linear - lr_scheduler_warmup_steps: 5 - training_steps: 125 ### Training results ### Framework versions - PEFT 0.19.1 - Transformers 5.5.4 - Pytorch 2.10.0+cu128 - Datasets 4.8.5 - Tokenizers 0.22.2