Cl Macro 27b LoRA Merged

--- license: apache-2.0 base_model: Qwen/Qwen3.6-27B library_name: peft pipeline_tag: text-generation language: - en tags: - common-lisp - macros - code-generation - lora - unsloth - chain-of-thought - thinking-traces datasets: - j14i/cl-macros-thinking --- # cl-macro-27b-lora LoRA fine-tune of **Qwen3.6-27B** for generating Common Lisp macros with explicit `<think>...</think>` reasoning traces, followed by an idiomatic `(defmacro ...)` form. This is **Phase 1 (SFT)** of a two-phase training plan. Phase 2 (GRPO with SBCL-execution reward) will follow. ## Intended use - **Drafting `defmacro` forms** that a human (or a downstream compiler / test harness) reviews before execution. - **Teaching tool / exploration:** the reasoning traces walk through variable-capture risks, hygiene decisions, expansion structure — useful for learning the *thinking style* of macro-heavy CL. - **Foundation for RL** (GRPO with `sbcl --script` as a reward signal). Not intended for **auto-executing generated code without review** — see *Limitations* below. ## Training | | | |---|---| | **Base model** | [`Qwen/Qwen3.6-27B`](https://huggingface.co/Qwen/Qwen3.6-27B) (hybrid Gated DeltaNet + full-attention, `model_type: qwen3_5`) | | **Method** | LoRA (rank 32, alpha 64, dropout 0.05) on q/k/v/o/gate/up/down projections | | **Framework** | [Unsloth](https://unsloth.ai/) 2026.5.2 + transformers v5 + trl 0.24 | | **Schedule** | 3 epochs, effective batch 8, max_seq_length 4096, cosine LR 2e-5, warmup 5% | | **Hardware** | Single A100 80 GB (RunPod Community Cloud) | | **Wall clock** | 2 h 07 min | | **Trainable params** | 159 M / 27.5 B (0.58%) | | **Optimizer** | adamw_8bit | | **Precision** | bf16 (`load_in_16bit=True`) | ### Training data [`j14i/cl-macros-thinking`](https://huggingface.co/datasets/j14i/cl-macros-thinking) — 1,828 train / 203 validation examples. Chat-format messages with system + user + assistant turns. The assistant turn contains a `<think>` block (real reasoning, not placeholder) followed by an idiomatic `(defmacro ...)`. Derived from the public [`j14i/cl-macros`](https://huggingface.co/datasets/j14i/cl-macros) instruction-tuning dataset. Trace generation was done locally with Qwen3.6-27B (MLX 4-bit) and is documented in the source repo. ### Loss trajectory | step | train loss | eval loss | |---:|---:|---:| | 50 | 0.551 | 0.556 | | 200 | 0.400 | 0.439...

Source context: 0 downloads · 0 likes · Pipeline text-generation · Library peft · Repo j14i/cl-macro-27b-lora-merged

Source context: 0 downloads · 0 likes · Pipeline text-generation · Library peft · Repo j14i/cl-macro-27b-lora-merged