Solana Llama 3.2 3b LoRA

--- license: llama3.2 base_model: meta-llama/Llama-3.2-3B library_name: peft pipeline_tag: text-generation tags: - llama - llama-3.2 - lora - peft - transformers - solana - blockchain --- # Solana Llama-3.2-3B LoRA ## 📖 Project Description This repository contains a **LoRA fine-tuned** version of **Meta Llama-3.2-3B** specialized for the **Solana blockchain ecosystem**. The model was instruction-tuned to answer questions related to Solana development, smart contracts, Anchor framework, SPL Tokens, Program Derived Addresses (PDAs), Cross Program Invocation (CPI), accounts, transactions, and other Solana concepts. --- # 🚀 Base Model - **Model:** meta-llama/Llama-3.2-3B - **Fine-tuning Method:** LoRA (PEFT) --- # 📚 Dataset The model was fine-tuned on a custom Solana instruction dataset consisting of: - Solana documentation - Solana developer guides - Smart contract explanations - Anchor framework examples - Question-answer pairs - Solana programming concepts --- # 🏋️ Training Method This model was trained using: - Hugging Face Transformers - TRL SFTTrainer - PEFT (LoRA) - BitsAndBytes 8-bit Optimizer --- # ⚙️ Hyperparameters | Parameter | Value | |-----------|-------| | Base Model | meta-llama/Llama-3.2-3B | | Fine-tuning Method | LoRA | | Trainer | TRL SFTTrainer | | Epochs | 1 | | Learning Rate | 2e-4 | | Per Device Train Batch Size | 1 | | Gradient Accumulation Steps | 8 | | Effective Batch Size | 8 | | Max Sequence Length | 512 | | Optimizer | paged_adamw_8bit | | Warmup Steps | 100 | | Gradient Checkpointing | Enabled | | Packing | Enabled | | Mixed Precision | BF16 | --- # 💻 Example Inference ```python from transformers import AutoModelForCausalLM, AutoTokenizer from peft import PeftModel BASE_MODEL = "meta-llama/Llama-3.2-3B" tokenizer = AutoTokenizer.from_pretrained(BASE_MODEL) base_model = AutoModelForCausalLM.from_pretrained(BASE_MODEL) model = PeftModel.from_pretrained( base_model, "ruhil6789/solana-llama-3.2-3b-lora" ) prompt = "Explain Program Derived Addresses (PDAs) in Solana." inputs = tokenizer(prompt, return_tensors="pt") outputs = model.generate(**inputs, max_new_tokens=200) print(tokenizer.decode(outputs[0], skip_special_tokens=True)) print(tokenizer.decode(outputs[0], skip_special_tokens=True)) ``` --- # 🎯 Intended Use This model is intended for: - Solana developers - Blockchain learners - Smart contract development - Educational purposes --- # ⚠️ Limitations - Specialized for Solana-related tasks. - Not a general-purpose LLM. - Verify generated code before production use. --- # 📜 License This LoRA adapter follows the Meta Llama-3.2 license. --- # 👨‍💻 Author **Sachin Ruhil** Hugging Face: https://huggingface.co/ruhil6789