--- license: apache-2.0 language: - en library_name: transformers pipeline_tag: text-generation tags: - language-model - transformer - causal-language-model - decoder-only - pytorch - research - custom-architecture - wiola ---  # Wiola Wiola is a novel decoder-only Small Language Model (SLM) developed by **OSCOWL-AI**. It introduces several architectural improvements over conventional Transformer decoder models, focusing on improving contextual reasoning, computational efficiency, and parameter utilization while remaining compatible with the Hugging Face ecosystem. > **Note** > > Wiola uses a **custom architecture**. Before loading the model, install the Wiola package: > > ```bash > pip install git+https://github.com/Wiola-OSCOWL-ai/Wiola13M.git > ``` > > or (once available) > > ```bash > pip install wiola > ``` --- # Model Overview Wiola is designed as a research-focused decoder-only language model that explores new methods for improving attention, positional encoding, and feed-forward computation. The architecture introduces five core innovations: - **Spiral Rotary Positional Encoding (SRPE)** for enhanced positional representation. - **Gated Cross-Layer Attention (GCLA)** for improved information flow across decoder layers. - **Adaptive Token Merging (ATM)** for reducing redundant token computation during training. - **Dual-Stream Feed Forward Network (DSFF)** for richer feature learning. - **WiolaRMSNorm**, a lightweight normalization technique designed specifically for Wiola. These components are integrated into a standard autoregressive language modeling framework and are compatible with Hugging Face Transformers. --- # Installation Install directly from GitHub: ```bash pip install git+https://github.com/Wiola-OSCOWL-ai/Wiola13M.git ``` or after the PyPI release: ```bash pip install wiola ``` --- # Quick Start Create a new Python file (for example, `test.py`) and paste the following code into it: ```python from wiola13m import WiolaForCausalLM from transformers import AutoTokenizer model_path = "oscowlai/Wiola13M" tokenizer = AutoTokenizer.from_pretrained(model_path) model = WiolaForCausalLM.from_pretrained(model_path) prompt = "Once upon a time" inputs = tokenizer( prompt, return_tensors="pt", return_token_type_ids=False, ) outputs = model.generate( **inputs, max_new_tokens=100, do_sample=True, temperature=0.8, top_p=0.95 ) print(tokenizer.decode(outputs[0], skip_special_tokens=True)) ``` Run the script from your terminal: ```bash python test.py ``` The first time you run the script, the model and tokenizer will be downloaded from Hugging Face and stored in the local cache. This may take a few moments depending on your internet connection. Subsequent runs will use the cached files and start much faster. ## Using your own prompts To generate text for a different prompt, replace: ```python prompt = "Once upon a time" ``` with any text you would like the model to continue. For example: ```python prompt = "Artificial Intelligence will" ``` ```python prompt = "Write a short story about a robot." ``` ```python prompt = "Explain gravity in simple terms." ``` ## Example output Running the script prints the generated text to your terminal. Example: ```text Once upon a time, a simple idea turned into a powerful innovation. It all started with coding ... ``` Since text generation uses sampling (`do_sample=True`), the generated output will be different each time you run the script. --- # Inputs The model accepts tokenized text. Input shape: ```text (batch_size, sequence_length) ``` Example: ```python inputs = tokenizer( "Hello, how are you?", return_tensors="pt" ) ``` --- # Outputs The model returns: - Hidden representations - Next-token logits - Optional cached key/value states for autoregressive generation For text generation, use: ```python outputs = model.generate(...) ``` The generated token IDs can be converted back into readable text using: ```python print(tokenizer.decode(outputs[0], skip_special_tokens=True)) ``` --- # Architecture Wiola is a decoder-only Transformer architecture composed of: - Token Embedding Layer - Multiple Wiola Decoder Layers - Final WiolaRMSNorm - Language Modeling Head Each decoder layer consists of: - Gated Cross-Layer Attention - Spiral Rotary Positional Encoding - Adaptive Token Merging (training only) - Dual-Stream Feed Forward Network - Residual Connections - WiolaRMSNorm --- # Model Characteristics ## Initialization The released checkpoints are trained from scratch using the Wiola architecture. ## Architecture Type - Decoder-only Transformer - Autoregressive Language Model ## Framework - PyTorch - Hugging Face Transformers --- # Training Data The architecture supports training on standard causal language modeling corpora such as: - OpenWebText - BooksCorpus - FineWeb - Wikipedia - Common Crawl derived datasets The released checkpoints may use different datasets depending on the model version. Training data is cleaned using standard preprocessing: - UTF-8 normalization - Deduplication - Document concatenation - Tokenization --- # Evaluation The model is designed to be evaluated using common language modeling benchmarks including: - WikiText - HellaSwag - ARC - PIQA - MMLU - GSM8K - HumanEval (code models) Benchmark results will be released alongside future checkpoints. --- # Hardware Requirements Training: - NVIDIA GPUs - CUDA-enabled PyTorch Inference: CPU: - Supported - Suitable for small checkpoints GPU: - Recommended - Significantly faster generation --- # Known Limitations Wiola is a research model and has several limitations: - May generate incorrect or fabricated information. - Does not possess factual understanding. - Performance depends heavily on training data. - Not intended for safety-critical or medical decision making. - Should always be used with human verification. --- # Intended Use Suitable for: - Language model research - NLP experimentation - Education - Fine-tuning - Benchmarking novel Transformer architectures Not recommended for: - Medical diagnosis - Legal advice - Financial decision making - Autonomous high-risk systems --- # Ethical Considerations Like other large language models, Wiola may inherit biases from training data. Developers should: - Review generated outputs. - Validate factual information. - Monitor for harmful or biased generations. - Apply appropriate safety filters in downstream applications. --- # Novel architecture ```text Input Tokens │ ▼ Token Embeddings │ ▼ ┌─────────────────────────────────────┐ │ Spiral Rotary Encoding │ │ Gated Cross-Layer Attention │ │ Adaptive Token Merging │ │ Dual-Stream FFN │ │ WiolaRMSNorm │ └─────────────────────────────────────┘ │ ▼ Repeated N Layers │ ▼ Final RMSNorm │ ▼ LM Head │ ▼ Generated Tokens ``` --- # Citation If you use Wiola in your research, please cite: ```bibtex @software{wiola2026, title={Wiola: A Novel Decoder-Only Small Language Model}, author={OSCOWL-AI}, year={2026}, url={https://github.com/Wiola-OSCOWL-ai/wiola} } ``` --- # License Apache License 2.0 --- # Links - GitHub: https://github.com/Wiola-OSCOWL-ai/Wiola13M - PyPI: https://pypi.org/project/wiola/ *(coming soon)* - Hugging Face: https://huggingface.co/OSCOWL-AI --- # Acknowledgements Wiola is an open research project developed by **OSCOWL-AI** to explore efficient and scalable Small Language Model architectures. Contributions, feedback, and research collaborations are welcome.
