Zum Inhalt springen
ModelAufgelistetExtern
Herunterladen

Qwen3 VL 1.0B A0.4B Instruct

This is a tiny version of Qwen/Qwen3-VL-30B-A3B-Instruct created for testing and development.

Quellehuggingface.co/inference-optimization/Qwen3-VL-1.0B-A0.4B-Instruct/tree/bf4f6e5fd93806e772988614075634976e3eda38
ArtText modelBasismodellQwen/Qwen3-VL-30B-A3B-InstructVersionBF16BBewertung

Modellquelle

Art
Text model
Dateigröße
2,67 GB
Basismodell
Qwen/Qwen3-VL-30B-A3B-Instruct
Version
vBF16
Aktualisiert
30. Okt. 2026
Downloads
3264
Quelle
Hugging Face

Quellenbeschreibung

Qwen3-VL-1.0B-A0.4B-Instruct

This is a tiny version of Qwen/Qwen3-VL-30B-A3B-Instruct created for testing and development.

Model Details

  • Base Model: Qwen/Qwen3-VL-30B-A3B-Instruct
  • Architecture: qwen3_vl_moe
  • Total Parameters: 1.434B
  • Activated Parameters: ~0.4B (8 of 128 experts active per token)

Configuration Changes

The following parameters were reduced from the original model:

ParameterOriginalTiny
text_config.num_hidden_layers481
vision_config.depth274
vision_config.deepstack_visual_indexes[8, 16, 24][1, 2, 3]
text_config.hidden_size20482048 (unchanged)
text_config.num_local_experts128128 (unchanged)
text_config.num_experts_per_tok88 (unchanged)
text_config.moe_intermediate_size768768 (unchanged)
text_config.num_attention_heads3232 (unchanged)
text_config.num_key_value_heads44 (unchanged)
vision_config.hidden_size11521152 (unchanged)

Checkpoint Structure

The model is saved as a single model.safetensors file (89 tensors). The checkpoint key structure matches the original Qwen/Qwen3-VL-30B-A3B-Instruct exactly.

Usage

from transformers import Qwen3VLMoeForConditionalGeneration, AutoTokenizer

model = Qwen3VLMoeForConditionalGeneration.from_pretrained(
    "inference-optimization/Qwen3-VL-1.0B-A0.4B-Instruct",
    device_map="auto",
)
tokenizer = AutoTokenizer.from_pretrained("inference-optimization/Qwen3-VL-1.0B-A0.4B-Instruct")

input_ids = tokenizer("According to all known laws", return_tensors="pt").input_ids.to(model.device)
output = model.generate(input_ids, max_new_tokens=20)
print(tokenizer.decode(output[0]))

Note: AutoModelForCausalLM does not support this model type. Use Qwen3VLMoeForConditionalGeneration or AutoModelForImageTextToText directly.

Creation Process

This model was created using the llm-compressor create-tiny-model claude skill.

  1. Config fetched from Qwen/Qwen3-VL-30B-A3B-Instruct without downloading weights (skip_weights_download)
  2. Text depth reduced from 48 → 1 layers; vision depth reduced from 27 → 4 blocks
  3. All parameters randomly initialized with initializer_range=0.02 (matching the base model)
  4. MoE expert parameters (gate_up_proj, down_proj, gate.weight) explicitly initialized since they are bare nn.Parameter tensors, not nn.Linear modules
  5. Fine-tuned on a toy text dataset (internet copypastas) until perplexity ≤ 3.0

Validation

Success: perplexity=1.0001 <= 10.0
Generated: According to all known laws of aviation, there is no way a bee should be able to fly. Its wings are too small to get its fat little

Notes

  • This is a text-only fine-tuned model. The vision encoder weights are randomly initialized and not fine-tuned; the model is intended for text-based testing only.
  • The MoE routing architecture (128 experts, 8 active per token) is preserved in full to match the original model's structural properties.
  • Fine-tuning converged in ~160 steps with learning rate 1e-4.

Quellen

1 Quelle
Hugging FacePrimärVersion bf4f6e5fd93806e772988614075634976e3eda38
huggingface.co/inference-optimization/Qwen3-VL-1.0B-A0.4B-Instruct/tree/bf4f6e5fd93806e772988614075634976e3eda38

Verifiziert 30. Okt.

Download files

Repository files

File inventory only; completeness is unverified.

Modellartefakte

1 Artefakt

model.safetensors

safetensors · BF16 (tensor header) · 2,67 GB · SHA-256 ca5c0650dd8c…21d6 · Hugging Face

Download file

Quellenauszüge

3 Auszüge
Readme14. Aug.huggingface.co/inference-optimization/Qwen3-VL-1.0B-A0.4B-Instruct

--- license: mit base_model: - Qwen/Qwen3-VL-30B-A3B-Instruct library_name: transformers --- # Qwen3-VL-1.0B-A0.4B-Instruct This is a tiny version of [Qwen/Qwen3-VL-30B-A3B-Instruct](https://huggingface.co/Qwen/Qwen3-VL-30B-A3B-Instruct) created for testing and development. ## Model Details - **Base Model**: Qwen/Qwen3-VL-30B-A3B-Instruct - **Architecture**: qwen3_vl_moe - **Total Parameters**: 1.434B - **Activated Parameters**: ~0.4B (8 of 128 experts active per token) ## Configuration Changes The following parameters were reduced from the original model: | Parameter | Original | Tiny | |-----------|----------|------| | `text_config.num_hidden_layers` | 48 | 1 | | `vision_config.depth` | 27 | 4 | | `vision_config.deepstack_visual_indexes` | [8, 16, 24] | [1, 2, 3] | | `text_config.hidden_size` | 2048 | 2048 (unchanged) | | `text_config.num_local_experts` | 128 | 128 (unchanged) | | `text_config.num_experts_per_tok` | 8 | 8 (unchanged) | | `text_config.moe_intermediate_size` | 768 | 768 (unchanged) | | `text_config.num_attention_heads` | 32 | 32 (unchanged) | | `text_config.num_key_value_heads` | 4 | 4 (unchanged) | | `vision_config.hidden_size` | 1152 | 1152 (unchanged) | ## Checkpoint Structure The model is saved as a single `model.safetensors` file (89 tensors). The checkpoint key structure matches the original `Qwen/Qwen3-VL-30B-A3B-Instruct` exactly. ## Usage ```python from transformers import Qwen3VLMoeForConditionalGeneration, AutoTokenizer model = Qwen3VLMoeForConditionalGeneration.from_pretrained( "inference-optimization/Qwen3-VL-1.0B-A0.4B-Instruct", device_map="auto", ) tokenizer = AutoTokenizer.from_pretrained("inference-optimization/Qwen3-VL-1.0B-A0.4B-Instruct") input_ids = tokenizer("According to all known laws", return_tensors="pt").input_ids.to(model.device) output = model.generate(input_ids, max_new_tokens=20) print(tokenizer.decode(output[0])) ``` Note: `AutoModelForCausalLM` does not support this model type. Use `Qwen3VLMoeForConditionalGeneration` or `AutoModelForImageTextToText` directly. ## Creation Process This model was created using the llm-compressor `create-tiny-model` claude skill. 1. Config fetched from `Qwen/Qwen3-VL-30B-A3B-Instruct` without downloading weights (`skip_weights_download`) 2. Text depth reduced from 48 → 1 layers; vision depth reduced from 27 → 4 blocks 3. All parameters randomly initialized...

Note30. Okt.huggingface.co/inference-optimization/Qwen3-VL-1.0B-A0.4B-Instruct

Source context: 3264 downloads · 1 likes · Pipeline image-text-to-text · Library transformers · Repo inference-optimization/Qwen3-VL-1.0B-A0.4B-Instruct

Note14. Aug.huggingface.co/inference-optimization/Qwen3-VL-1.0B-A0.4B-Instruct

Source context: 295 downloads · 1 likes · Pipeline image-text-to-text · Library transformers · Repo inference-optimization/Qwen3-VL-1.0B-A0.4B-Instruct