Aligndraw

--- license: mit tags: - text-to-image - draw - vae - attention - historical library_name: pytorch pipeline_tag: text-to-image --- # AlignDRAW (resurrected) — one of the first text-to-image models A trained-from-scratch **PyTorch resurrection** of **AlignDRAW** (Mansimov, Parisotto, Ba, Salakhutdinov — *Generating Images from Captions with Attention*, ICLR 2016), one of the earliest neural text-to-image models. The original [Theano code](https://github.com/mansimov/text2image) shipped **no pretrained weights** and the framework is long dead — so this model was re-implemented in PyTorch and **trained on the original still-online COCO 32×32 caption data** (`cs.toronto.edu/~emansim`), for [`mediasynthesismuseum/aligndraw`](https://huggingface.co/spaces/mediasynthesismuseum/aligndraw). It's a recurrent VAE that **draws an image onto a canvas over 32 steps**, attending to the caption words at each step (Bahdanau alignment) with a learned sequential prior. Output is a blurry **32×32** blob whose colour/layout tracks the caption — exactly the fidelity the 2015 model had. ## Files - `aligndraw_coco.pt` — trained weights (33M params; dims match the original `coco-captions-32x32`). - `model.py` — the faithful PyTorch `AlignDRAW` definition. - `dictionary.pkl` — the original 25,322-word caption dictionary (word → index; includes `UNK`). ## Usage ```python import torch, pickle from model import AlignDRAW m = AlignDRAW(); m.load_state_dict(torch.load("aligndraw_coco.pt", map_location="cpu")); m.eval() d = pickle.load(open("dictionary.pkl","rb"), encoding="latin1") toks = [d.get(w, d["UNK"]) for w in "a red stop sign on a street corner .".split()] y = torch.tensor([toks]).expand(8, -1) imgs = m.generate(y) # (8, 3, 32, 32) in [0,1] ``` Credit: Elman Mansimov, Emilio Parisotto, Jimmy Ba, Ruslan Salakhutdinov (original AlignDRAW); built on DRAW (Gregor et al.).

Source context: 0 downloads · 0 likes · Pipeline text-to-image · Library pytorch · Repo mediasynthesismuseum/aligndraw