TokenFD 4096 English Seg

--- license: mit pipeline_tag: image-feature-extraction base_model: TokenFD base_model_relation: finetune --- <center> <h1 style="color: black;">A Token-level Text Image Foundation Model for Document Understanding</h1> [\[📂 GitHub\]](https://github.com/Token-family/TokenFD) [\[📖 Paper\]](https://arxiv.org/pdf/2503.02304) [\[🆕 Project_page\]](https://token-family.github.io/project_page/) [\[🤗 HF Demo\]](https://huggingface.co/spaces/TongkunGuan/TokenFD) [\[🚀 Quick Start\]](#quick-start) </center> <!-- <div align="center"> <img width="500" alt="image" src="https://cdn-uploads.huggingface.co/production/uploads/650d4a36cbd0c7d550d3b41b/dQ_JfK_I91WXzIq52D015.png"> </div> --> <center> <!-- # Introduction --> <h2 style="color: #4CAF50;">Introduction</h2> </center> We are excited to announce the release of **`TokenFD`**, the first token-level visual foundation model specifically tailored for text-image-related tasks, designed to support a variety of traditional downstream applications. To facilitate the pretraining of TokenFD, we also devise a high-quality data production pipeline that constructs the first token-level image text dataset, **`TokenIT`**, comprising 20 million images and 1.8 billion token-mask pairs. Furthermore, leveraging this foundation with exceptional image-as-text capability, we seamlessly replace previous VFMs with TokenFD to construct a document-level MLLM, **`TokenVL`**, for VQA-based document understanding tasks. <center> <!-- # Token Family --> <h2 style="color: #4CAF50;">Token Family</h2> </center> <!-- ## TokenIT --> <h2 style="color: #4CAF50;">TokenIT</h2> In the following picture, we provide an overview of the self-constructed token-level **TokenIT** dataset, comprising 20 million images and 1.8 billion text-mask pairs. As depicted in Figure 2 (a), each sample in this dataset includes a raw image, a mask image, and a JSON file. The JSON file provides the question-answer pairs and several BPE tokens randomly selected from the answer, along with the ordinal number of each BPE token in the answer and its corresponding pixel value on the mask image. Consequently, **each BPE token corresponds one-to-one with a pixel-level mask**. The data ratios are summarized in Figure 2 (b). Figure 2 (c) and (d) further provide the number distribution of tokens per image type and a word cloud of the top 100 tokens, respectively. <di...