Charm

--- license: apache-2.0 pipeline_tag: image-feature-extraction tags: - pytorch - aesthetics metrics: - pearsonr - spearmanr - accuracy base_model: - facebook/dinov2-small --- 💫 Official implementation of Charm: The Missing Piece in ViT fine-tuning for Image Aesthetic Assessment > [**Accepted at CVPR 2025**](https://cvpr.thecvf.com/virtual/2025/poster/34423)<br> > [**arXiv**](https://arxiv.org/abs/2504.02522)<br> <div align="left"> <a href="https://github.com/FBehrad/Charm"> <img src="https://github.com/FBehrad/Charm/blob/main/Figures/MainFigure.jpg?raw=true" alt="Overall framework" width="400"/> </a> </div> We introduce **Charm** , a novel tokenization approach that preserves **C**omposition, **H**igh-resolution, **A**spect **R**atio, and **M**ulti-scale information simultaneously. By preserving critical information, <em> Charm </em> works like a charm for image aesthetic and quality assessment 🌟. ### Quick Inference * Step 1) Check our [GitHub Page](https://github.com/FBehrad/Charm/) and install the requirements. ```setup pip install -r requirements.txt ``` ___ * Step 2) Install Charm tokenizer. ```setup pip install Charm-tokenizer ``` ___ * Step 3) Tokenization + Position embedding preparation <div align="center"> <a href="https://github.com/FBehrad/Charm"> <img src="https://github.com/FBehrad/Charm/blob/main/Figures/charm.gif?raw=true" alt="Charm tokenizer" width="700"/> </a> </div> Charm Tokenizer has the following input args: * patch_selection (str): The method for selecting important patches * Options: 'saliency', 'random', 'frequency', 'gradient', 'entropy', 'original'. * training_dataset (str): Used to set the number of ViT input tokens to match a specific training dataset from the paper. * Aesthetic assessment datasets: 'ava', 'aadb', 'tad66k', 'para', 'baid'. * Quality assessment datasets: 'spaq', 'koniq10k'. * backbone (str): The ViT backbone model (default: 'facebook/dinov2-small' **(for all datasets except for AVA)** and 'facebook/dinov2-large' **(Just for AVA)**. * factor (float): The downscaling factor for less important patches (default: 0.5). * scales (int): The number of scales used for multiscale processing (default: 2). * random_crop_size (tuple): Used for the 'original' patch selection strategy (default: (224, 224)). * downscale_shortest_edge (int): Used for the 'original' patch selection strategy (default: 256). * wit...