mlx-DeepDanbooru

Pure MLX implementation of DeepDanbooru Neural Network for Apple Silicon Chips: M1, M2, M3, M4; mlx-DeepDanBooru is available for: MacBook Pro / Air, Mac mini, iMac.

no pytorch needed

Usage

Image-to-Text, captioning, CLIP by using DeepDanBooru Model on Apple Devices.

MLX DeepDanBooru Model

This mlx-DeepDanBooru Model implementation is inspired by a PyTorch implementation of AUTOMATIC1111/TorchDeepDanbooru

Installation

conda create -n mlx026 python=3.12
conda activate mlx026
#
pip install numpy
pip install pillow

MLX is available on PyPI. To install the Python API, run:

pip install mlx

mlx-DeepDanbooru is base on mlx version: 0.26.1

Inference

#
python ./infer.py  /Volumes/HDD4/hazhu/example/1.png
#
# inference folder's all images one-by-one
# python ./infer.py  /Volumes/HDD4/hazhu/example

#
# inference folder's all images parallelly
# python ./infer_multiprocessing.py  /Volumes/HDD4/hazhu/example

Segments Inference

python ./segment_infer.py /Volumes/HDD4/hazhu/example
#

segment_infer.py will crop the image into 3 segments: top, middle, bottom; then inference each segment one-by-one, and merge tags at last.

Cache

segment_infer.py can enable cache, as the same input will always get the same tags, you can cache the results for speedup.

Image Interrogate:

import numpy as np
from PIL import Image

# using apple silicon's MLX 
# not Pytorch
import mlx.core as mx
from mlxDeepDanBooru.mlx_deep_danbooru_model import mlxDeepDanBooruModel

model_path = "models/model-resnet_custom_v3_fp32_mlx.safetensors"
model_tags = np.load('models/tags-resnet_custom_v3_mlx.npy')

mlx_dan = mlxDeepDanBooruModel()
mlx_dan.load_weights(model_path)
mx.eval(mlx_dan.parameters())

def danbooru_tags(fpath):
    tags = []

    try:
        pic = Image.open(fpath).convert("RGB").resize((512, 512))
        a = np.expand_dims(np.array(pic, dtype=np.float32), 0) / 255
        #
        x = mx.array(a)
        y = mlx_dan(x)[0]
        ylen = len(y)
        ylst = y.tolist()
        for i in range(ylen):
            if ylst[i] >= 0.55:
                # 0.55 can be changed for demand: 0.0 ~ 1.0             
                #print(model_tags[i].item(), p)
                tags.append(model_tags[i].item())

    except Exception as err:
        print(err)
        tags = []

    return tags

def image_infer(fpath):
    tags = danbooru_tags(fpath)
    return tags

tags_1 = image_infer("example/1.png")
tags_2 = image_infer("example/2.png")

print(tags_1)
# will show tags: ['1girl', 'beach', 'black_hair', 'blurry', 'blurry_background', 'blurry_foreground', 'building', 'bush', 'christmas_tree', 'day', 'depth_of_field', 'field', 'grass', 'lake', 'looking_at_viewer', 'mountain', 'nature', 'outdoors', 'palm_leaf', 'palm_tree', 'park', 'park_bench', 'path', 'photo_background', 'plant', 'river', 'road', 'skirt', 'sky', 'smile', 'striped', 'striped_dress', 'striped_shirt', 'tree', 'vertical-striped_shirt', 'vertical_stripes', 'rating:safe']

print(tags_2)
# will show tags: ['1girl', '3d', 'blurry', 'blurry_background', 'blurry_foreground', 'brown_eyes', 'brown_hair', 'bush', 'christmas_tree', 'cosplay_photo', 'day', 'depth_of_field', 'field', 'floral_print', 'foliage', 'forest', 'garden', 'grass', 'jungle', 'lips', 'long_hair', 'long_sleeves', 'looking_at_viewer', 'nature', 'on_grass', 'outdoors', 'palm_tree', 'park', 'path', 'plant', 'potted_plant', 'realistic', 'smile', 'solo', 'tree', 'upper_body', 'white_dress', 'rating:safe']

Performance

• 1024x1024 pixel image

In the example folder, on Mac Mini M4, mlx-DeepDanBooru inference Speed (image one-by-one):

SPEED: 0.12 seconds per image

• 3840x5760 pixel image

On Mac mini M4, mlx-DeepDanBooru with multiprocessing(4 parallel tasks), run infer_multiprocessing.py:

SPEED: 0.18 seconds per image 

inference 81 images(3840x5760), speed (seconds per image, shorter is better):

Pytorch + CPU : 0.54 
Pytorch + MPS : 0.34

mlx-DeepDanBooru : 0.18