File size: 4,691 Bytes
fa7b380
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
## Goal

For every image included in the downloaded ZIP, also add:

- `<same-basename>.txt` β€” UTF-8 tags file (space-separated, one line, matches the image filename exactly except for extension).
- `<post-id>.md` β€” human-readable markdown containing all extracted post metadata (all tags, source link β€” blank if missing, file size, sample link, original file link, rating, score, dimensions, uploader, created date, md5, etc.).

## Approach

The server currently only receives `{id, fileUrl, ext}` per post. To write rich metadata into the ZIP without a second network round-trip per post, the frontend will forward the full normalized post (plus a raw metadata bag) to `/api/download-zip`, and the server will emit the extra files alongside each image.

To capture "any data related to that post", `searchPosts` will additionally return the raw JSON entry for each post (as `raw: unknown`). The Markdown generator will pretty-print known fields first, then append the full raw JSON as a fenced code block so nothing is lost.

New optional fields (`sampleUrl`, `fileSize`, `width`, `height`, `md5`, `uploader`, `createdAt`) get added to `SiteConfig.fields` as optional dotted paths, defaulted for the built-in e621 site. Existing custom sites keep working because every new field is optional.

## Files to change

### `src/lib/grabber/sites.ts`

- Extend `FieldMap` with optional: `sampleUrl?`, `fileSize?`, `width?`, `height?`, `md5?`, `uploader?`, `createdAt?`.
- Extend `NormalizedPost` with the same optional fields plus `raw: unknown`.
- Update `normalizePost()` to resolve each new path when present and attach `raw` (the original entry).
- Update `BUILTIN_SITES` e621 entry `fields` with: `sampleUrl: "sample.url"`, `fileSize: "file.size"`, `width: "file.width"`, `height: "file.height"`, `md5: "file.md5"`, `uploader: "uploader_id"`, `createdAt: "created_at"`.

### `src/lib/grabber/search.functions.ts`

- Extend the `SiteSchema.fields` zod object with the new optional string paths.
- No handler logic changes needed β€” normalization already carries `raw` and new fields through.

### `src/components/grabber/AddSiteDialog.tsx`

- Add the new optional field keys to the `(["id",...] as const)` list on line 73 so users can map them for custom sites (`sampleUrl`, `fileSize`, `width`, `height`, `md5`, `uploader`, `createdAt`).
- Extend `DEFAULT` (lines 22–31) with sensible Danbooru mappings (`sampleUrl: "large_file_url"`, `fileSize: "file_size"`, `width: "image_width"`, `height: "image_height"`, `md5: "md5"`, `uploader: "uploader_name"`, `createdAt: "created_at"`).

### `src/routes/api/download-zip.ts` (major rewrite)

- Expand `BodySchema.posts` item to accept the full normalized post shape (all new optional fields + `tags: string[]` + `raw: unknown`).
- For each post:
  1. Fetch the image bytes as today. Build image filename `base = "<siteName>_<id>"` and `<base>.<ext>`.
  2. Write `<base>.txt` β€” `strToU8(tags.join(" "))` (UTF-8). Uses the exact same basename as the image so tag files pair 1:1.
  3. Write `<id>.md` β€” pretty markdown with:
     - `# Post <id>` heading
     - Table/list of: Source (blank line if missing), Rating, Score, Dimensions (`width x height`), File size (human-readable KB/MB), MD5, Uploader, Created at, Original file URL, Sample URL, Preview URL, Ext.
     - `## Tags` section with the tags space-separated.
     - `## Raw metadata` fenced ```json block containing `JSON.stringify(raw, null, 2)`.
- Keep the existing `_errors.txt` behavior and 200-post cap.
- Sanitizer stays the same; md/txt names use `sanitize(id)` so filenames match the image exactly.

### `src/routes/index.tsx`

- In `downloadZip()` (the `fetch("/api/download-zip", …)` call), send the full post objects instead of a stripped `{id,fileUrl,ext}` triple. Change:
  ```
  posts: chosen.map((p) => ({ id: p.id, fileUrl: p.fileUrl, ext: p.ext })),
  ```
  to:
  ```
  posts: chosen, // full NormalizedPost incl. tags + raw + optional metadata
  ```
- No other UI changes.

## ZIP layout example

```text
grabber-2026-07-17T...zip
β”œβ”€β”€ e621_1234567.jpg
β”œβ”€β”€ e621_1234567.txt          ← "tag1 tag2 tag3 ..."
β”œβ”€β”€ 1234567.md                ← human-readable metadata + raw JSON
β”œβ”€β”€ e621_1234568.png
β”œβ”€β”€ e621_1234568.txt
β”œβ”€β”€ 1234568.md
└── _errors.txt               (only if any fetch failed)
```

## Notes / non-goals

- No new dependencies (still `fflate` + `zod`).
- No DB, no auth, no schema changes β€” this is presentation + payload shape only.
- Custom sites without the new mappings still work; missing fields render as blank lines in the `.md` and the raw JSON section still contains everything from the API.