gitmigration.md

# HuggingFace Spaces Git Migration: LFS → Xet Storage

## Problem

HuggingFace Spaces deprecated Git LFS and now requires **Xet storage** for binary files (images, audio, models, etc.).

When pushing to HF Spaces with LFS-tracked files:
```
remote: Your push was rejected because it contains binary files.
remote: Please use https://huggingface.co/docs/hub/xet to store binary files.
```

**Why:** HF's pre-receive hook scans git history and rejects any commits with binary files routed via LFS. It requires Xet protocol instead.

---

## ⚠️ Important: Problematic Commits to Clean Up

The following commits attempted to add PNG files **before git-xet was properly installed and configured**. These caused the rejection error:

- **`d9520f4`** - "feat: replace Unsplash cover URLs with local assets/covers/ PNG paths"
- **`f423e05`** - "feat: replace Unsplash cover URLs with local assets/covers/ PNG paths" (duplicate)

**Status:** ✅ These commits were cleaned up via `git-filter-repo` and PNG files were restored via proper git-xet workflow.

**For future reference:** Always ensure git-xet is installed and configured **before** committing large binary files. If this happens again:

```bash
# Remove problematic commits from history
git filter-repo --path assets/covers/ --invert-paths --force

# Restore from backup or re-add with git-xet properly configured
# Then push
```

---

## The Solution: Use git-xet

Store large binary files in git using **Xet protocol** instead of deprecated LFS.

**Why git-xet:**
- ✅ Binary files remain in git history (version-controlled)
- ✅ Standard git workflow (no special handling)
- ✅ Automatic deduplication & fast transfers
- ✅ HF officially supports and recommends it
- ✅ Works with existing `.gitattributes` configuration

---

## How We Fixed It (Step-by-Step)

### Step 1: Verify git-xet is Installed
```bash
git xet --version
# Output: git-xet 0.2.1
```

If not installed:
```bash
# macOS/Linux
brew install git-xet
git xet install

# Or download from: https://github.com/huggingface/xet-core/releases
```

### Step 2: Verify .gitattributes Configuration
Your `.gitattributes` should route binary files through LFS (git-xet intercepts):

```
*.png filter=lfs diff=lfs merge=lfs -text
*.jpg filter=lfs diff=lfs merge=lfs -text
*.jpeg filter=lfs diff=lfs merge=lfs -text
*.wav filter=lfs diff=lfs merge=lfs -text
*.mp3 filter=lfs diff=lfs merge=lfs -text
```

With git-xet installed, these files automatically route through **Xet protocol** instead of legacy LFS.

### Step 3: Remove from .gitignore
If binaries are in `.gitignore`, remove them so they can be tracked:

```bash
# Edit .gitignore and remove any lines excluding your binaries
# Example: remove "assets/covers/" line
```

### Step 4: Add and Commit Binaries
```bash
git add assets/covers/
git commit -m "feat: add story cover images via Xet storage"
```

### Step 5: Push to HF Space
```bash
git push steve-hf main
```

**What happens:**
- ✅ Git detects binary files routed via `filter=lfs`
- ✅ git-xet intercepts and uses **Xet protocol** (not legacy LFS)
- ✅ Files upload via chunk-deduplication
- ✅ HF pre-receive hook accepts them (no "binary files rejected" error)

**Verification:**
```bash
# Verify files are tracked in git
git ls-files | grep -i png

# Verify push succeeded
git log --oneline -2
```

---

## Key Takeaways

| Aspect | Before | After |
|--------|--------|-------|
| **Problem** | HF rejected LFS-tracked binary files | ✅ Resolved with git-xet |
| **Git history** | Contained PNG LFS pointers | Properly tracked via Xet |
| **PNG files** | Attempted LFS upload (rejected) | Committed via Xet protocol |
| **Storage backend** | Legacy HF LFS (no longer supported) | HF Xet (official standard) |
| **Push status** | ❌ "binary files rejected" hook error | ✅ Successful push |

---

## Future Workflows

### Adding New Binary Files
Once git-xet is installed and `.gitattributes` is configured, just use normal git:

```bash
git add new_file.png
git commit -m "Add new image"
git push origin main
# Files automatically routed through Xet protocol
```

### Cloning the Repository
With git-xet installed, cloning and pulling works normally:

```bash
git clone git@hf.co:spaces/sshtel/Huggingface_Hack
cd Huggingface_Hack
# Binary files auto-downloaded via Xet deduplication
```

### Updating Existing Binary Files
```bash
# Modify a PNG file
# (e.g., replace cover image)

git add assets/covers/changed_image.png
git commit -m "Update cover image"
git push steve-hf main
# Only changed chunks uploaded via Xet (efficient)
```

---

## Files Changed

- **Added:** `assets/covers/*.png` (10 story cover images, 19 MB total)
  - Tracked in git via Xet protocol
  - Routed through `filter=lfs` in `.gitattributes`
- **Created:** `gitmigration.md` (this file)
- **Kept:** `.gitattributes` (LFS syntax, git-xet intercepts)
- **Cleaned:** `.gitignore` (removed `assets/covers/` exclusion)

---

## References

- [HuggingFace Docs: Xet Storage with Git](https://huggingface.co/docs/hub/xet/using-xet-storage#git)
- [Git Xet Repository](https://github.com/huggingface/xet-core)
- [HF Docs: Upload Files](https://huggingface.co/docs/huggingface_hub/guides/upload)
- [Xet Protocol: Chunk Deduplication](https://huggingface.co/docs/xet/index)

## Troubleshooting

### "Your push was rejected because it contains binary files"
**Solution:** Ensure git-xet is installed and `.gitattributes` is configured:
```bash
git xet --version          # Verify installation
git config --global --list | grep xet   # Verify config
cat .gitattributes         # Verify routing rules
```

### Large file uploads are slow
**Solution:** git-xet has adaptive concurrency. For high-bandwidth machines:
```bash
export HF_XET_HIGH_PERFORMANCE=1
git push
```

### Need to revert git-xet
```bash
git xet uninstall
# Files in repos still work via LFS bridge (slower, but compatible)
```