svn-git-server/README.md

# svn-git-mirror

> **License:** [PolyForm Noncommercial 1.0.0](https://polyformproject.org/licenses/noncommercial/1.0.0)
> Copyright (c) 2026 Uberwald. All rights reserved.
> Commercial use requires a separate license from Uberwald. Send email to contact@uberwald.me

Py sidecar SVN↔Git bidirectional mirror. Uses own mapping DB (not `git svn`) so Git commit hashes never change.

Validated against huge SVN/GIT repos with 70K commits and more than 120 branches

Works with any Git server that exposes bare repos on disk (Gitea, Forgejo,
GitLab, bare `git init --bare`, etc.). The Gitea-specific integration code
(`svn_mirror/gitea.py`, `svn_mirror/webhook.py`) serves as a **working
example** and can be adapted to other Git servers — see
[Adapting to other Git servers](#adapting-to-other-git-servers).

## Architecture

```
┌──────────────┐   SVN poll    ┌───────────────────┐
│   SVN remote │ ◄───────────  │  svn-mirror       │
│  (http/svn+) │ ────────────► │  daemon / sync    │
└──────────────┘               │                   │
                               │  ┌──────────────┐ │
┌──────────────┐   git push    │  │ canonical    │ │
│   Git server │ ◄───────────  │  │ bare repo    │ │
│  (webhook)   │ ────────────► │  └──────────────┘ │
└──────────────┘   git fetch   │                   │
                               │  SQLite mapping   │
                               └───────────────────┘
```

- **Canonical bare repo**: the single source of truth on the Git side
- **Mapping DB**: SQLite table `(svn_revision, svn_branch) ↔ git_commit_hash`
- **SVN WC**: per-branch working copy checked out for applying Git→SVN diffs
- **Daemon**: polls SVN on interval, pushes Git→SVN on webhook or interval

## Requirements

- Python 3.11+
- `git` + `git-svn` (for one-time `git svn clone --no-metadata` during init)
- `svn` CLI (Subversion client)
- A Git server with bare repos accessible on disk (Gitea, Forgejo, GitLab, or plain `git init --bare`)
- UTF-8 system locale (SVN needs it for non-ASCII filenames)

### Install system deps

```bash
# Debian / Ubuntu
apt install python3 python3-pip git git-svn subversion

# RHEL / Rocky / Alma
yum install python3 python3-pip git git-svn subversion

# OpenSUSE
zypper install python311 python311-pip git git-svn subversion glibc-locale
```

### UTF-8 locale

SVN requires a UTF-8 locale to handle filenames with accents/spaces:

```bash
sudo localedef -i en_US -f UTF-8 en_US.UTF-8
export LC_ALL=en_US.UTF-8
```

Add to `~/.bashrc` or `~/.profile` to make it permanent.

## Installation

```bash
# Clone
git clone https://github.com/your-org/svn-git-server /opt/svn-git-server
cd /opt/svn-git-server

# Install Python deps
pip install -r requirements.txt
```

Only dependency: `PyYAML`. The rest is stdlib + CLI tools.

## Configuration

Create `/etc/svn-mirror/config.yml`:

```yaml
data_dir: /var/svn-mirror

mirrors:
  - id: my-project
    enabled: true

    svn:
      url: https://svn.example.com/svn/myproject
      layout: std              # "std" for standard /trunk /branches /tags
      # For non-standard layout:
      # layout: custom
      # trunk: trunk
      # branches: branches
      # tags: tags
      username:                # SVN login (omit if anonymous access)
      password:                # SVN password

    # ── Git server integration ──────────────────────────────────
    # The config below uses "gitea" as the section name for historical
    # reasons. In practice this works with any Git server that exposes
    # bare repos on disk. See "Adapting to other Git servers" below.
    gitea:
      owner: myorg
      repo: myproject
      repos_path: /var/lib/gitea/data/repositories  # path to bare repos on disk
      webhook_secret: "choose-a-random-secret"      # shared HMAC secret
      webhook_host: "0.0.0.0"
      webhook_port: 8080

    authors:
      "john": "John Doe <john@example.com>"
      "jane": "Jane Doe <jane@example.com>"

    sync_interval: 120   # SVN→Git poll interval (seconds)
```

### `repos_path` — finding your Git server's repos on disk

The sidecar accesses your Git server's repos **on disk**, not via HTTP API.
Find the path:

| Setup | Typical path |
|-------|-------------|
| Gitea binary | `/var/lib/gitea/data/repositories` |
| Gitea Docker | `/data/git/repositories` |
| Forgejo | `/var/lib/forgejo/data/repositories` |
| GitLab | `/var/opt/gitlab/git-data/repositories` |
| `git` user home | `/home/git/repositories` |
| Plain bare repos | wherever you ran `git init --bare` |

If unset the tool tries common defaults. Set it explicitly to be safe.

**Note:** `repo` should be the repo name **without** `.git` suffix — the tool appends it.
For example: `repo: myproject`, not `repo: myproject.git`.

## Quick start

### 1. Validate config

```bash
python -m svn_mirror check --config /etc/svn-mirror/config.yml
```

### 2. Create mirror directories

```bash
python -m svn_mirror create --mirror my-project --config /etc/svn-mirror/config.yml
```

Creates `{data_dir}/mirrors/{id}/` with:
- `canonical-repo.git` — bare Git repo
- `mapping.db` — SQLite mapping DB
- `state.json` — mirror state
- `sync.lock` — flock-based concurrency lock

### 3. Initial SVN import

```bash
python -m svn_mirror init --mirror my-project --config /etc/svn-mirror/config.yml
```

Runs `git svn clone --no-metadata` (one-time bootstrap), then tears down
`git-svn` metadata. All ongoing sync is handled by the tool's own engine.

**Auth:** If the SVN server requires login, set `username` and `password`
in the config. The password is piped to `git svn`'s stdin prompt
(`git svn` does not accept `--password` on the command line).

The tool also caches credentials via `svn info --username X --password Y`
before `git svn clone` so the SVN auth cache is populated.

**External definitions:** If the SVN repo uses `svn:externals` pointing to
different repositories, the tool passes `--ignore-externals` to all
`svn checkout` / `svn update` calls.  The externals are not followed.

### 4. Sync SVN → Git

```bash
python -m svn_mirror sync --mirror my-project --config /etc/svn-mirror/config.yml
```

### 5. Push Git → SVN

```bash
python -m svn_mirror push --mirror my-project --config /etc/svn-mirror/config.yml
```

### 6. Check status

```bash
python -m svn_mirror status --mirror my-project --config /etc/svn-mirror/config.yml
```

## Deployment modes

### Mode A: Polling daemon (simple)

```bash
python -m svn_mirror daemon --config /etc/svn-mirror/config.yml
```

**Note:** `daemon` does **not** accept `--mirror` — it processes all enabled
mirrors.  Use `screen` (or `tmux`) to run in background:

```bash
screen -dmS svn-mirror bash -c 'python3.11 -m svn_mirror daemon --config /etc/svn-mirror/config.yml 2>&1 | tee /tmp/daemon.log'
```

Polls SVN every `sync_interval` seconds. Pushes new Git commits to SVN
on the same interval.

#### systemd unit

```ini
# /etc/systemd/system/svn-mirror.service
[Unit]
Description=svn-git-mirror daemon
After=network-online.target

[Service]
Type=simple
ExecStart=/usr/bin/python3 -m svn_mirror daemon --config /etc/svn-mirror/config.yml
Restart=always
User=root
WorkingDirectory=/opt/svn-git-server

[Install]
WantedBy=multi-user.target
```

```bash
systemctl daemon-reload
systemctl enable --now svn-mirror
```

### Mode B: Webhook-only (event-driven)

```bash
python -m svn_mirror webhook --config /etc/svn-mirror/config.yml
```

Listens for push webhooks on `POST /webhook`. When a push arrives
it fetches the new commits from the Git server and runs `sync_git_to_svn`.

**Example — Gitea webhook config:** Go to repo → Settings → Webhooks → Add:
- Target URL: `http://your-server:8080/webhook`
- Secret: same as `webhook_secret` in config
- Events: "Push"

**Example — GitLab webhook config:** Go to repo → Settings → Webhooks:
- URL: `http://your-server:8080/webhook`
- Secret token: same as `webhook_secret` in config
- Trigger: "Push events"

> **Note:** The webhook receiver currently parses the Gitea payload format
> (`X-Gitea-Signature` header, `repository.owner.login` field). To use
> another Git server, adapt the header name and payload parsing in
> `svn_mirror/webhook.py`. See
> [Adapting to other Git servers](#adapting-to-other-git-servers).

### Mode C: systemd service (recommended)

The daemon handles **bidirectional** sync: SVN→Git polling AND Git→SVN push
on each cycle. One service is all you need.

#### Quick install via script

Use the provided install script to generate and enable the systemd service
running as a given user (e.g. `git`):

```bash
sudo ./deploy/install.sh --install-dir /opt/svn-git-server --user git
```

This will:
- Create a Python virtualenv at `<install-dir>/.venv` and install deps
- Generate `svn-mirror.service`
- Enable the service (auto-start on boot)

Options:

| Option | Default | Description |
|--------|---------|-------------|
| `--install-dir DIR` | *(required)* | Where the code is installed |
| `--user USER` | `gitea` | System user to run the service as |
| `--config FILE` | `<install-dir>/config.yml` | Config file path |
| `--venv DIR` | `<install-dir>/.venv` | Virtualenv path |
| `--uninstall` | — | Remove service instead of installing |

> **Note:** The default `--user` is `gitea` for historical reasons. Use
> whatever user owns your Git server's repos (e.g. `git`, `forgejo`,
> `gitlab`).

After install, start the service:

```bash
sudo systemctl start svn-mirror
```

Check logs:

```bash
journalctl -u svn-mirror -f
```

To uninstall:

```bash
sudo ./deploy/install.sh --uninstall
```

#### Manual setup (screen/tmux)

```bash
screen -dmS svn-mirror python3.11 -m svn_mirror daemon --config /etc/svn-mirror/config.yml
```

### Mode D: Post-receive hook (filesystem)

For setups without network access to a webhook receiver, install a
post-receive hook in your Git server's repo that calls sync directly.
**Example — Gitea/Forgejo:**

```bash
# Create hook (one-time setup)
tee /home/gitea/repo/myorg/myproject.git/hooks/post-receive.d/sync-to-svn << 'HOOK'
#!/bin/bash
cd /opt/svn-git-server
. .venv/bin/activate
python3.11 -m svn_mirror sync --mirror my-project --config /etc/svn-mirror/config.yml
HOOK
chmod +x /home/gitea/repo/myorg/myproject.git/hooks/post-receive.d/sync-to-svn
```

Each `git push` to the Git server then triggers an immediate SVN sync.

**Note:** Direct filesystem `git push` to a Git server's on-disk bare repo
may be blocked by the server's `pre-receive` hook (Gitea and GitLab both
do this).  The daemon/sync uses its own `push_to_gitea()` to work around
this by using `git fetch` instead of `git push`.  To force-push (e.g.
initial import), temporarily disable the hook:

```bash
mv /path/to/repo.git/hooks/pre-receive{,.disabled}
# push...
mv /path/to/repo.git/hooks/pre-receive{.disabled,}
```

## Reconciliation (fixing divergence)

If the sync was interrupted (daemon down for days) and commits landed on
both SVN and Git independently, the tool can re-sync them.

```bash
# 1. Assess divergence (dry-run, no changes)
python -m svn_mirror reconcile --assess --mirror my-project --config /etc/svn-mirror/config.yml

# 2. Reconcile (creates SVN commit merging Git changes into SVN)
python -m svn_mirror reconcile --mirror my-project --config /etc/svn-mirror/config.yml

# 3. Normal sync to bring Git in line
python -m svn_mirror sync --mirror my-project --config /etc/svn-mirror/config.yml
```

The reconciliation:
- Backs up the mapping DB
- Applies each Git developer commit (not yet in DB) onto the SVN trunk WC
- Commits to SVN as a single `[reconcile]` revision
- Records the mapping so `sync_git_to_svn` doesn't re-push those commits
- Leaves `last_svn_revision` unchanged — `sync` processes pending SVN revisions naturally

**SVN author fix:** After `svn commit` / `svn copy`, the tool sets the correct
author via `svn propset --revprop -r REV svn:author "Real Author"`.
This requires the `pre-revprop-change` hook on the SVN server:

```bash
# On the SVN server — create this script, make it executable
# /srv/svn/pixkit/hooks/pre-revprop-change
#!/bin/bash
REPOS="$1"; REV="$2"; USER="$3"; PROPNAME="$4"; ACTION="$5"
[ "$PROPNAME" = "svn:author" ] || [ "$PROPNAME" = "svn:log" ] || exit 1
exit 0
```

**Dry-run mode:**

```bash
python -m svn_mirror reconcile --dry-run --mirror my-project --config /etc/svn-mirror/config.yml
```

Shows the assessment report and what would be done, without making any changes.

## All commands

| Command | Description |
|---------|-------------|
| `check` | Validate config and prerequisites |
| `create` | Create mirror directory structure |
| `init` | Run `git svn clone` initial import |
| `destroy` | Delete all mirror data |
| `status` | Show mirror status |
| `sync` | Bidirectional sync (SVN→Git + Git→SVN) |
| `push` | Git→SVN direction only |
| `daemon` | Continuous polling loop |
| `webhook` | Webhook receiver (Gitea format; adaptable) |
| `reconcile` | Fix divergence (see above) |

## File layout

```
{data_dir}/mirrors/{id}/
├── canonical-repo.git/      # bare Git repo (canonical Git copy)
├── mapping.db               # SQLite mapping DB
├── state.json               # mirror metadata
├── authors.txt              # git svn authors file
├── sync.lock                # flock-based concurrency lock
└── svn-wc/                  # SVN working copies (per branch)
```

## Caveats

- **Auth**: SVN `username`/`password` in config is sent to `git svn clone`
  pipe and CLI args.  Password is stored in plaintext in config file.
- **Binary files**: supported (fix in `sync.py` uses raw bytes for `hash-object`)
- **Empty directories**: SVN tracks them, Git does not — skipped during sync
- **Large repos**: initial `git svn clone` time depends on SVN history size;
  6+ GiB repos can take several hours over HTTPS
- **One trunk only**: reconciliation only handles `refs/heads/master` ↔ SVN trunk.
  Other branches are not yet supported by the reconcile tool.
- **`git svn` username**: `--username` is supported but `--password` is not
  (the tool pipes password to stdin instead).
- **All SVN branches become Git branches**:  If the SVN repo uses branches for
  release tags, the Git canonical repo will contain hundreds of branches.
  Consider filtering via `refs/tags/` in the push refspec.
- **Git server filesystem push**: Direct `git push` to a Git server's
  on-disk bare repo may be blocked by the server's `pre-receive` hook
  (Gitea, GitLab).  The daemon/sync uses its own `push_to_gitea()` to
  work around this by using `git fetch` instead.

## Adapting to other Git servers

The core SVN↔Git sync engine is server-agnostic.  The Gitea-specific
code is limited to two files and can be adapted to other Git servers
(Forgejo, GitLab, Gitea, plain bare repos, etc.):

| What | Where | What to change |
|-------|-------|----------------|
| Push to Git server | `svn_mirror/gitea.py` — `push_to_gitea()` | Uses `git fetch` into the bare repo (generic). The `_trigger_gitea_post_receive()` function calls the `gitea` binary — replace with your server's hook trigger or remove it. |
| Webhook receiver | `svn_mirror/webhook.py` | Parses `X-Gitea-Signature` header and Gitea JSON payload. Adapt the header name (`X-Gitlab-Token`, etc.) and payload fields (`repository.owner.login` → your server's equivalent). |
| Config section | `svn_mirror/config.py` — `GiteaConfig` | The `gitea:` config block. The fields (`owner`, `repo`, `repos_path`, `webhook_*`) are generic; only the section name is Gitea-specific. |
| Default repo paths | `svn_mirror/config.py` — `repo_dir` | Hardcoded Gitea defaults. Add your server's path or set `repos_path` explicitly. |

For **plain bare repos** (no Git server), no adaptation is needed — just
set `repos_path` to the directory containing your bare repos and skip
the webhook/post-receive integration (use the daemon or manual `sync`).