feat: initial import — SVN↔Git bi-directional mirror for Gitea
Server-side sidecar daemon. Uses own SQLite mapping table (not git svn) so Git commit hashes never change. Pure Python, zero framework deps. Includes: - SVN→Git sync engine with nested-directory tree reconstruction - Git→SVN push with author mapping and working-copy management - Branch/tag sync (svn copy → git branch/tag and reverse) - Gitea integration (on-disk push/fetch, webhook receiver) - Reconciliation tool for divergence recovery (assess + auto-fix) - Full end-to-end regression test (bash, 34 checks) - systemd unit and deployment docs Only external dependency: PyYAML (config parsing).
This commit is contained in:
@@ -0,0 +1,273 @@
|
||||
# gitea-svn-mirror
|
||||
|
||||
Py sidecar SVN↔Git bidirectional mirror for Gitea. Uses own mapping DB (not `git svn`) so Git commit hashes never change.
|
||||
|
||||
## Architecture
|
||||
|
||||
```
|
||||
┌──────────────┐ SVN poll ┌──────────────────┐
|
||||
│ SVN remote │ ◄─────────── │ svn-mirror │
|
||||
│ (http/svn+) │ ────────────► │ daemon / sync │
|
||||
└──────────────┘ │ │
|
||||
│ ┌──────────────┐ │
|
||||
┌──────────────┐ git push │ │ canonical │ │
|
||||
│ Gitea │ ◄─────────── │ │ bare repo │ │
|
||||
│ (webhook) │ ────────────► │ └──────────────┘ │
|
||||
└──────────────┘ git fetch │ │
|
||||
│ SQLite mapping │
|
||||
└──────────────────┘
|
||||
```
|
||||
|
||||
- **Canonical bare repo**: the single source of truth on the Git side
|
||||
- **Mapping DB**: SQLite table `(svn_revision, svn_branch) ↔ git_commit_hash`
|
||||
- **SVN WC**: per-branch working copy checked out for applying Git→SVN diffs
|
||||
- **Daemon**: polls SVN on interval, pushes Git→SVN on webhook or interval
|
||||
|
||||
## Requirements
|
||||
|
||||
- Python 3.11+
|
||||
- `git` + `git-svn` (for one-time `git svn clone --no-metadata` during init)
|
||||
- `svn` CLI (Subversion client)
|
||||
- Gitea instance (bare repos accessible on disk, not just via HTTP API)
|
||||
|
||||
### Install system deps
|
||||
|
||||
```bash
|
||||
# Debian / Ubuntu
|
||||
apt install python3 python3-pip git git-svn subversion
|
||||
|
||||
# RHEL / Rocky / Alma
|
||||
yum install python3 python3-pip git git-svn subversion
|
||||
```
|
||||
|
||||
## Installation
|
||||
|
||||
```bash
|
||||
# Clone
|
||||
git clone https://github.com/your-org/svn-git-server /opt/svn-git-server
|
||||
cd /opt/svn-git-server
|
||||
|
||||
# Install Python deps
|
||||
pip install -r requirements.txt
|
||||
```
|
||||
|
||||
Only dependency: `PyYAML`. The rest is stdlib + CLI tools.
|
||||
|
||||
## Configuration
|
||||
|
||||
Create `/etc/svn-mirror/config.yml`:
|
||||
|
||||
```yaml
|
||||
data_dir: /var/svn-mirror
|
||||
|
||||
mirrors:
|
||||
- id: my-project
|
||||
enabled: true
|
||||
|
||||
svn:
|
||||
url: https://svn.example.com/svn/myproject
|
||||
layout: std # "std" for standard /trunk /branches /tags
|
||||
# For non-standard layout:
|
||||
# layout: custom
|
||||
# trunk: trunk
|
||||
# branches: branches
|
||||
# tags: tags
|
||||
|
||||
gitea:
|
||||
owner: myorg
|
||||
repo: myproject
|
||||
repos_path: /var/lib/gitea/data/repositories # path to Gitea's bare repos
|
||||
webhook_secret: "choose-a-random-secret" # shared HMAC secret
|
||||
webhook_host: "0.0.0.0"
|
||||
webhook_port: 8080
|
||||
|
||||
authors:
|
||||
"john": "John Doe <john@example.com>"
|
||||
"jane": "Jane Doe <jane@example.com>"
|
||||
|
||||
sync_interval: 120 # SVN→Git poll interval (seconds)
|
||||
```
|
||||
|
||||
### `repos_path` — finding Gitea's repos on disk
|
||||
|
||||
The sidecar accesses Gitea's repos **on disk**, not via HTTP API. Find the path:
|
||||
|
||||
| Setup | Typical path |
|
||||
|-------|-------------|
|
||||
| Gitea binary | `/var/lib/gitea/data/repositories` |
|
||||
| Gitea Docker | `/data/git/repositories` |
|
||||
| `git` user home | `/home/git/repositories` |
|
||||
|
||||
If unset the tool tries common defaults. Set it explicitly to be safe.
|
||||
|
||||
## Quick start
|
||||
|
||||
### 1. Validate config
|
||||
|
||||
```bash
|
||||
python -m svn_mirror check --config /etc/svn-mirror/config.yml
|
||||
```
|
||||
|
||||
### 2. Create mirror directories
|
||||
|
||||
```bash
|
||||
python -m svn_mirror create --mirror my-project --config /etc/svn-mirror/config.yml
|
||||
```
|
||||
|
||||
Creates `{data_dir}/mirrors/{id}/` with:
|
||||
- `canonical-repo.git` — bare Git repo
|
||||
- `mapping.db` — SQLite mapping DB
|
||||
- `state.json` — mirror state
|
||||
- `sync.lock` — flock-based concurrency lock
|
||||
|
||||
### 3. Initial SVN import
|
||||
|
||||
```bash
|
||||
python -m svn_mirror init --mirror my-project --config /etc/svn-mirror/config.yml
|
||||
```
|
||||
|
||||
Runs `git svn clone --no-metadata` (one-time bootstrap), then tears down
|
||||
`git-svn` metadata. All ongoing sync is handled by the tool's own engine.
|
||||
|
||||
### 4. Sync SVN → Git
|
||||
|
||||
```bash
|
||||
python -m svn_mirror sync --mirror my-project --config /etc/svn-mirror/config.yml
|
||||
```
|
||||
|
||||
### 5. Push Git → SVN
|
||||
|
||||
```bash
|
||||
python -m svn_mirror push --mirror my-project --config /etc/svn-mirror/config.yml
|
||||
```
|
||||
|
||||
### 6. Check status
|
||||
|
||||
```bash
|
||||
python -m svn_mirror status --mirror my-project --config /etc/svn-mirror/config.yml
|
||||
```
|
||||
|
||||
## Deployment modes
|
||||
|
||||
### Mode A: Polling daemon (simple)
|
||||
|
||||
```bash
|
||||
python -m svn_mirror daemon --config /etc/svn-mirror/config.yml
|
||||
```
|
||||
|
||||
Polls SVN every `sync_interval` seconds. Pushes new Git commits to SVN
|
||||
on the same interval.
|
||||
|
||||
#### systemd unit
|
||||
|
||||
```ini
|
||||
# /etc/systemd/system/svn-mirror.service
|
||||
[Unit]
|
||||
Description=gitea-svn-mirror daemon
|
||||
After=network-online.target
|
||||
|
||||
[Service]
|
||||
Type=simple
|
||||
ExecStart=/usr/bin/python3 -m svn_mirror daemon --config /etc/svn-mirror/config.yml
|
||||
Restart=always
|
||||
User=root
|
||||
WorkingDirectory=/opt/svn-git-server
|
||||
|
||||
[Install]
|
||||
WantedBy=multi-user.target
|
||||
```
|
||||
|
||||
```bash
|
||||
systemctl daemon-reload
|
||||
systemctl enable --now svn-mirror
|
||||
```
|
||||
|
||||
### Mode B: Webhook-only (event-driven)
|
||||
|
||||
```bash
|
||||
python -m svn_mirror webhook --config /etc/svn-mirror/config.yml
|
||||
```
|
||||
|
||||
Listens for Gitea push webhooks on `POST /webhook`. When a push arrives
|
||||
it fetches the new commits from Gitea and runs `sync_git_to_svn`.
|
||||
|
||||
**Gitea webhook config:** Go to repo → Settings → Webhooks → Add:
|
||||
- Target URL: `http://your-server:8080/webhook`
|
||||
- Secret: same as `webhook_secret` in config
|
||||
- Events: "Push"
|
||||
|
||||
### Mode C: Both (recommended)
|
||||
|
||||
Run daemon AND webhook. The daemon handles SVN→Git polling; the webhook
|
||||
handles instant Git→SVN on developer push.
|
||||
|
||||
```bash
|
||||
python -m svn_mirror daemon --config /etc/svn-mirror/config.yml &
|
||||
python -m svn_mirror webhook --config /etc/svn-mirror/config.yml &
|
||||
```
|
||||
|
||||
## Reconciliation (fixing divergence)
|
||||
|
||||
If the sync was interrupted (daemon down for days) and commits landed on
|
||||
both SVN and Git independently, the tool can re-sync them.
|
||||
|
||||
```bash
|
||||
# 1. Assess divergence (dry-run, no changes)
|
||||
python -m svn_mirror reconcile --assess --mirror my-project --config /etc/svn-mirror/config.yml
|
||||
|
||||
# 2. Reconcile (creates SVN commit merging Git changes into SVN)
|
||||
python -m svn_mirror reconcile --mirror my-project --config /etc/svn-mirror/config.yml
|
||||
|
||||
# 3. Normal sync to bring Git in line
|
||||
python -m svn_mirror sync --mirror my-project --config /etc/svn-mirror/config.yml
|
||||
```
|
||||
|
||||
The reconciliation:
|
||||
- Backs up the mapping DB
|
||||
- Applies each Git developer commit (not yet in DB) onto the SVN trunk WC
|
||||
- Commits to SVN as a single `[reconcile]` revision
|
||||
- Records the mapping so `sync_git_to_svn` doesn't re-push those commits
|
||||
- Leaves `last_svn_revision` unchanged — `sync` processes pending SVN revisions naturally
|
||||
|
||||
**Dry-run mode:**
|
||||
|
||||
```bash
|
||||
python -m svn_mirror reconcile --dry-run --mirror my-project --config /etc/svn-mirror/config.yml
|
||||
```
|
||||
|
||||
Shows the assessment report and what would be done, without making any changes.
|
||||
|
||||
## All commands
|
||||
|
||||
| Command | Description |
|
||||
|---------|-------------|
|
||||
| `check` | Validate config and prerequisites |
|
||||
| `create` | Create mirror directory structure |
|
||||
| `init` | Run `git svn clone` initial import |
|
||||
| `destroy` | Delete all mirror data |
|
||||
| `status` | Show mirror status |
|
||||
| `sync` | Bidirectional sync (SVN→Git + Git→SVN) |
|
||||
| `push` | Git→SVN direction only |
|
||||
| `daemon` | Continuous polling loop |
|
||||
| `webhook` | Gitea webhook receiver |
|
||||
| `reconcile` | Fix divergence (see above) |
|
||||
|
||||
## File layout
|
||||
|
||||
```
|
||||
{data_dir}/mirrors/{id}/
|
||||
├── canonical-repo.git/ # bare Git repo (canonical Git copy)
|
||||
├── mapping.db # SQLite mapping DB
|
||||
├── state.json # mirror metadata
|
||||
├── authors.txt # git svn authors file
|
||||
├── sync.lock # flock-based concurrency lock
|
||||
└── svn-wc/ # SVN working copies (per branch)
|
||||
```
|
||||
|
||||
## Caveats
|
||||
|
||||
- **Binary files**: supported (fix in `sync.py` uses raw bytes for `hash-object`)
|
||||
- **Empty directories**: SVN tracks them, Git does not — skipped during sync
|
||||
- **Large repos**: initial `git svn clone` time depends on SVN history size
|
||||
- **One trunk only**: reconciliation only handles `refs/heads/master` ↔ SVN trunk.
|
||||
Other branches are not yet supported by the reconcile tool.
|
||||
Reference in New Issue
Block a user