Gallus_Pub/backend/SQLITE_MIGRATION.md

# SQLite Migration Summary

## Changes Made

The backend has been migrated from PostgreSQL to SQLite for both local development and production (Fly.io).

### Benefits of SQLite

1. **Simplified Deployment** - No separate database service needed
2. **Lower Cost** - Save ~$15/month (no Postgres hosting)
3. **Easier Development** - No need to install/run PostgreSQL locally
4. **Single File Database** - Easy backups and migrations
5. **Perfect for this use case** - Low concurrent writes, simple queries

## Modified Files

### Dependencies
- **package.json**
  - Removed: `pg`, `@types/pg`
  - Added: `better-sqlite3`, `@types/better-sqlite3`

### Database Configuration
- **src/config/database.ts**
  - Changed from `drizzle-orm/node-postgres` to `drizzle-orm/better-sqlite3`
  - Uses `DATABASE_PATH` instead of `DATABASE_URL`
  - Enabled WAL mode for better concurrent access

- **src/config/env.ts**
  - Changed `DATABASE_URL` to `DATABASE_PATH`
  - Default: `./data/gallus_cms.db`

- **src/db/schema.ts**
  - Changed from `pgTable` to `sqliteTable`
  - Changed `uuid()` to `text()` with `crypto.randomUUID()`
  - Changed `jsonb()` to `text(..., { mode: 'json' })`
  - Changed `timestamp()` to `integer(..., { mode: 'timestamp' })`
  - Changed `boolean()` to `integer(..., { mode: 'boolean' })`
  - Uses `sql\`(unixepoch())\`` for default timestamps

- **drizzle.config.ts**
  - Changed dialect from `postgresql` to `sqlite`
  - Uses `DATABASE_PATH` instead of `DATABASE_URL`

### Environment Files
- **.env** and **.env.example**
  - Changed `DATABASE_URL=postgresql://...` to `DATABASE_PATH=./data/gallus_cms.db`
  - Changed `GIT_WORKSPACE_DIR=/tmp/gallus-repo` to `./data/workspace`

### Docker Configuration
- **Dockerfile**
  - Added build tools for `better-sqlite3` native module (python3, make, g++)
  - Added `sqlite` CLI tool
  - Creates `/app/data` directory for database
  - Sets `DATABASE_PATH=/app/data/gallus_cms.db`
  - Proper permissions for non-root user

- **fly.toml**
  - Added `DATABASE_PATH` and `GIT_WORKSPACE_DIR` to [env]
  - Changed volume mount from `gallus_repo_workspace` to `gallus_data`
  - Mount destination: `/app/data` (contains both DB and git workspace)

### Documentation
- **README.md** - Updated setup instructions
- **DEPLOYMENT.md** - Removed Postgres setup, updated volume creation
- **SQLITE_MIGRATION.md** - This file!

## Local Development

### Setup
```bash
# Dependencies already installed
pnpm install

# Create data directory (done)
mkdir -p data

# Database will be created automatically at ./data/gallus_cms.db
```

### Generate and Run Migrations
```bash
# Generate migration files from schema
pnpm run db:generate

# Run migrations to create tables
pnpm run db:migrate
```

### Start Development Server
```bash
pnpm run dev
```

The database file will be created at `./data/gallus_cms.db` on first run.

## Production (Fly.io)

### Volume Setup
```bash
# Create single volume for both database and git workspace
flyctl volumes create gallus_data --size 2 --region ams
```

### Environment Variables
Set in fly.toml (non-sensitive):
- `DATABASE_PATH=/app/data/gallus_cms.db`
- `GIT_WORKSPACE_DIR=/app/data/workspace`

Set as secrets (sensitive):
- All other env vars (OAuth credentials, tokens, etc.)

### Deployment
```bash
flyctl deploy
```

Database will be created automatically on first start. No need for separate database service!

## Database Location

### Local Development
- **Database:** `./data/gallus_cms.db`
- **WAL files:** `./data/gallus_cms.db-wal`, `./data/gallus_cms.db-shm`
- **Git workspace:** `./data/workspace/`

### Production (Fly.io)
- **Database:** `/app/data/gallus_cms.db` (on volume)
- **Git workspace:** `/app/data/workspace/` (on volume)
- **Volume name:** `gallus_data` (2GB)

## Backup Strategy

### Manual Backup
```bash
# Local
cp data/gallus_cms.db data/gallus_cms.backup.db

# Production (Fly.io)
flyctl ssh console
sqlite3 /app/data/gallus_cms.db ".backup /app/data/backup.db"
# Then copy back: flyctl ssh sftp get /app/data/backup.db
```

### Automated Backup (Optional)
Consider setting up a cron job or Fly.io machine to periodically:
1. Create SQLite backup
2. Upload to S3/Backblaze/etc.

## Performance Notes

SQLite is perfect for this use case because:
- **Low write concurrency** - Single admin user making changes
- **Read-heavy** - Mostly reading content for publish operations
- **Small dataset** - Events, gallery images, content sections
- **Simple queries** - No complex joins or aggregations

WAL mode is enabled for:
- Better concurrent read access
- Safer writes (crash recovery)
- Improved performance

## Migration from Existing Data

If you had PostgreSQL data to migrate:

1. Export from Postgres:
```sql
\copy events TO 'events.csv' CSV HEADER;
\copy gallery_images TO 'gallery.csv' CSV HEADER;
-- etc.
```

2. Import to SQLite:
```sql
.mode csv
.import events.csv events
.import gallery.csv gallery_images
-- etc.
```

## Known Limitations

1. **No native UUID type** - Using TEXT with UUID format
2. **No native JSON type** - Using TEXT with JSON serialization (Drizzle handles this)
3. **No native TIMESTAMP** - Using INTEGER with Unix epoch (Drizzle handles this)
4. **Single writer** - Only one write transaction at a time (not an issue for this use case)

## Troubleshooting

### "Database is locked" error
- WAL mode should prevent this
- Check if multiple processes are accessing the database
- Ensure proper file permissions

### Native module build errors
- Make sure build tools are installed: `apt-get install python3 make g++` (Linux)
- On Alpine: `apk add python3 make g++`
- Try rebuilding: `pnpm rebuild better-sqlite3`

### Database file not found
- Check `DATABASE_PATH` is set correctly
- Ensure `data/` directory exists
- Check file permissions

## Next Steps

1. ✅ Update dependencies
2. ✅ Update database configuration
3. ✅ Update schema
4. ✅ Update Docker configuration
5. ⏳ Generate migrations: `pnpm run db:generate`
6. ⏳ Run migrations: `pnpm run db:migrate`
7. ⏳ Test development server: `pnpm run dev`
8. ⏳ Test publish flow
9. ⏳ Deploy to Fly.io

The migration is complete! Just need to generate/run migrations and test.