Gallus_Pub/backend/SQLITE_MIGRATION.md

SQLite Migration Summary

Changes Made

The backend has been migrated from PostgreSQL to SQLite for both local development and production (Fly.io).

Benefits of SQLite

1. Simplified Deployment - No separate database service needed
2. Lower Cost - Save ~$15/month (no Postgres hosting)
3. Easier Development - No need to install/run PostgreSQL locally
4. Single File Database - Easy backups and migrations
5. Perfect for this use case - Low concurrent writes, simple queries

Modified Files

Dependencies

• package.json
  • Removed: pg, @types/pg
  • Added: better-sqlite3, @types/better-sqlite3

Database Configuration

• src/config/database.ts

  • Changed from drizzle-orm/node-postgres to drizzle-orm/better-sqlite3
  • Uses DATABASE_PATH instead of DATABASE_URL
  • Enabled WAL mode for better concurrent access

• src/config/env.ts

  • Changed DATABASE_URL to DATABASE_PATH
  • Default: ./data/gallus_cms.db

• src/db/schema.ts

  • Changed from pgTable to sqliteTable
  • Changed uuid() to text() with crypto.randomUUID()
  • Changed jsonb() to text(..., { mode: 'json' })
  • Changed timestamp() to integer(..., { mode: 'timestamp' })
  • Changed boolean() to integer(..., { mode: 'boolean' })
  • Uses sql\(unixepoch())`` for default timestamps

• drizzle.config.ts

  • Changed dialect from postgresql to sqlite
  • Uses DATABASE_PATH instead of DATABASE_URL

Environment Files

• .env and .env.example
  • Changed DATABASE_URL=postgresql://... to DATABASE_PATH=./data/gallus_cms.db
  • Changed GIT_WORKSPACE_DIR=/tmp/gallus-repo to ./data/workspace

Docker Configuration

• Dockerfile

  • Added build tools for better-sqlite3 native module (python3, make, g++)
  • Added sqlite CLI tool
  • Creates /app/data directory for database
  • Sets DATABASE_PATH=/app/data/gallus_cms.db
  • Proper permissions for non-root user

• fly.toml

  • Added DATABASE_PATH and GIT_WORKSPACE_DIR to [env]
  • Changed volume mount from gallus_repo_workspace to gallus_data
  • Mount destination: /app/data (contains both DB and git workspace)

Documentation

• README.md - Updated setup instructions
• DEPLOYMENT.md - Removed Postgres setup, updated volume creation
• SQLITE_MIGRATION.md - This file!

Local Development

Setup

# Dependencies already installed
pnpm install

# Create data directory (done)
mkdir -p data

# Database will be created automatically at ./data/gallus_cms.db

Generate and Run Migrations

# Generate migration files from schema
pnpm run db:generate

# Run migrations to create tables
pnpm run db:migrate

Start Development Server

pnpm run dev

The database file will be created at ./data/gallus_cms.db on first run.

Production (Fly.io)

Volume Setup

# Create single volume for both database and git workspace
flyctl volumes create gallus_data --size 2 --region ams

Environment Variables

Set in fly.toml (non-sensitive):

• DATABASE_PATH=/app/data/gallus_cms.db
• GIT_WORKSPACE_DIR=/app/data/workspace

Set as secrets (sensitive):

• All other env vars (OAuth credentials, tokens, etc.)

Deployment

flyctl deploy

Database will be created automatically on first start. No need for separate database service!

Database Location

Local Development

• Database: ./data/gallus_cms.db
• WAL files: ./data/gallus_cms.db-wal, ./data/gallus_cms.db-shm
• Git workspace: ./data/workspace/

Production (Fly.io)

• Database: /app/data/gallus_cms.db (on volume)
• Git workspace: /app/data/workspace/ (on volume)
• Volume name: gallus_data (2GB)

Backup Strategy

Manual Backup

# Local
cp data/gallus_cms.db data/gallus_cms.backup.db

# Production (Fly.io)
flyctl ssh console
sqlite3 /app/data/gallus_cms.db ".backup /app/data/backup.db"
# Then copy back: flyctl ssh sftp get /app/data/backup.db

Automated Backup (Optional)

Consider setting up a cron job or Fly.io machine to periodically:

1. Create SQLite backup
2. Upload to S3/Backblaze/etc.

Performance Notes

SQLite is perfect for this use case because:

• Low write concurrency - Single admin user making changes
• Read-heavy - Mostly reading content for publish operations
• Small dataset - Events, gallery images, content sections
• Simple queries - No complex joins or aggregations

WAL mode is enabled for:

• Better concurrent read access
• Safer writes (crash recovery)
• Improved performance

Migration from Existing Data

If you had PostgreSQL data to migrate:

1. Export from Postgres:

\copy events TO 'events.csv' CSV HEADER;
\copy gallery_images TO 'gallery.csv' CSV HEADER;
-- etc.

2. Import to SQLite:

.mode csv
.import events.csv events
.import gallery.csv gallery_images
-- etc.

Known Limitations

1. No native UUID type - Using TEXT with UUID format
2. No native JSON type - Using TEXT with JSON serialization (Drizzle handles this)
3. No native TIMESTAMP - Using INTEGER with Unix epoch (Drizzle handles this)
4. Single writer - Only one write transaction at a time (not an issue for this use case)

Troubleshooting

"Database is locked" error

• WAL mode should prevent this
• Check if multiple processes are accessing the database
• Ensure proper file permissions

Native module build errors

• Make sure build tools are installed: apt-get install python3 make g++ (Linux)
• On Alpine: apk add python3 make g++
• Try rebuilding: pnpm rebuild better-sqlite3

Database file not found

• Check DATABASE_PATH is set correctly
• Ensure data/ directory exists
• Check file permissions

Next Steps

1. ✅ Update dependencies
2. ✅ Update database configuration
3. ✅ Update schema
4. ✅ Update Docker configuration
5. ⏳ Generate migrations: pnpm run db:generate
6. ⏳ Run migrations: pnpm run db:migrate
7. ⏳ Test development server: pnpm run dev
8. ⏳ Test publish flow
9. ⏳ Deploy to Fly.io

The migration is complete! Just need to generate/run migrations and test.