feat(backend): initial setup for cms backend service
This commit is contained in:
Fx64b
217
backend/SQLITE_MIGRATION.md
Normal file
217
backend/SQLITE_MIGRATION.md
Normal file
@ -0,0 +1,217 @@
|
||||
# SQLite Migration Summary
|
||||
|
||||
## Changes Made
|
||||
|
||||
The backend has been migrated from PostgreSQL to SQLite for both local development and production (Fly.io).
|
||||
|
||||
### Benefits of SQLite
|
||||
|
||||
1. **Simplified Deployment** - No separate database service needed
|
||||
2. **Lower Cost** - Save ~$15/month (no Postgres hosting)
|
||||
3. **Easier Development** - No need to install/run PostgreSQL locally
|
||||
4. **Single File Database** - Easy backups and migrations
|
||||
5. **Perfect for this use case** - Low concurrent writes, simple queries
|
||||
|
||||
## Modified Files
|
||||
|
||||
### Dependencies
|
||||
- **package.json**
|
||||
- Removed: `pg`, `@types/pg`
|
||||
- Added: `better-sqlite3`, `@types/better-sqlite3`
|
||||
|
||||
### Database Configuration
|
||||
- **src/config/database.ts**
|
||||
- Changed from `drizzle-orm/node-postgres` to `drizzle-orm/better-sqlite3`
|
||||
- Uses `DATABASE_PATH` instead of `DATABASE_URL`
|
||||
- Enabled WAL mode for better concurrent access
|
||||
|
||||
- **src/config/env.ts**
|
||||
- Changed `DATABASE_URL` to `DATABASE_PATH`
|
||||
- Default: `./data/gallus_cms.db`
|
||||
|
||||
- **src/db/schema.ts**
|
||||
- Changed from `pgTable` to `sqliteTable`
|
||||
- Changed `uuid()` to `text()` with `crypto.randomUUID()`
|
||||
- Changed `jsonb()` to `text(..., { mode: 'json' })`
|
||||
- Changed `timestamp()` to `integer(..., { mode: 'timestamp' })`
|
||||
- Changed `boolean()` to `integer(..., { mode: 'boolean' })`
|
||||
- Uses `sql\`(unixepoch())\`` for default timestamps
|
||||
|
||||
- **drizzle.config.ts**
|
||||
- Changed dialect from `postgresql` to `sqlite`
|
||||
- Uses `DATABASE_PATH` instead of `DATABASE_URL`
|
||||
|
||||
### Environment Files
|
||||
- **.env** and **.env.example**
|
||||
- Changed `DATABASE_URL=postgresql://...` to `DATABASE_PATH=./data/gallus_cms.db`
|
||||
- Changed `GIT_WORKSPACE_DIR=/tmp/gallus-repo` to `./data/workspace`
|
||||
|
||||
### Docker Configuration
|
||||
- **Dockerfile**
|
||||
- Added build tools for `better-sqlite3` native module (python3, make, g++)
|
||||
- Added `sqlite` CLI tool
|
||||
- Creates `/app/data` directory for database
|
||||
- Sets `DATABASE_PATH=/app/data/gallus_cms.db`
|
||||
- Proper permissions for non-root user
|
||||
|
||||
- **fly.toml**
|
||||
- Added `DATABASE_PATH` and `GIT_WORKSPACE_DIR` to [env]
|
||||
- Changed volume mount from `gallus_repo_workspace` to `gallus_data`
|
||||
- Mount destination: `/app/data` (contains both DB and git workspace)
|
||||
|
||||
### Documentation
|
||||
- **README.md** - Updated setup instructions
|
||||
- **DEPLOYMENT.md** - Removed Postgres setup, updated volume creation
|
||||
- **SQLITE_MIGRATION.md** - This file!
|
||||
|
||||
## Local Development
|
||||
|
||||
### Setup
|
||||
```bash
|
||||
# Dependencies already installed
|
||||
pnpm install
|
||||
|
||||
# Create data directory (done)
|
||||
mkdir -p data
|
||||
|
||||
# Database will be created automatically at ./data/gallus_cms.db
|
||||
```
|
||||
|
||||
### Generate and Run Migrations
|
||||
```bash
|
||||
# Generate migration files from schema
|
||||
pnpm run db:generate
|
||||
|
||||
# Run migrations to create tables
|
||||
pnpm run db:migrate
|
||||
```
|
||||
|
||||
### Start Development Server
|
||||
```bash
|
||||
pnpm run dev
|
||||
```
|
||||
|
||||
The database file will be created at `./data/gallus_cms.db` on first run.
|
||||
|
||||
## Production (Fly.io)
|
||||
|
||||
### Volume Setup
|
||||
```bash
|
||||
# Create single volume for both database and git workspace
|
||||
flyctl volumes create gallus_data --size 2 --region ams
|
||||
```
|
||||
|
||||
### Environment Variables
|
||||
Set in fly.toml (non-sensitive):
|
||||
- `DATABASE_PATH=/app/data/gallus_cms.db`
|
||||
- `GIT_WORKSPACE_DIR=/app/data/workspace`
|
||||
|
||||
Set as secrets (sensitive):
|
||||
- All other env vars (OAuth credentials, tokens, etc.)
|
||||
|
||||
### Deployment
|
||||
```bash
|
||||
flyctl deploy
|
||||
```
|
||||
|
||||
Database will be created automatically on first start. No need for separate database service!
|
||||
|
||||
## Database Location
|
||||
|
||||
### Local Development
|
||||
- **Database:** `./data/gallus_cms.db`
|
||||
- **WAL files:** `./data/gallus_cms.db-wal`, `./data/gallus_cms.db-shm`
|
||||
- **Git workspace:** `./data/workspace/`
|
||||
|
||||
### Production (Fly.io)
|
||||
- **Database:** `/app/data/gallus_cms.db` (on volume)
|
||||
- **Git workspace:** `/app/data/workspace/` (on volume)
|
||||
- **Volume name:** `gallus_data` (2GB)
|
||||
|
||||
## Backup Strategy
|
||||
|
||||
### Manual Backup
|
||||
```bash
|
||||
# Local
|
||||
cp data/gallus_cms.db data/gallus_cms.backup.db
|
||||
|
||||
# Production (Fly.io)
|
||||
flyctl ssh console
|
||||
sqlite3 /app/data/gallus_cms.db ".backup /app/data/backup.db"
|
||||
# Then copy back: flyctl ssh sftp get /app/data/backup.db
|
||||
```
|
||||
|
||||
### Automated Backup (Optional)
|
||||
Consider setting up a cron job or Fly.io machine to periodically:
|
||||
1. Create SQLite backup
|
||||
2. Upload to S3/Backblaze/etc.
|
||||
|
||||
## Performance Notes
|
||||
|
||||
SQLite is perfect for this use case because:
|
||||
- **Low write concurrency** - Single admin user making changes
|
||||
- **Read-heavy** - Mostly reading content for publish operations
|
||||
- **Small dataset** - Events, gallery images, content sections
|
||||
- **Simple queries** - No complex joins or aggregations
|
||||
|
||||
WAL mode is enabled for:
|
||||
- Better concurrent read access
|
||||
- Safer writes (crash recovery)
|
||||
- Improved performance
|
||||
|
||||
## Migration from Existing Data
|
||||
|
||||
If you had PostgreSQL data to migrate:
|
||||
|
||||
1. Export from Postgres:
|
||||
```sql
|
||||
\copy events TO 'events.csv' CSV HEADER;
|
||||
\copy gallery_images TO 'gallery.csv' CSV HEADER;
|
||||
-- etc.
|
||||
```
|
||||
|
||||
2. Import to SQLite:
|
||||
```sql
|
||||
.mode csv
|
||||
.import events.csv events
|
||||
.import gallery.csv gallery_images
|
||||
-- etc.
|
||||
```
|
||||
|
||||
## Known Limitations
|
||||
|
||||
1. **No native UUID type** - Using TEXT with UUID format
|
||||
2. **No native JSON type** - Using TEXT with JSON serialization (Drizzle handles this)
|
||||
3. **No native TIMESTAMP** - Using INTEGER with Unix epoch (Drizzle handles this)
|
||||
4. **Single writer** - Only one write transaction at a time (not an issue for this use case)
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### "Database is locked" error
|
||||
- WAL mode should prevent this
|
||||
- Check if multiple processes are accessing the database
|
||||
- Ensure proper file permissions
|
||||
|
||||
### Native module build errors
|
||||
- Make sure build tools are installed: `apt-get install python3 make g++` (Linux)
|
||||
- On Alpine: `apk add python3 make g++`
|
||||
- Try rebuilding: `pnpm rebuild better-sqlite3`
|
||||
|
||||
### Database file not found
|
||||
- Check `DATABASE_PATH` is set correctly
|
||||
- Ensure `data/` directory exists
|
||||
- Check file permissions
|
||||
|
||||
## Next Steps
|
||||
|
||||
1. ✅ Update dependencies
|
||||
2. ✅ Update database configuration
|
||||
3. ✅ Update schema
|
||||
4. ✅ Update Docker configuration
|
||||
5. ⏳ Generate migrations: `pnpm run db:generate`
|
||||
6. ⏳ Run migrations: `pnpm run db:migrate`
|
||||
7. ⏳ Test development server: `pnpm run dev`
|
||||
8. ⏳ Test publish flow
|
||||
9. ⏳ Deploy to Fly.io
|
||||
|
||||
The migration is complete! Just need to generate/run migrations and test.
|
||||
Reference in New Issue
Block a user