fix(seeder): use fully qualified table names for location seeding
docs: update DATABASE.md and DEVOPS.md for local environment setup
This commit is contained in:
Yamamoto
parent
c9fbcc22bb
commit
e59e15dd35
3 changed files with 187 additions and 227 deletions
|
|
@ -2,14 +2,81 @@
|
||||||
|
|
||||||
Complete database documentation for the GoHorseJobs platform.
|
Complete database documentation for the GoHorseJobs platform.
|
||||||
|
|
||||||
> **Last Updated:** 2024-12-26
|
> **Last Updated:** 2026-01-03
|
||||||
> **Database:** PostgreSQL 16+
|
> **Database:** PostgreSQL 16+ (Local `postgres-main` container)
|
||||||
|
> **Connection:** Internal `gohorsejobs_dev` database via `web_proxy` network
|
||||||
> **ID Strategy:** UUID v7 for core tables, SERIAL for reference tables
|
> **ID Strategy:** UUID v7 for core tables, SERIAL for reference tables
|
||||||
> **Migrations:** 30 SQL files in `backend/migrations/`
|
> **Migrations:** 30 SQL files in `backend/migrations/`
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
## 📊 Entity Relationship Diagram
|
## <20>️ Development Environment Structure
|
||||||
|
|
||||||
|
The development environment (`apolo` server) uses a **Local Containerized Strategy** to ensure isolation and speed.
|
||||||
|
|
||||||
|
### 🏗️ Topology
|
||||||
|
|
||||||
|
```mermaid
|
||||||
|
graph TD
|
||||||
|
subgraph VPS ["Apolo Server (VPS)"]
|
||||||
|
subgraph Net ["Docker Network: web_proxy"]
|
||||||
|
PG[("postgres-main")]
|
||||||
|
BE["Backend API"]
|
||||||
|
BO["Backoffice"]
|
||||||
|
SE["Seeder API"]
|
||||||
|
end
|
||||||
|
Traefik["Traefik Proxy"]
|
||||||
|
end
|
||||||
|
|
||||||
|
Traefik --> BE
|
||||||
|
Traefik --> BO
|
||||||
|
Traefik --> SE
|
||||||
|
|
||||||
|
BE -- "internal:5432" --> PG
|
||||||
|
BO -- "internal:5432" --> PG
|
||||||
|
SE -- "internal:5432" --> PG
|
||||||
|
|
||||||
|
style PG fill:#336791,stroke:#fff,stroke-width:2px,color:#fff
|
||||||
|
```
|
||||||
|
|
||||||
|
### 🔌 Connection Details
|
||||||
|
|
||||||
|
All services connect to the database via the internal Docker network.
|
||||||
|
|
||||||
|
| Parameter | Value | Notes |
|
||||||
|
|-----------|-------|-------|
|
||||||
|
| **Host** | `postgres-main` | Internal Container Hostname |
|
||||||
|
| **Port** | `5432` | Internal Port |
|
||||||
|
| **Database** | `gohorsejobs_dev` | Dedicated Dev DB (Isolated from `main_db`) |
|
||||||
|
| **User** | `yuki` | Owner of public schema |
|
||||||
|
| **Network** | `web_proxy` | Shared Bridge Network |
|
||||||
|
| **SSL Mode** | `disable` | Internal traffic is unencrypted |
|
||||||
|
|
||||||
|
### 🚀 Access & Management
|
||||||
|
|
||||||
|
Since the database runs inside a container and is not exposed to the public internet, use the following methods for access:
|
||||||
|
|
||||||
|
**1. CLI Access (via SSH)**
|
||||||
|
```bash
|
||||||
|
# Connect to PostgreSQL shell
|
||||||
|
ssh root@apolo 'podman exec -it postgres-main psql -U yuki -d gohorsejobs_dev'
|
||||||
|
```
|
||||||
|
|
||||||
|
**2. Run Migrations**
|
||||||
|
Migrations are applied using the Backend service or manually piped:
|
||||||
|
```bash
|
||||||
|
# Manual Pipe (from local machine)
|
||||||
|
cat backend/migrations/*.sql | ssh root@apolo 'podman exec -i postgres-main psql -U yuki -d gohorsejobs_dev'
|
||||||
|
```
|
||||||
|
|
||||||
|
**3. Seeding Data**
|
||||||
|
Trigger the Seeder API (running locally) to populate data:
|
||||||
|
```bash
|
||||||
|
curl -X POST https://seeder.gohorsejobs.com/seed
|
||||||
|
```
|
||||||
|
|
||||||
|
|
||||||
|
## <20>📊 Entity Relationship Diagram
|
||||||
|
|
||||||
```mermaid
|
```mermaid
|
||||||
erDiagram
|
erDiagram
|
||||||
|
|
|
||||||
335
docs/DEVOPS.md
335
docs/DEVOPS.md
|
|
@ -1,262 +1,155 @@
|
||||||
# DevOps - GoHorseJobs
|
# DevOps - GoHorseJobs (Development Environment)
|
||||||
|
|
||||||
Infraestrutura, CI/CD e deploy do projeto GoHorseJobs.
|
Infraestrutura, CI/CD e deploy do projeto GoHorseJobs no servidor `apolo`.
|
||||||
|
|
||||||
> **Last Updated:** 2024-12-26
|
> **Last Updated:** 2026-01-03
|
||||||
|
> **Server:** Apolo VPS
|
||||||
|
> **Tech Stack:** Podman, Systemd (Quadlet), Traefik, PostgreSQL (Local)
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
## 📁 Estrutura
|
## 🏗️ Architecture Diagram
|
||||||
|
|
||||||
```
|
Simplified view of the container hierarchy, networking, and storage.
|
||||||
.
|
|
||||||
├── .drone.yml # Pipeline CI/CD (Drone)
|
```mermaid
|
||||||
├── k8s/
|
graph TD
|
||||||
│ ├── dev/ # Kubernetes - Desenvolvimento
|
subgraph Host ["Apolo VPS (Host)"]
|
||||||
│ ├── hml/ # Kubernetes - Homologação
|
|
||||||
│ └── prd/ # Kubernetes - Produção
|
subgraph FS ["File System (/mnt/data)"]
|
||||||
│
|
EnvBE["/gohorsejobs/backend/.env"]
|
||||||
├── backend/
|
EnvBO["/gohorsejobs/backoffice/.env"]
|
||||||
│ ├── Dockerfile # Go API (multi-stage, ~73MB)
|
EnvSE["/gohorsejobs/seeder-api/.env"]
|
||||||
│ └── .env.example
|
DBData[("postgres-general")]
|
||||||
│
|
end
|
||||||
├── backoffice/
|
|
||||||
│ ├── Dockerfile # NestJS (Fastify)
|
subgraph Net ["Network: web_proxy"]
|
||||||
│ └── .env.example
|
Traefik("Traefik")
|
||||||
│
|
|
||||||
├── frontend/
|
subgraph App ["Application Containers"]
|
||||||
│ ├── Dockerfile # Next.js
|
BE["Backend API (:8521)"]
|
||||||
│ └── .env.example
|
BO["Backoffice (:3001)"]
|
||||||
│
|
SE["Seeder API (:8080)"]
|
||||||
├── seeder-api/ # Node.js seeder
|
FE["Frontend (:3000)"]
|
||||||
└── docs/ # Documentation
|
end
|
||||||
|
|
||||||
|
PG[("postgres-main (:5432)")]
|
||||||
|
end
|
||||||
|
end
|
||||||
|
|
||||||
|
%% Ingress
|
||||||
|
Internet((Internet)) --> Traefik
|
||||||
|
|
||||||
|
%% Routing
|
||||||
|
Traefik -- "dev.gohorsejobs.com" --> FE
|
||||||
|
Traefik -- "api-tmp.gohorsejobs.com" --> BE
|
||||||
|
Traefik -- "b-tmp.gohorsejobs.com" --> BO
|
||||||
|
Traefik -- "seeder.gohorsejobs.com" --> SE
|
||||||
|
|
||||||
|
%% Config Mounts
|
||||||
|
EnvBE -.-> BE
|
||||||
|
EnvBO -.-> BO
|
||||||
|
EnvSE -.-> SE
|
||||||
|
|
||||||
|
%% Data Persistence
|
||||||
|
PG -.-> DBData
|
||||||
|
|
||||||
|
%% Database Connections
|
||||||
|
BE --> PG
|
||||||
|
BO --> PG
|
||||||
|
SE --> PG
|
||||||
|
|
||||||
|
style PG fill:#336791,stroke:#fff,color:#fff
|
||||||
|
style Traefik fill:#f5a623,stroke:#fff,color:#fff
|
||||||
```
|
```
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
## 🌍 Ambientes
|
## 💾 Storage & Persistence (`/mnt/data`)
|
||||||
|
|
||||||
| Ambiente | Branch | Namespace | Réplicas | URL |
|
All persistent data and configuration files are stored in `/mnt/data` on the host.
|
||||||
|----------|--------|-----------|----------|-----|
|
|
||||||
| **DEV** | `dev` | `gohorsejobsdev` | 1 | gohorsejobs-dev.appwrite.network |
|
| Host Path | Container Path | Purpose | Type |
|
||||||
| **HML** | `hml` | `gohorsejobshml` | 2 | - |
|
|-----------|----------------|---------|------|
|
||||||
| **PRD** | `main` | `gohorsejobs` | 3 | gohorsejobs.com |
|
| `/mnt/data/gohorsejobs/backend/.env` | (Injected Env) | **Backend Config:** Secrets, DB URL, Port settings. | File |
|
||||||
|
| `/mnt/data/gohorsejobs/backoffice/.env` | (Injected Env) | **Backoffice Config:** Secrets, DB URL. | File |
|
||||||
|
| `/mnt/data/gohorsejobs/seeder-api/.env` | (Injected Env) | **Seeder Config:** Secrets, DB URL. | File |
|
||||||
|
| `/mnt/data/postgres-general` | `/var/lib/postgresql/data` | **Database Storage:** Main storage for `postgres-main` container. Contains `gohorsejobs_dev` DB. | Directory |
|
||||||
|
|
||||||
|
> **Backup Note:** To backup the environment, ensure `/mnt/data/gohorsejobs` and `/mnt/data/postgres-general` are included in snapshots.
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
## 🔄 Pipeline CI/CD (Drone)
|
## 🌍 Service Maps & Networking
|
||||||
|
|
||||||
### Fluxo
|
### 🚦 Traefik Routing
|
||||||
|
Services are exposed via Traefik labels defined in the Quadlet `.container` files.
|
||||||
|
|
||||||
```
|
| Domain | Service | Internal Port | Host Port (Debug) |
|
||||||
dev branch → build → push (Harbor) → deploy (K8s dev)
|
|--------|---------|---------------|-------------------|
|
||||||
↓
|
| `dev.gohorsejobs.com` | `gohorsejobs-frontend-dev` | `3000` | `8523` |
|
||||||
hml branch → build → push (Harbor) → deploy (K8s hml)
|
| `api-tmp.gohorsejobs.com` | `gohorsejobs-backend-dev` | `8521` | `8521` |
|
||||||
↓
|
| `b-tmp.gohorsejobs.com` | `gohorsejobs-backoffice-dev` | `3001` | - |
|
||||||
main branch → build → push (Harbor) → deploy (K8s prd)
|
| `seeder.gohorsejobs.com` | `gohorsejobs-seeder-dev` | `8080` | `8522` |
|
||||||
```
|
|
||||||
|
|
||||||
### Stages
|
### 🛑 Security
|
||||||
|
- **Backend/Seeder/Frontend** expose ports to the Host (`85xx`) for debugging/direct access if needed.
|
||||||
1. **build-and-push** - Docker build + push to Harbor
|
- **Backoffice** is *only* accessible via Traefik (internal network).
|
||||||
2. **export-envs-to-k8s** - Create/update secrets
|
- **PostgreSQL** is *only* accessible internally via `web_proxy` network (no host port binding).
|
||||||
3. **deploy** - Apply manifests + rollout restart
|
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
## 🔐 Secrets (Drone CI)
|
## 🛠️ Operational Guide
|
||||||
|
|
||||||
### Registry
|
|
||||||
| Secret | Description |
|
|
||||||
|--------|-------------|
|
|
||||||
| `HARBOR_USERNAME` | Harbor username |
|
|
||||||
| `HARBOR_PASSWORD` | Harbor password |
|
|
||||||
|
|
||||||
### Database
|
|
||||||
| Secret | Description |
|
|
||||||
|--------|-------------|
|
|
||||||
| `DATABASE_URL` | PostgreSQL connection string |
|
|
||||||
| `DB_SSLMODE` | `require` or `disable` |
|
|
||||||
|
|
||||||
### Application
|
|
||||||
| Secret | Description |
|
|
||||||
|--------|-------------|
|
|
||||||
| `JWT_SECRET` | JWT secret (min 32 chars) |
|
|
||||||
| `PASSWORD_PEPPER` | Password pepper |
|
|
||||||
| `CORS_ORIGINS` | Allowed origins |
|
|
||||||
|
|
||||||
### External Services
|
|
||||||
| Secret | Description |
|
|
||||||
|--------|-------------|
|
|
||||||
| `RSA_PRIVATE_KEY_BASE64` | RSA key for credentials |
|
|
||||||
| `STRIPE_SECRET_KEY` | Stripe API key |
|
|
||||||
| `STRIPE_WEBHOOK_SECRET` | Stripe webhook secret |
|
|
||||||
| `FIREBASE_SERVICE_ACCOUNT` | Firebase Admin SDK JSON |
|
|
||||||
|
|
||||||
### Storage (S3/R2)
|
|
||||||
| Secret | Description |
|
|
||||||
|--------|-------------|
|
|
||||||
| `AWS_ACCESS_KEY_ID` | Access Key |
|
|
||||||
| `AWS_SECRET_ACCESS_KEY` | Secret Key |
|
|
||||||
| `AWS_ENDPOINT` | S3-compatible endpoint |
|
|
||||||
| `S3_BUCKET` | Bucket name |
|
|
||||||
|
|
||||||
### LavinMQ
|
|
||||||
| Secret | Description |
|
|
||||||
|--------|-------------|
|
|
||||||
| `AMQP_URL` | LavinMQ connection URL |
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## ☸️ Kubernetes
|
|
||||||
|
|
||||||
### Create Namespaces
|
|
||||||
|
|
||||||
|
### 1. View & Manage Configs
|
||||||
|
Configurations are **not** inside containers. Edit them on the host:
|
||||||
```bash
|
```bash
|
||||||
kubectl create namespace gohorsejobsdev
|
# Edit Backend Config
|
||||||
kubectl create namespace gohorsejobshml
|
vim /mnt/data/gohorsejobs/backend/.env
|
||||||
kubectl create namespace gohorsejobs
|
|
||||||
|
# Apply changes
|
||||||
|
systemctl restart gohorsejobs-backend-dev
|
||||||
```
|
```
|
||||||
|
|
||||||
### Registry Secret
|
### 2. Full Environment Restart
|
||||||
|
To restart all GoHorseJobs related services (excluding Database):
|
||||||
```bash
|
```bash
|
||||||
kubectl create secret docker-registry harbor-registry \
|
systemctl restart gohorsejobs-backend-dev gohorsejobs-backoffice-dev gohorsejobs-seeder-dev gohorsejobs-frontend-dev
|
||||||
--docker-server=in.gohorsejobs.com \
|
|
||||||
--docker-username=<user> \
|
|
||||||
--docker-password=<pass> \
|
|
||||||
-n gohorsejobsdev
|
|
||||||
```
|
```
|
||||||
|
|
||||||
### Deploy Manual
|
### 3. Database Access
|
||||||
|
Access the local database directly via the `postgres-main` container:
|
||||||
```bash
|
```bash
|
||||||
# DEV
|
# Internal Connection
|
||||||
kubectl apply -f k8s/dev/ -n gohorsejobsdev
|
docker exec -it postgres-main psql -U yuki -d gohorsejobs_dev
|
||||||
|
|
||||||
# Check pods
|
|
||||||
kubectl get pods -n gohorsejobsdev
|
|
||||||
|
|
||||||
# Check logs
|
|
||||||
kubectl logs -f deployment/gohorse-backend -n gohorsejobsdev
|
|
||||||
|
|
||||||
# Rollout restart
|
|
||||||
kubectl rollout restart deployment/gohorse-backend -n gohorsejobsdev
|
|
||||||
```
|
```
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
## 🐳 Docker
|
## 🚀 Deployment Pipeline (Manual)
|
||||||
|
|
||||||
### Build Local
|
Current workflow uses **Local Build** -> **Forgejo Registry** -> **Server Pull**.
|
||||||
|
|
||||||
|
### 1. Build & Push (Local Machine)
|
||||||
```bash
|
```bash
|
||||||
# Backend
|
# Login
|
||||||
cd backend && docker build -t gohorse-backend .
|
podman login forgejo-gru.rede5.com.br
|
||||||
|
|
||||||
# Backoffice
|
# Build
|
||||||
cd backoffice && docker build -t gohorse-backoffice .
|
|
||||||
|
|
||||||
# Frontend
|
|
||||||
cd frontend && docker build -t gohorse-frontend .
|
|
||||||
```
|
|
||||||
|
|
||||||
### Docker Compose (Dev)
|
|
||||||
|
|
||||||
```yaml
|
|
||||||
version: '3.8'
|
|
||||||
services:
|
|
||||||
backend:
|
|
||||||
build: ./backend
|
|
||||||
ports:
|
|
||||||
- "8521:8521"
|
|
||||||
env_file:
|
|
||||||
- ./backend/.env
|
|
||||||
|
|
||||||
backoffice:
|
|
||||||
build: ./backoffice
|
|
||||||
ports:
|
|
||||||
- "3001:3001"
|
|
||||||
env_file:
|
|
||||||
- ./backoffice/.env
|
|
||||||
|
|
||||||
frontend:
|
|
||||||
build: ./frontend
|
|
||||||
ports:
|
|
||||||
- "3000:3000"
|
|
||||||
env_file:
|
|
||||||
- ./frontend/.env
|
|
||||||
```
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## 🗄️ Database
|
|
||||||
|
|
||||||
### Connection
|
|
||||||
|
|
||||||
```bash
|
|
||||||
# PostgreSQL hosted
|
|
||||||
Host: db-60059.dc-sp-1.absamcloud.com
|
|
||||||
Port: 26868
|
|
||||||
SSL: require
|
|
||||||
```
|
|
||||||
|
|
||||||
### Databases
|
|
||||||
|
|
||||||
| Environment | Database |
|
|
||||||
|-------------|----------|
|
|
||||||
| DEV | `gohorsejobs_dev` |
|
|
||||||
| HML | `gohorsejobs_hml` |
|
|
||||||
| PRD | `gohorsejobs` |
|
|
||||||
|
|
||||||
### Migrations
|
|
||||||
|
|
||||||
```bash
|
|
||||||
cd backend
|
cd backend
|
||||||
go run ./cmd/manual_migrate
|
podman build -t forgejo-gru.rede5.com.br/rede5/gohorsejobs-backend:latest .
|
||||||
|
|
||||||
|
# Push
|
||||||
|
podman push forgejo-gru.rede5.com.br/rede5/gohorsejobs-backend:latest
|
||||||
```
|
```
|
||||||
|
|
||||||
### Seeder
|
### 2. Deploy (On Apolo Server)
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
cd seeder-api
|
ssh root@apolo
|
||||||
npm install
|
|
||||||
npm run seed
|
# Pull new image
|
||||||
|
podman pull forgejo-gru.rede5.com.br/rede5/gohorsejobs-backend:latest
|
||||||
|
|
||||||
|
# Restart service (Systemd handles container recreation)
|
||||||
|
systemctl restart gohorsejobs-backend-dev
|
||||||
```
|
```
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## 👤 Test Users
|
|
||||||
|
|
||||||
> **Nota:** O SuperAdmin foi atualizado via migration `032_update_superadmin_lol.sql`.
|
|
||||||
|
|
||||||
### SuperAdmin
|
|
||||||
- **Login:** `lol`
|
|
||||||
- **Email:** `lol@gohorsejobs.com`
|
|
||||||
- **Nome:** Dr. Horse Expert
|
|
||||||
- **Password:** *trocar no primeiro acesso* (status `force_change_password`)
|
|
||||||
|
|
||||||
### Company Admin
|
|
||||||
| Login | Password |
|
|
||||||
|-------|----------|
|
|
||||||
| `takeshi_yamamoto` | `Takeshi@2025` |
|
|
||||||
|
|
||||||
### Candidate
|
|
||||||
| Login | Password |
|
|
||||||
|-------|----------|
|
|
||||||
| `paulo_santos` | `User@2025` |
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## 📋 Deploy Checklist
|
|
||||||
|
|
||||||
- [ ] Create K8s namespace
|
|
||||||
- [ ] Create `harbor-registry` secret
|
|
||||||
- [ ] Add Drone CI secrets
|
|
||||||
- [ ] Create database
|
|
||||||
- [ ] Run migrations
|
|
||||||
- [ ] Run seeder (optional)
|
|
||||||
- [ ] Push to branch
|
|
||||||
- [ ] Verify pipeline logs
|
|
||||||
- [ ] Test `/health` endpoint
|
|
||||||
- [ ] Test `/docs` endpoint
|
|
||||||
|
|
|
||||||
|
|
@ -13,9 +13,9 @@ const SQL_DIR = join(__dirname, '..', '..', 'sql');
|
||||||
* Table name mapping from SQL dumps to our schema
|
* Table name mapping from SQL dumps to our schema
|
||||||
*/
|
*/
|
||||||
const TABLE_MAPPING = {
|
const TABLE_MAPPING = {
|
||||||
'public.regions': 'continents',
|
'public.regions': 'public.continents',
|
||||||
'regions': 'continents',
|
'regions': 'public.continents',
|
||||||
'public.subregions': 'subregions',
|
'public.subregions': 'public.subregions',
|
||||||
};
|
};
|
||||||
|
|
||||||
/**
|
/**
|
||||||
|
|
|
||||||
Loading…
Reference in a new issue