From e59e15dd35564c6fb89c5341249914ad53e22a74 Mon Sep 17 00:00:00 2001 From: Yamamoto Date: Sat, 3 Jan 2026 16:55:45 -0300 Subject: [PATCH] fix(seeder): use fully qualified table names for location seeding docs: update DATABASE.md and DEVOPS.md for local environment setup --- docs/DATABASE.md | 73 ++++- docs/DEVOPS.md | 335 ++++++++-------------- seeder-api/src/seeders/location-loader.js | 6 +- 3 files changed, 187 insertions(+), 227 deletions(-) diff --git a/docs/DATABASE.md b/docs/DATABASE.md index b00cda4..d822263 100644 --- a/docs/DATABASE.md +++ b/docs/DATABASE.md @@ -2,14 +2,81 @@ Complete database documentation for the GoHorseJobs platform. -> **Last Updated:** 2024-12-26 -> **Database:** PostgreSQL 16+ +> **Last Updated:** 2026-01-03 +> **Database:** PostgreSQL 16+ (Local `postgres-main` container) +> **Connection:** Internal `gohorsejobs_dev` database via `web_proxy` network > **ID Strategy:** UUID v7 for core tables, SERIAL for reference tables > **Migrations:** 30 SQL files in `backend/migrations/` --- -## πŸ“Š Entity Relationship Diagram +## �️ Development Environment Structure + +The development environment (`apolo` server) uses a **Local Containerized Strategy** to ensure isolation and speed. + +### πŸ—οΈ Topology + +```mermaid +graph TD + subgraph VPS ["Apolo Server (VPS)"] + subgraph Net ["Docker Network: web_proxy"] + PG[("postgres-main")] + BE["Backend API"] + BO["Backoffice"] + SE["Seeder API"] + end + Traefik["Traefik Proxy"] + end + + Traefik --> BE + Traefik --> BO + Traefik --> SE + + BE -- "internal:5432" --> PG + BO -- "internal:5432" --> PG + SE -- "internal:5432" --> PG + + style PG fill:#336791,stroke:#fff,stroke-width:2px,color:#fff +``` + +### πŸ”Œ Connection Details + +All services connect to the database via the internal Docker network. + +| Parameter | Value | Notes | +|-----------|-------|-------| +| **Host** | `postgres-main` | Internal Container Hostname | +| **Port** | `5432` | Internal Port | +| **Database** | `gohorsejobs_dev` | Dedicated Dev DB (Isolated from `main_db`) | +| **User** | `yuki` | Owner of public schema | +| **Network** | `web_proxy` | Shared Bridge Network | +| **SSL Mode** | `disable` | Internal traffic is unencrypted | + +### πŸš€ Access & Management + +Since the database runs inside a container and is not exposed to the public internet, use the following methods for access: + +**1. CLI Access (via SSH)** +```bash +# Connect to PostgreSQL shell +ssh root@apolo 'podman exec -it postgres-main psql -U yuki -d gohorsejobs_dev' +``` + +**2. Run Migrations** +Migrations are applied using the Backend service or manually piped: +```bash +# Manual Pipe (from local machine) +cat backend/migrations/*.sql | ssh root@apolo 'podman exec -i postgres-main psql -U yuki -d gohorsejobs_dev' +``` + +**3. Seeding Data** +Trigger the Seeder API (running locally) to populate data: +```bash +curl -X POST https://seeder.gohorsejobs.com/seed +``` + + +## οΏ½πŸ“Š Entity Relationship Diagram ```mermaid erDiagram diff --git a/docs/DEVOPS.md b/docs/DEVOPS.md index 9e8ea07..db62edd 100644 --- a/docs/DEVOPS.md +++ b/docs/DEVOPS.md @@ -1,262 +1,155 @@ -# DevOps - GoHorseJobs +# DevOps - GoHorseJobs (Development Environment) -Infraestrutura, CI/CD e deploy do projeto GoHorseJobs. +Infraestrutura, CI/CD e deploy do projeto GoHorseJobs no servidor `apolo`. -> **Last Updated:** 2024-12-26 +> **Last Updated:** 2026-01-03 +> **Server:** Apolo VPS +> **Tech Stack:** Podman, Systemd (Quadlet), Traefik, PostgreSQL (Local) --- -## πŸ“ Estrutura +## πŸ—οΈ Architecture Diagram -``` -. -β”œβ”€β”€ .drone.yml # Pipeline CI/CD (Drone) -β”œβ”€β”€ k8s/ -β”‚ β”œβ”€β”€ dev/ # Kubernetes - Desenvolvimento -β”‚ β”œβ”€β”€ hml/ # Kubernetes - Homologação -β”‚ └── prd/ # Kubernetes - Produção -β”‚ -β”œβ”€β”€ backend/ -β”‚ β”œβ”€β”€ Dockerfile # Go API (multi-stage, ~73MB) -β”‚ └── .env.example -β”‚ -β”œβ”€β”€ backoffice/ -β”‚ β”œβ”€β”€ Dockerfile # NestJS (Fastify) -β”‚ └── .env.example -β”‚ -β”œβ”€β”€ frontend/ -β”‚ β”œβ”€β”€ Dockerfile # Next.js -β”‚ └── .env.example -β”‚ -β”œβ”€β”€ seeder-api/ # Node.js seeder -└── docs/ # Documentation +Simplified view of the container hierarchy, networking, and storage. + +```mermaid +graph TD + subgraph Host ["Apolo VPS (Host)"] + + subgraph FS ["File System (/mnt/data)"] + EnvBE["/gohorsejobs/backend/.env"] + EnvBO["/gohorsejobs/backoffice/.env"] + EnvSE["/gohorsejobs/seeder-api/.env"] + DBData[("postgres-general")] + end + + subgraph Net ["Network: web_proxy"] + Traefik("Traefik") + + subgraph App ["Application Containers"] + BE["Backend API (:8521)"] + BO["Backoffice (:3001)"] + SE["Seeder API (:8080)"] + FE["Frontend (:3000)"] + end + + PG[("postgres-main (:5432)")] + end + end + + %% Ingress + Internet((Internet)) --> Traefik + + %% Routing + Traefik -- "dev.gohorsejobs.com" --> FE + Traefik -- "api-tmp.gohorsejobs.com" --> BE + Traefik -- "b-tmp.gohorsejobs.com" --> BO + Traefik -- "seeder.gohorsejobs.com" --> SE + + %% Config Mounts + EnvBE -.-> BE + EnvBO -.-> BO + EnvSE -.-> SE + + %% Data Persistence + PG -.-> DBData + + %% Database Connections + BE --> PG + BO --> PG + SE --> PG + + style PG fill:#336791,stroke:#fff,color:#fff + style Traefik fill:#f5a623,stroke:#fff,color:#fff ``` --- -## 🌍 Ambientes +## πŸ’Ύ Storage & Persistence (`/mnt/data`) -| Ambiente | Branch | Namespace | RΓ©plicas | URL | -|----------|--------|-----------|----------|-----| -| **DEV** | `dev` | `gohorsejobsdev` | 1 | gohorsejobs-dev.appwrite.network | -| **HML** | `hml` | `gohorsejobshml` | 2 | - | -| **PRD** | `main` | `gohorsejobs` | 3 | gohorsejobs.com | +All persistent data and configuration files are stored in `/mnt/data` on the host. + +| Host Path | Container Path | Purpose | Type | +|-----------|----------------|---------|------| +| `/mnt/data/gohorsejobs/backend/.env` | (Injected Env) | **Backend Config:** Secrets, DB URL, Port settings. | File | +| `/mnt/data/gohorsejobs/backoffice/.env` | (Injected Env) | **Backoffice Config:** Secrets, DB URL. | File | +| `/mnt/data/gohorsejobs/seeder-api/.env` | (Injected Env) | **Seeder Config:** Secrets, DB URL. | File | +| `/mnt/data/postgres-general` | `/var/lib/postgresql/data` | **Database Storage:** Main storage for `postgres-main` container. Contains `gohorsejobs_dev` DB. | Directory | + +> **Backup Note:** To backup the environment, ensure `/mnt/data/gohorsejobs` and `/mnt/data/postgres-general` are included in snapshots. --- -## πŸ”„ Pipeline CI/CD (Drone) +## 🌍 Service Maps & Networking -### Fluxo +### 🚦 Traefik Routing +Services are exposed via Traefik labels defined in the Quadlet `.container` files. -``` -dev branch β†’ build β†’ push (Harbor) β†’ deploy (K8s dev) - ↓ -hml branch β†’ build β†’ push (Harbor) β†’ deploy (K8s hml) - ↓ -main branch β†’ build β†’ push (Harbor) β†’ deploy (K8s prd) -``` +| Domain | Service | Internal Port | Host Port (Debug) | +|--------|---------|---------------|-------------------| +| `dev.gohorsejobs.com` | `gohorsejobs-frontend-dev` | `3000` | `8523` | +| `api-tmp.gohorsejobs.com` | `gohorsejobs-backend-dev` | `8521` | `8521` | +| `b-tmp.gohorsejobs.com` | `gohorsejobs-backoffice-dev` | `3001` | - | +| `seeder.gohorsejobs.com` | `gohorsejobs-seeder-dev` | `8080` | `8522` | -### Stages - -1. **build-and-push** - Docker build + push to Harbor -2. **export-envs-to-k8s** - Create/update secrets -3. **deploy** - Apply manifests + rollout restart +### πŸ›‘ Security +- **Backend/Seeder/Frontend** expose ports to the Host (`85xx`) for debugging/direct access if needed. +- **Backoffice** is *only* accessible via Traefik (internal network). +- **PostgreSQL** is *only* accessible internally via `web_proxy` network (no host port binding). --- -## πŸ” Secrets (Drone CI) - -### Registry -| Secret | Description | -|--------|-------------| -| `HARBOR_USERNAME` | Harbor username | -| `HARBOR_PASSWORD` | Harbor password | - -### Database -| Secret | Description | -|--------|-------------| -| `DATABASE_URL` | PostgreSQL connection string | -| `DB_SSLMODE` | `require` or `disable` | - -### Application -| Secret | Description | -|--------|-------------| -| `JWT_SECRET` | JWT secret (min 32 chars) | -| `PASSWORD_PEPPER` | Password pepper | -| `CORS_ORIGINS` | Allowed origins | - -### External Services -| Secret | Description | -|--------|-------------| -| `RSA_PRIVATE_KEY_BASE64` | RSA key for credentials | -| `STRIPE_SECRET_KEY` | Stripe API key | -| `STRIPE_WEBHOOK_SECRET` | Stripe webhook secret | -| `FIREBASE_SERVICE_ACCOUNT` | Firebase Admin SDK JSON | - -### Storage (S3/R2) -| Secret | Description | -|--------|-------------| -| `AWS_ACCESS_KEY_ID` | Access Key | -| `AWS_SECRET_ACCESS_KEY` | Secret Key | -| `AWS_ENDPOINT` | S3-compatible endpoint | -| `S3_BUCKET` | Bucket name | - -### LavinMQ -| Secret | Description | -|--------|-------------| -| `AMQP_URL` | LavinMQ connection URL | - ---- - -## ☸️ Kubernetes - -### Create Namespaces +## πŸ› οΈ Operational Guide +### 1. View & Manage Configs +Configurations are **not** inside containers. Edit them on the host: ```bash -kubectl create namespace gohorsejobsdev -kubectl create namespace gohorsejobshml -kubectl create namespace gohorsejobs +# Edit Backend Config +vim /mnt/data/gohorsejobs/backend/.env + +# Apply changes +systemctl restart gohorsejobs-backend-dev ``` -### Registry Secret - +### 2. Full Environment Restart +To restart all GoHorseJobs related services (excluding Database): ```bash -kubectl create secret docker-registry harbor-registry \ - --docker-server=in.gohorsejobs.com \ - --docker-username= \ - --docker-password= \ - -n gohorsejobsdev +systemctl restart gohorsejobs-backend-dev gohorsejobs-backoffice-dev gohorsejobs-seeder-dev gohorsejobs-frontend-dev ``` -### Deploy Manual - +### 3. Database Access +Access the local database directly via the `postgres-main` container: ```bash -# DEV -kubectl apply -f k8s/dev/ -n gohorsejobsdev - -# Check pods -kubectl get pods -n gohorsejobsdev - -# Check logs -kubectl logs -f deployment/gohorse-backend -n gohorsejobsdev - -# Rollout restart -kubectl rollout restart deployment/gohorse-backend -n gohorsejobsdev +# Internal Connection +docker exec -it postgres-main psql -U yuki -d gohorsejobs_dev ``` --- -## 🐳 Docker +## πŸš€ Deployment Pipeline (Manual) -### Build Local +Current workflow uses **Local Build** -> **Forgejo Registry** -> **Server Pull**. +### 1. Build & Push (Local Machine) ```bash -# Backend -cd backend && docker build -t gohorse-backend . +# Login +podman login forgejo-gru.rede5.com.br -# Backoffice -cd backoffice && docker build -t gohorse-backoffice . - -# Frontend -cd frontend && docker build -t gohorse-frontend . -``` - -### Docker Compose (Dev) - -```yaml -version: '3.8' -services: - backend: - build: ./backend - ports: - - "8521:8521" - env_file: - - ./backend/.env - - backoffice: - build: ./backoffice - ports: - - "3001:3001" - env_file: - - ./backoffice/.env - - frontend: - build: ./frontend - ports: - - "3000:3000" - env_file: - - ./frontend/.env -``` - ---- - -## πŸ—„οΈ Database - -### Connection - -```bash -# PostgreSQL hosted -Host: db-60059.dc-sp-1.absamcloud.com -Port: 26868 -SSL: require -``` - -### Databases - -| Environment | Database | -|-------------|----------| -| DEV | `gohorsejobs_dev` | -| HML | `gohorsejobs_hml` | -| PRD | `gohorsejobs` | - -### Migrations - -```bash +# Build cd backend -go run ./cmd/manual_migrate +podman build -t forgejo-gru.rede5.com.br/rede5/gohorsejobs-backend:latest . + +# Push +podman push forgejo-gru.rede5.com.br/rede5/gohorsejobs-backend:latest ``` -### Seeder - +### 2. Deploy (On Apolo Server) ```bash -cd seeder-api -npm install -npm run seed +ssh root@apolo + +# Pull new image +podman pull forgejo-gru.rede5.com.br/rede5/gohorsejobs-backend:latest + +# Restart service (Systemd handles container recreation) +systemctl restart gohorsejobs-backend-dev ``` - ---- - -## πŸ‘€ Test Users - -> **Nota:** O SuperAdmin foi atualizado via migration `032_update_superadmin_lol.sql`. - -### SuperAdmin -- **Login:** `lol` -- **Email:** `lol@gohorsejobs.com` -- **Nome:** Dr. Horse Expert -- **Password:** *trocar no primeiro acesso* (status `force_change_password`) - -### Company Admin -| Login | Password | -|-------|----------| -| `takeshi_yamamoto` | `Takeshi@2025` | - -### Candidate -| Login | Password | -|-------|----------| -| `paulo_santos` | `User@2025` | - ---- - -## πŸ“‹ Deploy Checklist - -- [ ] Create K8s namespace -- [ ] Create `harbor-registry` secret -- [ ] Add Drone CI secrets -- [ ] Create database -- [ ] Run migrations -- [ ] Run seeder (optional) -- [ ] Push to branch -- [ ] Verify pipeline logs -- [ ] Test `/health` endpoint -- [ ] Test `/docs` endpoint diff --git a/seeder-api/src/seeders/location-loader.js b/seeder-api/src/seeders/location-loader.js index 30efab5..c198582 100644 --- a/seeder-api/src/seeders/location-loader.js +++ b/seeder-api/src/seeders/location-loader.js @@ -13,9 +13,9 @@ const SQL_DIR = join(__dirname, '..', '..', 'sql'); * Table name mapping from SQL dumps to our schema */ const TABLE_MAPPING = { - 'public.regions': 'continents', - 'regions': 'continents', - 'public.subregions': 'subregions', + 'public.regions': 'public.continents', + 'regions': 'public.continents', + 'public.subregions': 'public.subregions', }; /**