sci-vault

AI-powered collaborative platform for research data management and discovery. Built on modern microservices architecture with gRPC communication, vector embeddings, and semantic search capabilities.

Features

• User Management — Registration, authentication, email verification, password reset with rate limiting
• Smart Recommendation Engine — AI-powered document enrichment via Gemini API and vector embeddings
• Vector Search — pgvector-based semantic search across research documents
• Modern UI — SvelteKit 5 with Tailwind CSS v4, dark/light theme, multi-language support (EN/ZH-CN)
• Microservices Architecture — Scalable service design with gRPC inter-service communication
• Cloud Storage — S3-compatible object storage via RustFS
• Caching & Rate Limiting — Redis-backed rate limits and session management

Architecture

flowchart TB
    Browser([Browser])

    subgraph edge["Edge"]
        Nginx["<b>nginx</b><br/>static assets · reverse proxy"]
    end

    subgraph services["Application services"]
        direction LR
        Gateway["<b>svc-gateway</b><br/>Go · Gin"]
        Recommender["<b>svc-recommender</b><br/>Python · gRPC"]
    end

    subgraph infra["Shared infrastructure"]
        direction LR
        DB[("PostgreSQL 18<br/>+ pgvector")]
        Cache[("Redis 8.6")]
        Storage[("RustFS<br/>S3-compatible")]
    end

    Gemini{{"Google Gemini API"}}

    %% ── Main data path (user-facing request flow) ──
    Browser <==>|HTTPS| Nginx
    Nginx ==>|/api/| Gateway
    Nginx ==>|/assets/ · /private/| Storage
    Gateway <==>|gRPC| Recommender

    %% ── Service ↔ infrastructure (internal) ──
    Gateway -.-> DB
    Gateway -.-> Cache
    Gateway -.-> Storage
    Recommender -.-> DB
    Recommender -.-> Cache
    Recommender -.-> Storage

    %% ── External dependency ──
    Recommender -.->|embeddings · metadata| Gemini

    classDef svc fill:#1f2937,stroke:#60a5fa,stroke-width:1px,color:#f8fafc;
    classDef store fill:#0f172a,stroke:#34d399,stroke-width:1px,color:#f8fafc;
    classDef ext fill:#312e81,stroke:#a78bfa,stroke-width:1px,color:#f8fafc;
    classDef client fill:#111827,stroke:#f59e0b,stroke-width:1px,color:#f8fafc;
    classDef proxy fill:#064e3b,stroke:#f472b6,stroke-width:1px,color:#f8fafc;

    class Gateway,Recommender svc;
    class DB,Cache,Storage store;
    class Gemini ext;
    class Browser client;
    class Nginx proxy;

    %% Highlight main data path edges in bold black, infra/external edges dimmed
    linkStyle 0,1,2,3 stroke:#000000,stroke-width:3px;
    linkStyle 4,5,6,7,8,9,10 stroke:#475569,stroke-width:1px,stroke-dasharray:4 3;

Tech Stack

Getting Started

Prerequisites

Option 1: Docker (Recommended) — Simplest, all-in-one setup

• Docker with Docker Compose

Option 2: Local Development — Run services directly on your machine

• Buf CLI — Required only when modifying .proto files
• Go 1.26+
• Python 3.14+ with uv
• Bun 1.0+

Quick Start (Docker)

⚠️ Development Only: The provided compose file and default configs are for local development only. For production, use hardened credentials and secrets management.

1. Prepare configuration files:

cp svc-gateway/config.docker.example.yaml svc-gateway/config.docker.yaml
cp svc-recommender/config.docker.example.yaml svc-recommender/config.docker.yaml
cp frontend/nginx.example.conf frontend/nginx.conf

2. Update secrets in svc-gateway/config.docker.yaml:

3. Launch the stack:

docker compose up -d --build

4. Access the application:

All infra and backend ports are bound to the loopback interface — inter-service traffic flows over the compose network by service name, and host tools (psql, redis-cli, curl localhost:8080) still work. To reach them from another machine, either tunnel over SSH or widen the binding in docker-compose.yaml.

To stop:

docker compose down

See docker-compose.yaml for detailed service configuration.

Local Development

For developing individual services while keeping infrastructure in Docker:

1. Start infrastructure only:

docker compose up -d postgres redis rustfs rustfs-volume-helper

2. Generate gRPC stubs (if you modified .proto):

buf generate

3. Run services individually — see service-specific READMEs:

• svc-gateway — REST API, authentication, document management
• svc-recommender — Embedding generation, vector search, Gemini integration
• frontend — Web UI development guide

Project Structure

sci-vault/
├── svc-gateway/              # Go API Gateway service
│   ├── internal/             # Business logic (handlers, services, repositories)
│   ├── pkg/                  # Shared packages (auth, storage, cache)
│   └── config.*.yaml         # Configuration templates
├── svc-recommender/          # Python gRPC recommender service
│   ├── src/
│   │   ├── servicer/         # gRPC service implementation
│   │   ├── genai/            # Gemini API integration
│   │   └── repository/       # Database access
│   └── pyproject.toml        # Python dependencies (uv)
├── frontend/                 # SvelteKit web application
│   ├── src/
│   │   ├── routes/           # Page routes
│   │   ├── lib/components/   # UI components (shadcn-svelte)
│   │   └── lib/api/          # API client
│   └── nginx.conf            # Production web server config
├── proto/                    # Protocol Buffer definitions
│   └── recommender/          # Recommender service specs
├── docker-compose.yaml       # Infrastructure services
├── buf.yaml & buf.gen.yaml   # Code generation config
└── README.md

Development

Common Commands

Protocol Buffers — Only run when modifying .proto files:

buf generate

Gateway — See svc-gateway/README.md:

cd svc-gateway
go run .             # Run
go test ./...        # Test
go vet ./...         # Lint
docker build .       # Build image

Recommender — See svc-recommender/README.md:

cd svc-recommender
uv sync              # Install deps
uv run main.py       # Run
uvx ruff check .     # Lint
docker build .       # Build image

Frontend — See frontend/README.md:

cd frontend
bun install          # Install deps
bun run dev          # Dev server (localhost:5173)
bun run build        # Production build
bun run check        # Type & Svelte validation
bun run lint         # Format check

Workflow Tips

1. Configuration is runtime-mounted — Edit config.docker.yaml and restart without rebuilding:

   docker compose restart gateway recommender
   

2. Frontend nginx config — Also mounted at runtime:

   # Edit frontend/nginx.conf, then:
   docker compose restart frontend
   

3. HTTPS/TLS — Place certs at frontend/ssl/cert.pem and frontend/ssl/key.pem, uncomment in nginx.conf, then restart.

Production Deployment

⚠️ Do not deploy the default compose file to production. docker-compose.yaml and the config.docker.example.yaml templates ship with development defaults (weak passwords, no TLS, unauthenticated Redis). Infrastructure and backend ports are already bound to loopback in the baseline compose, but the remaining hardening below is still required. Production deployments must use hardened copies with rotated secrets.

1. Create a hardened compose file:

cp docker-compose.yaml docker-compose-prod.yaml

The pattern docker-compose-prod*.yaml is listed in .gitignore, so your production compose file (and any variants like docker-compose-production.yaml) will not be committed. Verify with git status before your first commit after copying.

2. Harden the compose file and service configs:

• Replace every default credential in docker-compose-prod.yaml (POSTGRES_PASSWORD, RUSTFS_ACCESS_KEY, RUSTFS_SECRET_KEY, etc.) with strong, unique secrets — ideally injected from a secrets manager or .env file rather than committed inline.
• Mirror the same secrets in svc-gateway/config.docker.yaml and svc-recommender/config.docker.yaml (both are gitignored per the service-level .gitignore files).
• Rotate jwt.secret to a fresh high-entropy value.
• Enable Redis authentication — add --requirepass <strong-password> to the redis service's command, and mirror the password in the gateway/recommender configs. The baseline compose leaves Redis unauthenticated (acceptable only because the port is loopback-only).
• Tighten RUSTFS_CORS_ALLOWED_ORIGINS / RUSTFS_CONSOLE_CORS_ALLOWED_ORIGINS to your actual frontend origin (e.g. https://your-domain.example) instead of the loopback defaults.
• Verify the loopback port bindings still match your topology. The baseline binds postgres, redis, rustfs (9000/9001), recommender, and gateway to 127.0.0.1 — appropriate when everything runs on one host behind the frontend nginx. If you split services across hosts, widen only the specific bindings needed and front them with a firewall or private network.
• Consider disabling the RustFS console (RUSTFS_CONSOLE_ENABLE=false) unless you actively need it.
• Configure TLS on the frontend: place certs at frontend/ssl/cert.pem / frontend/ssl/key.pem and enable the HTTPS server block in nginx.conf.

3. Launch:

docker compose -f docker-compose-prod.yaml up -d --build

4. Verify nothing sensitive is tracked:

git status --ignored

The hardened compose file and config.docker.yaml files should appear under Ignored files, never Untracked or Changes.

Documentation

• Gateway API — Endpoints, authentication, configuration
• Recommender Engine — Gemini integration, embeddings, gRPC specs
• Frontend — UI components, SvelteKit setup, i18n

License

This project is licensed under the MIT License.

Contributors

Thanks to all our contributors! View the full contributor list.