sci-vault

sci-vault is an AI-powered collaborative platform for intelligent management and discovery of laboratory research data. The system leverages modern microservices architecture with gRPC communication, embedding-based recommendations, and vector search capabilities to provide researchers with powerful tools for data exploration and insight extraction.

Overview

This monorepo contains a complete microservices-based application for research data management:

• svc-gateway: REST API gateway built with Go and Gin. Handles user authentication (registration, login, email verification, password reset), profile management, avatar uploads, JWT middleware, and routes recommendation requests to svc-recommender via gRPC.
• svc-recommender: gRPC-based recommendation engine built with Python, utilizing embedding models and pgvector for intelligent content discovery.
• frontend: Modern SvelteKit (Svelte 5) web application with dark/light theme, i18n support (en / zh-CN), and full integration with the gateway API.

Technology Stack

Prerequisites

Before getting started, ensure you have the following tools installed:

• Buf — For code generation from protobuf definitions
• Go 1.26+ — For the gateway service
• Python 3.10+ with uv — For the recommender service
• Bun 1.0+ — For the frontend application
• Docker with Docker Compose — For running infrastructure services

Quick Start

1. Generate gRPC Code

Generate the necessary gRPC stubs for both services using Buf. This step is required before running the application:

buf generate

This reads the protobuf definitions from the proto/ directory and generates the required stubs for svc-gateway and svc-recommender. Re-run this command whenever you modify any .proto files.

2. Start Everything with Docker Compose

⚠️ WARNING for Production environments: The provided docker-compose.yaml and default configurations are intended for local development and testing only. When deploying to a production environment, you MUST use your own secure parameters, strong passwords, and proper secrets management. It is highly recommended to create and use a dedicated docker-compose-production.yaml with hardened configurations.

A docker-compose.yaml is provided at the root to spin up the entire application (Frontend, Gateway, Recommender) along with all required infrastructure services (PostgreSQL, Redis, RustFS) in one command.

First, prepare the configuration files for each service:

cp svc-gateway/config.docker.example.yaml svc-gateway/config.docker.yaml
cp svc-recommender/config.docker.example.yaml svc-recommender/config.docker.yaml
cp frontend/nginx.example.conf frontend/nginx.conf

Then open svc-gateway/config.docker.yaml and fill in your secrets:

RustFS S3 credentials: RustFS uses the same RUSTFS_ACCESS_KEY/RUSTFS_SECRET_KEY values from docker-compose.yaml directly as its S3 API credentials — no need to log into the web console to generate separate keys. For local development the defaults (rustfsadmin / rustfsadmin) work out of the box.

Then run the cluster:

# For Local Development
docker compose up -d --build

# For Production (using production-specific config)
cp docker-compose.yaml docker-compose-production.yaml # If you haven't created one yet
# Edit docker-compose-production.yaml and svc-gateway/config.docker.yaml with production-grade secrets
docker compose -f docker-compose-production.yaml up -d --build

This starts the following containers:

Updating configuration: config.docker.yaml is mounted into each service container at runtime (not baked into the image). After editing, only a restart is needed — no rebuild:

docker compose restart gateway recommender
# or for production:
docker compose -f docker-compose-production.yaml restart gateway recommender

Frontend nginx config: frontend/nginx.conf (copied from nginx.example.conf) is mounted at runtime. You can edit it (e.g. tune proxy settings, add security headers) and restart without rebuilding:

docker compose restart frontend

HTTPS / TLS: To enable HTTPS for the frontend:

1. Place your certificate and private key at frontend/ssl/cert.pem and frontend/ssl/key.pem
2. Uncomment the listen 443 ssl block and certificate directives in frontend/nginx.conf
3. Update the server_name in frontend/nginx.conf to your actual domain
4. Restart the frontend container: docker compose restart frontend

The frontend/ssl/ directory is mounted read-only into the container at /etc/nginx/ssl/. The directory is git-ignored — never commit your private key.

To stop the infrastructure:

# For Local Development
docker compose down

# For Production
docker compose -f docker-compose-production.yaml down

3. Set Up Local Infrastructure only (Optional)

If you prefer to develop services locally while running only the back-end infrastructure (DB, Redis, S3) in Docker:

# For Local Development
docker compose up -d postgres redis rustfs # add rustfs-volume-helper if first time to initialize the volume

# For Production
docker compose -f docker-compose-production.yaml up -d postgres redis rustfs

4. Set Up Individual Services for Local Development (Optional)

Each service has its own setup and runtime requirements. Navigate to the respective directories and follow their specific README:

svc-gateway (API Gateway)

cd svc-gateway
cp config.example.yaml config.yaml  # then fill in your values
go mod tidy
go run .

See svc-gateway/README.md for the full configuration reference and API endpoint documentation.

svc-recommender (Recommender Engine)

cd svc-recommender
uv sync
uv run --env-file .env main.py

See svc-recommender/README.md for detailed instructions.

frontend (Web Client)

cd frontend
bun install
bun run dev

See frontend/README.md for detailed instructions.

Project Structure

sci-vault/
├── proto/                  # Protocol buffer definitions for gRPC
│   └── recommender/        # Recommender service proto
├── svc-gateway/            # Go-based API gateway
├── svc-recommender/        # Python-based recommendation engine
├── frontend/               # SvelteKit web application
├── docker-compose.yaml     # Infrastructure services (PostgreSQL, Redis, RustFS)
├── buf.yaml                # Buf configuration for code generation
├── buf.gen.yaml            # Buf generation settings
└── README.md               # This file

Development Workflow

1. Start infrastructure: docker compose up -d
2. Update Proto Definitions: Modify files in proto/ as needed, then run buf generate
3. Develop Services: Make changes to individual service code
4. Run Services: Start svc-recommender, svc-gateway, and frontend independently

Service Communication

Browser
  │  REST/HTTP
  ▼
svc-gateway ──gRPC──► svc-recommender
  │
  ├── PostgreSQL  (user data, profiles)
  ├── Redis       (email verification codes, cache)
  └── RustFS      (avatar & asset storage)

Roadmap

• Additional recommendation algorithms and personalization features
• Enhanced CI/CD pipeline with automated testing and building

License

This project is licensed under the LICENSE file in the root directory.