sci-vault

sci-vault is an AI-powered collaborative platform for intelligent management and discovery of laboratory research data. The system leverages modern microservices architecture with gRPC communication, embedding-based recommendations, and vector search capabilities to provide researchers with powerful tools for data exploration and insight extraction.

Overview

This monorepo contains a complete microservices-based application for research data management:

• svc-gateway: REST API gateway built with Go and Gin. Handles user authentication (registration, login, email verification, password reset), profile management, avatar uploads, JWT middleware, and routes recommendation requests to svc-recommender via gRPC.
• svc-recommender: gRPC-based recommendation engine built with Python, utilizing embedding models and pgvector for intelligent content discovery.
• frontend: Modern SvelteKit (Svelte 5) web application with dark/light theme, i18n support (en / zh-CN), and full integration with the gateway API.

Technology Stack

Prerequisites

For Docker Deployment (Recommended):

• Docker with Docker Compose

For Manual Local Development:

• Buf — Only needed when .proto files are changed
• Go 1.26+ — For the gateway service
• Python 3.14+ with uv — For the recommender service
• Bun 1.0+ — For the frontend application

Quick Start Using Docker Compose

⚠️ WARNING for Production environments: The provided docker-compose.yaml and default configurations are intended for local development and testing only. When deploying to a production environment, you MUST use your own secure parameters, strong passwords, and proper secrets management. It is highly recommended to create and use a dedicated docker-compose-production.yaml with hardened configurations.

A docker-compose.yaml is provided at the root to spin up the entire application (Frontend, Gateway, Recommender) along with all required infrastructure services (PostgreSQL, Redis, RustFS) in one command.

Note: Generated gRPC stubs are committed in this repository. Docker builds no longer generate stubs during image build. Run buf generate only when you modify .proto files.

1. Prepare the configuration files for each service:

cp svc-gateway/config.docker.example.yaml svc-gateway/config.docker.yaml
cp svc-recommender/config.docker.example.yaml svc-recommender/config.docker.yaml
cp frontend/nginx.example.conf frontend/nginx.conf

2. Open svc-gateway/config.docker.yaml and fill in your secrets:

RustFS S3 credentials: RustFS uses the same RUSTFS_ACCESS_KEY/RUSTFS_SECRET_KEY values from docker-compose.yaml directly as its S3 API credentials — no need to log into the web console to generate separate keys. For local development the defaults (rustfsadmin / rustfsadmin) work out of the box.

3. Run the cluster:

# For Local Development
docker compose up -d --build

# For Production (using production-specific config)
cp docker-compose.yaml docker-compose-production.yaml # If you haven't created one yet
# Edit docker-compose-production.yaml and svc-gateway/config.docker.yaml with production-grade secrets
docker compose -f docker-compose-production.yaml up -d --build

This starts the following containers:

Updating configuration: config.docker.yaml is mounted into each service container at runtime (not baked into the image). After editing, only a restart is needed — no rebuild:

docker compose restart gateway recommender
# or for production:
docker compose -f docker-compose-production.yaml restart gateway recommender

Frontend nginx config: frontend/nginx.conf (copied from nginx.example.conf) is mounted at runtime. You can edit it (e.g. tune proxy settings, add security headers) and restart without rebuilding:

docker compose restart frontend

HTTPS / TLS: To enable HTTPS for the frontend:

1. Place your certificate and private key at frontend/ssl/cert.pem and frontend/ssl/key.pem
2. Uncomment the listen 443 ssl block and certificate directives in frontend/nginx.conf
3. Update the server_name in frontend/nginx.conf to your actual domain
4. Restart the frontend container: docker compose restart frontend

The frontend/ssl/ directory is mounted read-only into the container at /etc/nginx/ssl/. The directory is git-ignored — never commit your private key.

To stop the infrastructure:

# For Local Development
docker compose down

# For Production
docker compose -f docker-compose-production.yaml down

Manual Local Development

If you prefer to run the services directly on your host machine instead of using Docker, follow these steps.

1. Start Infrastructure Dependencies

If you prefer to develop services locally while running only the back-end infrastructure (DB, Redis, S3) in Docker:

docker compose up -d postgres redis rustfs # Add rustfs-volume-helper if first time

2. Generate gRPC Code (Only If You Changed .proto)

This step is optional unless you changed files in proto/. Make sure Buf is installed.

buf generate

Important: Re-run this command whenever you modify any .proto files in the proto/ directory.

3. Set Up Individual Services

Each service has its own setup and runtime requirements. Navigate to the respective directories and follow their specific README:

svc-gateway (API Gateway)

cd svc-gateway
cp config.example.yaml config.yaml  # then fill in your values
go mod tidy
go run .

See svc-gateway/README.md for the full configuration reference and API endpoint documentation.

svc-recommender (Recommender Engine)

cd svc-recommender
uv sync
uv run --env-file .env main.py

See svc-recommender/README.md for detailed instructions.

frontend (Web Client)

cd frontend
bun install
bun run dev

See frontend/README.md for detailed instructions.

Project Structure

sci-vault/
├── proto/                  # Protocol buffer definitions for gRPC
│   └── recommender/        # Recommender service proto
├── svc-gateway/            # Go-based API gateway
├── svc-recommender/        # Python-based recommendation engine
├── frontend/               # SvelteKit web application
├── docker-compose.yaml     # Infrastructure services (PostgreSQL, Redis, RustFS)
├── buf.yaml                # Buf configuration for code generation
├── buf.gen.yaml            # Buf generation settings
└── README.md               # This file

Development Workflow

1. Start infrastructure: docker compose up -d
2. Update Proto Definitions (optional): Only if you modify files in proto/, run buf generate
3. Develop Services: Make changes to individual service code
4. Run Services: Start svc-recommender, svc-gateway, and frontend independently

Service Communication

Browser
  │  REST/HTTP
  ▼
svc-gateway ──gRPC──► svc-recommender
  │
  ├── PostgreSQL  (user data, profiles)
  ├── Redis       (email verification codes, cache)
  └── RustFS      (avatar & asset storage)

Roadmap

• Additional recommendation algorithms and personalization features
• Enhanced CI/CD pipeline with automated testing and building

License

This project is licensed under the LICENSE file in the root directory.