ChaosConnect

The modern, distributed and scalable implementation of a game inspired by Connect Four.

Gameplay

Rules

• There are two main factions: Yellow vs Red
• There's only one big playing field for all players
• Each player can make a move after a certain timeout
• Other players cannot place a chip close to another chip for a certain timeout
• If more people join, the timeouts are altered, and the playing field scales automatically
  • More people => make playing field wider
  • Fewer people => Mark (now superfluous) rows and columns as soon to be deleted, delete after a certain time passed
• If a disk you placed is part of a 4 disk long line you get points, and the associated columns are cleared

Future Plans

• You get more points if you're on the losing faction for balance purposes
• You can exchange your points for perks and skins
• To ensure fairness, some bots might be added to balance the teams

Architecture

Big Picture

Separation of Concerns

The service names are all a reference to the popular anime JoJo's Bizarre Adventure.

Name	Role	Description
Doppio	Frontend	Svelte-based web client
Speedwagon	Load balancer	Envoy-based load balancing reverse proxy
Joestar	Scaling backend	Micronaut server for user authentication and caching
Rohan	Central backend	Micronaut server for central storage and processing of the game and its users

Frontend

The frontend is written in Svelte. As it's main purpose is to display the current state of the board, we decided that frameworks such as Angular are overkill.

Load balancing

We use Envoy as a reverse proxy to handle load balancing. We had to use Envoy as it's the only reverse proxy which currently supports grpc-web.

Backend

The backend is split into two parts:

• Scaling: Communicates directly with our Frontend, issues and validates JWT, caches game state and sends game updates to all clients
• Central: Manages the actual game state, synchronizes requests and handles game logic, stores persistent information, such as user credentials and scores, in a json document

Both backends use Micronaut with Kotlin.

Storage

We decided not to use a database but instead store the (very minimalistic) data in a JSON document.

Communication

Bi-directionality

Bidirectional communication is enabled through gRPC. For example, this allows Rohan to send a game update event to all Joestar instances, which then forward them realtime to all Doppio clients.

Authentication

Because we use gRPC for communication, which is based on HTTP, we send the authentication token in the Authorization Header. We use a custom solution based on symmetric JWT for authentication, as we did not want to commit to a vendor-specific or work-in-progress solution (see Symmetric JWT for details).

Control flow

We use streaming to propagate game state updates from Rohan to Joestar servers and from Joestar servers to Doppio clients. All other communication is request-based.

Asynchronicity

The streaming API offers hooks for various events. The request-based API is asynchronous by nature as well, but allows for convenient programming styles that are similar to those common in synchronous contexts.

Control Flow	Kotlin	TypeScript
Streams	kotlinx.coroutines.flow.Flow + callback methods	ClientReadableStream + callback methods
'Synchronous' Requests	suspend fun + kotlinx.coroutines.BuildersKt.runBlocking	Promise + async + await

JWT

JWTs are issued and terminated by Joestar servers.

Configuration

We use Micronaut Application Configuration to make some parts of the application configurable. These configurations can be set through the application.yml file at compile time or through an environment variable at runtime. The following configurations are probably the most interesting ones to configure, for a full list see the source code:

• Joestar
  • JOESTAR_PORT: The port at which Rohan is listening
  • ROHAN_SERVER: The hostname or ip of the Rohan server
  • ROHAN_PORT: The port at which Rohan is listening
  • JWT_SECRET: The base-64 encoded 512-bit private key used for signing the JWT
• Rohan
  • ROHAN_PORT: The port at which Rohan is listening

Maintainability

We use state-of-the-art technology and are using the latest version to take advantage of the latest language and framework features. For example, many Kotlin coroutine features we rely upon to easily implement real time updates were released as stable May 14th, just in time for the v1 release.

We try to avoid unnecessary code cohesion and favor testability instead. Currently, we have more than 200 unit tests.

Infrastructure

Docker setup

In order to enable easy deployment every service is dockerized. We do not use docker containers for developing, but you can easily build the containers locally to test their cross-container communication. We also use docker to run gRPC code generation for grpc-web to ensure the code gets generated with the same compiler version on every device.

The docker images are generally optimized for file size and try to use the smallest available base image.

Our images also implement Docker Healthchecks which can be used to determine if a server is irrecoverably broken.

Reverse Proxy

We use a dockerized Envoy reverse proxy server as a load balancer (see Envoy for details).

CI/CD

We use GitHub Actions to test and build ChaosConnect automatically. Every push is tested and built and Pull Requests can only be merged if all tests succeed. Creating a new Release automatically builds and publishes all required Packages such that users can run ChaosConnect with a specific version without needing to build the images themselves.

Hosting

At the time of writing this, the latest stable version of ChaosConnect is deployed at chaos.honegger.dev. We decided to use the cloud provider Linode because they provide fair pricing and a good free initial credit.

Setting up hosting was very easy, you just copy the docker-compose.yml file, replace the placeholder jwt-secret with a generated one and you're almost ready to start. If you don't already have a valid certificate around, you can easily generate one using Certbot using a command similar to the following and mounting them to their corresponding location within the Speedwagon container:

sudo docker run -it --rm --name certbot -v "/etc/letsencrypt:/etc/letsencrypt" -v "/var/lib/letsencrypt:/var/lib/letsencrypt" -p 80:80 certbot/certbot certonly --standalone

Design Decisions

gRPC

We wanted real time updates and gRPC provides a way to stream real time updates. Implementing real time update from server-to-server and server-to-browser is no easy task and implementing a type safe variant through websockets would have taken a lot longer than just using a gRPC library.

Micronaut

Micronaut offers a couple of benefits:

• It supports gRPC out-of-the-box
• It offers Configuration Management
• It provides fast startup and performance by using build-time dependency injection

Kotlin

Kotlin is a modern, concise, and null safe programming language, which offer great language and ecosystem support for gRPC. For example, implementing parallel code using Kotlin coroutines is vastly easier than using plain old java.

Svelte

Svelte is a modern, light-weight SPA framework that meets the needs of small projects like ours.

Envoy

Envoy is the only reverse proxy with official and native grpc-web support known to us.

Envoy uses round-robin for load balancing and only considers servers for load balancing which fulfill a readiness check. For example, a Joestar instance without a valid rohan connection is not considered ready and will not be load-balanced.

Symmetric JWT

We couldn't easily use the micronaut-security package because the feature is still WIP for gRPC (see micronaut-grpc issue #164 for details). The official token-based authentication works by using Google as a token provider, but we didn't want to have a vendor lock-in.

In the end, we decided to use simple symmetric tokens because of the project scope.

The client does use the metadata to send the token which was recommended back in 2018 by the grpc-web team.

Notifications

In order to preserve network bandwidth, which can be pricey depending on the hosting provider, we use real-time updates with light-weight change events instead of complete states.

Operation

The easiest way to get ChaosConnect running is using docker-compose. We do not provide support for running the software components otherwise.

Prerequisites

The docker-compose.yml file contains a placeholder for the JWT signing key. This base64-encoded 512 bit secret is crucial for verifying user authenticity and must be configured before starting.

Another crucial point are HTTPS certificates, which have to be mounted to /etc/envoy/certs/cc.key and /etc/envoy/certs/cc.cert within the container. If you're running on localhost, see the development guide below on how to generate self-signed certificates.

Starting

# Run services under https://localhost/ using images published to GitHub Packages
# Valid certificates are expected to be placed under ./certs/cc.key and ./certs/cc.cert
# Valid environment variables have to be configured 
docker-compose up --scale joestar=5 -d

Development

Prerequisites

For local development JDK 16, Node 16 and a modern version of docker need to be installed. Our project also contains run configurations for IntelliJ, which is our IDE of choice.

Getting started

In order to run components natively in development mode, the following commands are good to get started:

# Required once at the beginning and afterwards once the protocol buffer contract changes
docker-compose -f docker-compose.gen.yml up gen_grpc_joestar_client

# Run proxy (proxies http://localhost:5001 => http://localhost:5000 and http://localhost:5001/api => http://localhost:8080/)

# (Docker for Windows / Docker for Mac)
docker-compose -f docker-compose.proxy.yml up -d
# (Docker on Linux)
docker-compose -f docker-compose.proxy-linux.yml up -d

# Svelte Frontend
cd frontend
npm run dev

# backend
cd backend
# run joestar
./gradlew joestar:run
# run rohan
./gradlew rohan:run

Or if you want to test the release configuration locally, you can build and run them locally:

# Generate self-signed certs, puts them in the ./proxy/certs directory
# Ensure the permissions are set such that the envoy docker user can read the certificate
docker-compose -f docker-compose.gen.yml up gen_self_signed_cert

# Run services hosted under http://localhost:5001, built locally, run 2 joestar instances
docker-compose -f docker-compose.dev.yml up --build --scale joestar=2