CLARITY

https://clarity.connorjlink.com

A complement to my custom programming language compiler, haze, clarity offers a visual presentation of the stages and transformations of the compilation process, from source code level through AST and IR symbols down to bare-metal machine code and then platform-dependent native executable files (i.e., Windows PE .exe). Clarity includes a fully-custom source code editor powered by an LSP-compliant language server written in TypeScript with dual web workers seen in this repository, which are just two constituents of my custom language server for the haze compiler. This repository also includes a web-worker-based LSIF 0.6.0-compliant indexer designed to facilitate serializing and storing language server server response data for future reference by the language server client via the source code editor.

Technical Details

The source editor interfaces with the language server client to request and receive processed symbol information to update the web DOM and provide real-time syntax and semantic highlighting for Haze source code. The client liasons with LSP language server server using standard browser web workermessage passing using JSON-RPC designed to track document versions and peridiocally synchronize symbol information with surgical precision for maximum performance--information transmits and updates only as absolutely necessary. The language server backend maintains a symbol database cache with several model representations for efficient queries and rapid response time to language server client symbol requests. It currently explicitly provides no native token recognition support (even keywords) and instead delegates to the native C++ compiler executable to compute "real" complex semantic highlighting information for macros, variables, functions, and more on the fly. I designed both the compiler and language server sever around error detection and correction with the abililty to panic-recover from extremely abnormal states and continue to provide realisitic and meaningful symbol information.

The C++ compiler runs as a separate daemon process and communications over a custom WinHTTP-based WebSocket connection to the language server server running in the browser as a web worker. It supports a wide variety of run modes and instrumentation designed to facilitate flexible requests for stage-specific information in a proprietary format designed for efficiency while retaining format characteristics that make the conversion process to JSON-RPC-compliant data members easy. For example, it permits AST-only builds, skipping executable generation to hasten object code output, and a reduced-accuracy "fuzzy" mode to find latch keypoints between which to interpolate symbol information in extreme panic cases--nested incomplete declarations, among others.

The debug server runs also as a separate daemon process written instead in unsafe Rust (due to the sheet number of low-level Windows API calls required) and similarly communicates over a WebSocket connection to the language server client via a debug adapater protocol (DAP)-compliant shim written as a web worker in TypeScript. The debugger facilitates x86 process debugging only, and it currently supports hardware and software breakpoints, single-step instruction execution using the trap flag, memory watch, and executable binary hot-patching. This functionality ties intimately with the UI and tools clarity provides.

More Information

The project remains in its infancy and will mature in its scope and goals in due time. For now, my primary goals follow as such:

• Powerful data visualization over the stages of compilation (source, preprocessed?, AST, IR, ocode, and exe)
• Complete source editor and language server client running locally in-browser compliant to LSP and LSIF
• Feature-rich debug adapter compliant to DAP

One of my long-term ideas includes running the full-stack setup in-browser hosted on my GitHub pages site. This will involve at a minimum:

• Targeting WASM output from the Haze compiler (automated through GitHub actions, probably far future since Emscripten seems far from full C++23 support?)
• Projecting multiple documents stored as serialized LSIF hosted at https://clarity.connorjlink.com