svelte-tuio

A Svelte 5 library for handling TUIO (Tangible User Interface Objects) protocol events through WebSocket connections. Built with runes and reactive state management for tracking tangible objects on interactive surfaces.

Features

• WebSocket-based TUIO Handler - Manages TUIO protocol events.
• Reactive State - Built with Svelte 5 runes for automatic reactivity.
• Tangibles Manager - Track and manage multiple tangible objects with positions and rotations.
• Debug Component - Visual debugger with real-time updates.
• Simple Hook API - Easy context-based access via useTUIO().

Overview

This library is designed to receive TUIO messages from a TUIO server (such as TouchDesigner) and make them available in your Svelte web frontend.

How It Works

1. TUIO Server - Your TouchDesigner (or other TUIO) project runs a WebSocket server
2. WebSocket Connection - Pass a WebSocket instance to TUIOProvider that connects to your TUIO server
3. Event Transmission - The TUIO server transmits touch and tangible events over the WebSocket
4. Reactive State - The library automatically updates reactive state that your Svelte components can consume

Requirements

• A TUIO server (e.g., TouchDesigner) running a WebSocket server on the same network
• The server should send TUIO events in JSON format with the following structure:

  {
    "timestamp": 123456789,
    "touchesStart": [...],
    "touchesMove": [...],
    "touchesEnd": [...],
    "touchesNoChange": [...]
  }
  

Installation

npm install svelte-tuio

Usage

Basic Setup

The simplest way to use the library is with TUIOProvider, which automatically creates a TUIOHandler for you:

<script>
    import { TUIOProvider } from 'svelte-tuio';
    import { SvelteSocket } from '@hardingjam/svelte-socket';
    let { children } = $props();

    // Create WebSocket connection to your TUIO server
    // Replace with your TouchDesigner WebSocket server address
    const svelteSocket = new SvelteSocket('ws://localhost:8080');
</script>

<TUIOProvider {svelteSocket}>
    <!-- Your app components can now access TUIO data via useTUIO() -->
    {@render children?.()}
</TUIOProvider>

With Configuration:

You can also pass configuration options directly to TUIOProvider:

<script>
    import { TUIOProvider } from 'svelte-tuio';
    import { SvelteSocket } from '@hardingjam/svelte-socket';
    let { children } = $props();

    const svelteSocket = new SvelteSocket('ws://localhost:8080');
</script>

<TUIOProvider
    {svelteSocket}
    config={{
        onFingerTouchEnd: (u, v) => {
            console.log(`Touch at ${u}, ${v}`);
        },
        throttleTime: 100
    }}
>
    {@render children?.()}
</TUIOProvider>

Pre-configured Handler:

Alternatively, you can create your own TUIOHandler with custom callbacks and pass it to the provider:

<script>
    import { TUIOProvider, TUIOHandler } from 'svelte-tuio';
    import { SvelteSocket } from '@hardingjam/svelte-socket';
    let { children } = $props();

    const svelteSocket = new SvelteSocket('ws://localhost:8080');

    const tuioHandler = new TUIOHandler({
        svelteSocket,
        onFingerTouchEnd: (u, v) => {
            console.log(`Touch at ${u}, ${v}`);
        },
        onPlaceTangible: (touch) => {
            console.log(`Tangible ${touch.classId} placed`);
        }
    });
</script>

<TUIOProvider {tuioHandler}>
    <!-- Your app components can now access TUIO data via useTUIO() -->
    {@render children?.()}
</TUIOProvider>

TouchDesigner Setup

Your TouchDesigner project needs to run a WebSocket server that sends TUIO events in the JSON format shown above.

Note: An example Python callbacks file for TouchDesigner is included in this package at src/python/example_tuio_callbacks.py

Using the Hook

Access reactive TUIO state in your components using $derived:

<script>
    import { useTUIO } from 'svelte-tuio';

    const tuioHandler = useTUIO();

    // Access reactive state with $derived
    let tangibles = $derived(tuioHandler.tangiblesManager.tangibles);
    let tangibleClassIds = $derived(tuioHandler.tangiblesManager.tangibleClassIds);
    let isConnected = $derived(tuioHandler.svelteSocket.isConnected);

    // Derived computations
    let tangibleCount = $derived(tangibles.length);
    let hasTangibles = $derived(tangibleCount > 0);
</script>

<div>
    <p>Connected: {isConnected}</p>
    <p>Active tangibles: {tangibleCount}</p>

    {#if hasTangibles}
        {#each tangibles as tangible (tangible.classId)}
            <div>
                Tangible {tangible.classId} at ({tangible.u.toFixed(2)}, {tangible.v.toFixed(2)})
            </div>
        {/each}
    {:else}
        <p>No tangibles detected</p>
    {/if}
</div>

Debug Component

<script>
    import { TangiblesDebugger } from 'svelte-tuio';
</script>

<TangiblesDebugger />

API

TUIOProvider

Wrapper component that sets up TUIO context for your app.

Props:

• tuioHandler - Optional pre-configured TUIOHandler instance. If provided, svelteSocket and config are ignored.
• svelteSocket - SvelteSocket instance from @hardingjam/svelte-socket. Required if tuioHandler is not provided.
• config - Optional configuration object for creating a TUIOHandler. Only used if tuioHandler is not provided.

Usage:

You can use TUIOProvider in two ways:

1. Automatic handler creation - Pass svelteSocket (and optionally config):

   <TUIOProvider {svelteSocket} config={{ throttleTime: 100 }}>
       <!-- Your app -->
   </TUIOProvider>
   

2. Pre-configured handler - Create your own TUIOHandler and pass it:

   <TUIOProvider {tuioHandler}>
       <!-- Your app -->
   </TUIOProvider>
   

TUIOHandler

Main class for managing TUIO WebSocket connections.

import type { TUIOHandlerConfig } from 'svelte-tuio';
import { SvelteSocket } from '@hardingjam/svelte-socket';

const svelteSocket = new SvelteSocket('ws://localhost:8080');

const config: TUIOHandlerConfig = {
    svelteSocket,
    onFingerTouchEnd: (u, v) => {
        /* ... */
    },
    onFingerTouchStart: (u, v) => {
        /* ... */
    },
    onPlaceTangible: (touch) => {
        /* ... */
    },
    onRemoveTangible: (touch) => {
        /* ... */
    },
    onMoveTangible: (touch) => {
        /* ... */
    },
    throttleTime: 100, // Throttle callbacks to once per 100ms
    touchZones: [
        // Optional: initial touch zones to register
        {
            id: 'my-zone',
            u: 0.1,
            v: 0.1,
            normalisedWidth: 0.3,
            normalisedHeight: 0.3,
            onPlaceTangible: (touch) => {
                /* ... */
            }
        }
    ]
};

const handler = new TUIOHandler(config);

// Properties
handler.tangiblesManager; // TangiblesManager instance
handler.touchZones; // Array of touch zones ($state)
handler.svelteSocket; // SvelteSocket instance

Touch Zones

Touch zones allow you to define regions of the screen and attach callbacks for touch/tangible events within those regions. Touch zones are user-managed - you register zones and implement your own logic to check if touches fall within zones.

You can register touch zones in two ways:

1. In the constructor:

import type { TouchZone, TUIOHandlerConfig } from 'svelte-tuio';

const zone: TouchZone = {
    id: 'my-zone',
    u: 0.1, // Horizontal position (0=left, 1=right)
    v: 0.1, // Vertical position in TUIO coords (0=bottom, 1=top)
    normalisedWidth: 0.3,
    normalisedHeight: 0.3,
    onPlaceTangible: (touch) => {
        console.log('Tangible placed in zone', touch);
    },
    onRemoveTangible: (touch) => {
        console.log('Tangible removed from zone', touch);
    },
    onMoveTangible: (touch) => {
        console.log('Tangible moved in zone', touch);
    },
    onTouchStart: (touch) => {
        console.log('Finger touch started in zone', touch);
    },
    onTouchMove: (touch) => {
        console.log('Finger touch moved in zone', touch);
    },
    onTouchEnd: (touch) => {
        console.log('Finger touch ended in zone', touch);
    }
};

const config: TUIOHandlerConfig = {
    svelteSocket,
    touchZones: [zone] // Register zones at creation time
};

const handler = new TUIOHandler(config);

2. Using the registerTouchZone method:

import type { TouchZone } from 'svelte-tuio';

const tuioHandler = useTUIO();

// Define a touch zone
const zone: TouchZone = {
    id: 'my-zone',
    u: 0.1, // Horizontal position (0=left, 1=right)
    v: 0.1, // Vertical position in TUIO coords (0=bottom, 1=top)
    normalisedWidth: 0.3,
    normalisedHeight: 0.3,
    onPlaceTangible: (touch) => {
        console.log('Tangible placed in zone', touch);
    },
    onRemoveTangible: (touch) => {
        console.log('Tangible removed from zone', touch);
    },
    onMoveTangible: (touch) => {
        console.log('Tangible moved in zone', touch);
    },
    onTouchStart: (touch) => {
        console.log('Finger touch started in zone', touch);
    },
    onTouchMove: (touch) => {
        console.log('Finger touch moved in zone', touch);
    },
    onTouchEnd: (touch) => {
        console.log('Finger touch ended in zone', touch);
    }
};

// Register the zone
tuioHandler.registerTouchZone(zone);

// Remove a zone when done
tuioHandler.unregisterTouchZone('my-zone');

Note: The library automatically handles hit detection. When a touch or tangible event occurs, the handler checks if it falls within any registered zone's bounds and calls the appropriate callbacks.

TangiblesManager

Manages tangible objects with reactive state. All tangible operations are handled internally by TUIOHandler.

// Public Properties (reactive $state - use with $derived)
manager.tangibles; // TUIOTouch[] - All active tangibles with full data
manager.tangibleClassIds; // number[] - Just the class IDs for performance

Example: Accessing Reactive State

<script>
    import { useTUIO } from 'svelte-tuio';

    const tuioHandler = useTUIO();
    const manager = tuioHandler.tangiblesManager;

    // Use $derived to reactively access state
    let tangibles = $derived(manager.tangibles);
    let classIds = $derived(manager.tangibleClassIds);

    // Derived computations update automatically
    let specificTangible = $derived(tangibles.find((t) => t.classId === 14));
    let hasSpecificTangible = $derived(classIds.includes(14));
</script>

{#if specificTangible}
    <p>Tangible 14 is at position: {specificTangible.u}, {specificTangible.v}</p>
{/if}

useTUIO()

Hook to access TUIOHandler from context.

const tuioHandler = useTUIO();

Custom Event Callbacks

The library provides optional callbacks for all TUIO events. By default:

• onFingerTouchEnd - Does nothing (no-op)
• onFingerTouchStart - Does nothing (no-op)
• Tangible events (onPlaceTangible, onRemoveTangible, onMoveTangible) - Automatically update TangiblesManager state, custom callbacks are called in addition to state updates

Tip: If you want finger touches to simulate clicks on HTML elements, you can implement a click handler like this:

const tuioHandler = new TUIOHandler({
    svelteSocket,
    onFingerTouchEnd: (u, v) => {
        const x = u * window.innerWidth;
        const y = (1 - v) * window.innerHeight; // Invert v for screen coordinates
        const element = document.elementFromPoint(x, y);
        if (element instanceof HTMLElement) {
            element.click();
        }
    }
});

Custom Callbacks:

You can provide custom callbacks for any TUIO event:

<script>
    import { TUIOProvider, TUIOHandler } from 'svelte-tuio';
    import { SvelteSocket } from '@hardingjam/svelte-socket';

    const svelteSocket = new SvelteSocket('ws://localhost:8080');

    const tuioHandler = new TUIOHandler({
        svelteSocket,
        onFingerTouchEnd: (u, v) => {
            console.log(`Finger touch end at: ${u}, ${v}`);
        },
        onFingerTouchStart: (u, v) => {
            console.log(`Finger touch start at: ${u}, ${v}`);
        },
        onPlaceTangible: (touch) => {
            console.log(`Tangible ${touch.classId} placed`);
        },
        onRemoveTangible: (touch) => {
            console.log(`Tangible ${touch.classId} removed`);
        },
        onMoveTangible: (touch) => {
            console.log(`Tangible ${touch.classId} moved to ${touch.u}, ${touch.v}`);
        }
    });
</script>

<TUIOProvider {tuioHandler}>
    <!-- Your app -->
</TUIOProvider>

Note: Custom tangible callbacks are called in addition to the automatic TangiblesManager updates, allowing you to add extra functionality without losing the built-in state management.

Throttling Callbacks

For performance optimization, especially with high-frequency touch events, you can throttle callback invocations using the throttleTime option:

const tuioHandler = new TUIOHandler({
    svelteSocket,
    throttleTime: 100, // Minimum 100ms between callback invocations
    onMoveTangible: (touch) => {
        // This will only be called at most once per 100ms per tangible
        console.log(`Tangible ${touch.classId} moved`);
    }
});

How it works:

• throttleTime sets the minimum time (in milliseconds) between callback invocations
• Each callback type (fingerTouchStart, fingerTouchEnd, etc.) is throttled independently
• For tangible callbacks, throttling is per tangible (by classId)
• TangiblesManager state updates are NOT throttled (state always stays up-to-date)
• Set to 0 (default) to disable throttling

Use cases:

• Throttle onMoveTangible to reduce expensive operations during dragging
• Throttle touch events to prevent UI lag from rapid touches
• Improve performance on lower-end devices

TypeScript Support

Full TypeScript support with exported types:

import type { TUIOTouch, TUIOEvent, TouchZone, TUIOHandlerConfig } from 'svelte-tuio';

Development

npm install
npm run dev
npm run test
npm run build

License

MIT