Spaces:
Running
Running
| title: RobotHub Arena Frontend | |
| tags: | |
| - robotics | |
| - control | |
| - simulation | |
| - svelte | |
| - frontend | |
| - realtime | |
| emoji: 🤖 | |
| colorFrom: blue | |
| colorTo: purple | |
| sdk: docker | |
| app_port: 8000 | |
| pinned: true | |
| license: mit | |
| fullWidth: true | |
| short_description: Web interface of the RobotHub platform | |
| # 🤖 RobotHub Arena – Frontend | |
| RobotHub is an **open-source, end-to-end robotics stack** that combines real-time communication, 3-D visualisation, and modern AI policies to control both simulated and physical robots. | |
| **This repository contains the *Frontend*** – a SvelteKit web application that runs completely in the browser (or inside Electron / Tauri). It talks to two backend micro-services that live in their own repositories: | |
| 1. **[RobotHub Transport Server](https://github.com/julien-blanchon/RobotHub-TransportServer)** | |
| – WebSocket / WebRTC switch-board for video streams & robot joint messages. | |
| 2. **[RobotHub Inference Server](https://github.com/julien-blanchon/RobotHub-InferenceServer)** | |
| – FastAPI service that loads large language- and vision-based policies (ACT, Pi-0, SmolVLA, …) and turns camera images + state into joint commands. | |
| ```text | |
| ┌────────────────────┐ ┌────────────────────────┐ ┌──────────────────────────┐ | |
| │ RobotHub Frontend │ HTTP │ Transport Server │ WebSocket │ Robot / Camera HW │ | |
| │ (this repo) │ <────► │ (rooms, WS, WebRTC) │ ◄──────────►│ – servo bus, USB… │ | |
| │ │ └────────────────────────┘ └──────────────────────────┘ | |
| │ 3-D scene (Threlte)│ | |
| │ UI / Settings │ ┌────────────────────────┐ | |
| │ Svelte 5 runes │ HTTP │ Inference Server │ HTTP/WS │ GPU (Torch, HF models) │ | |
| └────────────────────┘ <────► │ (FastAPI, PyTorch) │ ◄──────────►└──────────────────────────┘ | |
| └────────────────────────┘ | |
| ``` | |
| --- | |
| ## ✨ Key Features | |
| • **Digital-Twin 3-D Scene** – inspect robots, cameras & AI compute blocks in real-time. | |
| • **Multi-Workspace Collaboration** – share a hash URL and others join the *same* WS rooms instantly. | |
| • **Drag-&-Drop Add-ons** – spawn robots, cameras or AI models from the toolbar. | |
| • **Transport-Agnostic** – control physical hardware over USB, or send/receive via WebRTC rooms. | |
| • **Model Agnostic** – any policy exposed by the Inference Server can be used (ACT, Diffusion, …). | |
| • **Reactive Core** – built with *Svelte 5 runes* – state is automatically pushed into the UI. | |
| --- | |
| ## 📂 Repository Layout (short) | |
| | Path | Purpose | | |
| |-------------------------------|---------| | |
| | `src/` | SvelteKit app (routes, components) | | |
| | `src/lib/elements` | Runtime domain logic (robots, video, compute) | | |
| | `external/RobotHub-*` | Git sub-modules for the backend services – used for generated clients & tests | | |
| | `static/` | URDFs, STL meshes, textures, favicon | | |
| A more in-depth component overview can be found in `/src/lib/components/**` – every major popup/modal has its own Svelte file. | |
| --- | |
| ## 🚀 Quick Start (dev) | |
| ```bash | |
| # 1. clone with submodules (transport + inference) | |
| $ git clone --recurse-submodules https://github.com/julien-blanchon/RobotHub-Frontend robothub-frontend | |
| $ cd robothub-frontend | |
| # 2. install deps (uses Bun) | |
| $ bun install | |
| # 3. start dev server (http://localhost:5173) | |
| $ bun run dev -- --open | |
| ``` | |
| ### Running the full stack locally | |
| ```bash | |
| # 1. start Transport Server (rooms & streaming) | |
| $ cd external/RobotHub-InferenceServer/external/RobotHub-TransportServer/server | |
| $ uv run launch_with_ui.py # → http://localhost:8000 | |
| # 2. start Inference Server (AI brains) | |
| $ cd ../../.. | |
| $ python launch_simple.py # → http://localhost:8001 | |
| # 3. frontend (separate terminal) | |
| $ bun run dev -- --open # → http://localhost:5173 (hash = workspace-id) | |
| ``` | |
| The **workspace-id** in the URL hash ties all three services together. Share `http://localhost:5173/#<id>` and a collaborator instantly joins the same set of rooms. | |
| --- | |
| ## 🛠️ Usage Walk-Through | |
| 1. **Open the web-app** → a fresh *workspace* is created (☝ left corner shows 🌐 ID). | |
| 2. Click *Add Robot* → spawns an SO-100 6-DoF arm (URDF). | |
| 3. Click *Add Sensor → Camera* → creates a virtual camera element. | |
| 4. Click *Add Model → ACT* → spawns a *Compute* block. | |
| 5. On the Compute block choose *Create Session* – select model path (`./checkpoints/act_so101_beyond`) and cameras (`front`). | |
| 6. Connect: | |
| • *Video Input* – local webcam → `front` room. | |
| • *Robot Input* – robot → *joint-input* room (producer). | |
| • *Robot Output* – robot ← AI predictions (consumer). | |
| 7. Press *Start Inference* – the model will predict the next joint trajectory every few frames. 🎉 | |
| All modals (`AISessionConnectionModal`, `RobotInputConnectionModal`, …) expose precisely what is happening under the hood: which room ID, whether you are *producer* or *consumer*, and the live status. | |
| --- | |
| ## 🧩 Package Relations | |
| | Package | Role | Artifacts exposed to this repo | | |
| |---------|------|--------------------------------| | |
| | **Transport Server** | Low-latency switch-board (WS/WebRTC). Creates *rooms* for video & joint messages. | TypeScript & Python client libraries (imported from sub-module) | | |
| | **Inference Server** | Loads checkpoints (ACT, Pi-0, …) and manages *sessions*. Each session automatically asks the Transport Server to create dedicated rooms. | Generated TS SDK (`@robothub/inference-server-client`) – auto-called from `RemoteComputeManager` | | |
| | **Frontend (this repo)** | UI + 3-D scene. Manages *robots*, *videos* & *compute* blocks and connects them to the correct rooms. | – | | |
| > Because the two backend repos are included as git sub-modules you can develop & debug the whole trio in one repo clone. | |
| --- | |
| ## 📜 Important Components (frontend) | |
| • `RemoteComputeManager` – wraps the Inference Server REST API. | |
| • `RobotManager` – talks to Transport Server and USB drivers. | |
| • `VideoManager` – handles local/remote camera streams and WebRTC. | |
| Each element is a small class with `$state` fields which Svelte 5 picks up automatically. The modals listed below are *thin* UI shells around those classes: | |
| ``` | |
| AISessionConnectionModal – create / start / stop AI sessions | |
| RobotInputConnectionModal – joint-states → AI | |
| RobotOutputConnectionModal – AI commands → robot | |
| VideoInputConnectionModal – camera → AI or screen | |
| ManualControlSheet – slider control, runs when no consumer connected | |
| SettingsSheet – configure base URLs of the two servers | |
| ``` | |
| --- | |
| ## 🐳 Docker | |
| A production-grade image is provided (multi-stage, 24 MB with Bun runtime): | |
| ```bash | |
| $ docker build -t robothub-frontend . | |
| $ docker run -p 8000:8000 robothub-frontend # served by vite-preview | |
| ``` | |
| See `Dockerfile` for the full build – it also performs `bun test` & `bun run build` for the TS clients inside the sub-modules so that the image is completely self-contained. | |
| --- | |
| ## 🧑💻 Contributing | |
| PRs are welcome! The codebase is organised into **domain managers** (robot / video / compute) and **pure-UI** components. If you add a new feature, create a manager first so that business logic can be unit-tested without DOM. | |
| 1. `bun test` – unit tests. | |
| 2. `bun run typecheck` – strict TS config. | |
| Please run `bun format` before committing – ESLint + Prettier configs are included. | |
| --- | |
| ## 🙏 Special Thanks | |
| Huge gratitude to [Tim Qian](https://github.com/timqian) ([X/Twitter](https://x.com/tim_qian)) and the | |
| [bambot project](https://bambot.org/) for open-sourcing **feetech.js** – the | |
| delightful js driver that powers our USB communication layer. | |
| --- | |
| ## 📄 License | |
| MIT – see `LICENSE` in the root. | |
| ## 🌱 Project Philosophy | |
| RobotHub follows a **separation-of-concerns** design: | |
| * **Transport Server** is the single source of truth for *real-time* data – video frames, joint values, heart-beats. Every participant (browser, Python script, robot firmware) only needs one WebSocket/WebRTC connection, no matter how many peers join later. | |
| * **Inference Server** is stateless with regard to connectivity; it spins up / tears down *sessions* that rely on rooms in the Transport Server. This lets heavy AI models live on a GPU box while cameras and robots stay on the edge. | |
| * **Frontend** stays 100 % in the browser – no secret keys or device drivers required – and simply wires together rooms that already exist. | |
| > By decoupling the pipeline we can deploy each piece on separate hardware or even different clouds, swap alternative implementations (e.g. ROS bridge instead of WebRTC) and scale each micro-service independently. | |
| --- | |
| ## 🛰 Transport Server – Real-Time Router | |
| ``` | |
| Browser / Robot ⟷ 🌐 Transport Server ⟷ Other Browser / AI / HW | |
| ``` | |
| * **Creates rooms** – `POST /robotics/workspaces/{ws}/rooms` or `POST /video/workspaces/{ws}/rooms`. | |
| * **Manages roles** – every WebSocket identifies as *producer* (source) or *consumer* (sink). | |
| * **Does zero processing** – it only forwards JSON (robotics) or WebRTC SDP/ICE (video). | |
| * **Health-check** – `GET /api/health` returns a JSON heartbeat. | |
| Why useful? | |
| * You never expose robot hardware directly to the internet – it only speaks to the Transport Server. | |
| * Multiple followers can subscribe to the *same* producer without extra bandwidth on the producer side (server fans out messages). | |
| * Works across NAT thanks to WebRTC TURN support. | |
| ## 🏢 Workspaces – Lightweight Multi-Tenant Isolation | |
| A **workspace** is simply a UUID namespace in the Transport Server. Every room URL starts with: | |
| ``` | |
| /robotics/workspaces/{workspace_id}/rooms/{room_id} | |
| /video/workspaces/{workspace_id}/rooms/{room_id} | |
| ``` | |
| Why bother? | |
| 1. **Privacy / Security** – clients in workspace *A* can neither list nor join rooms from workspace *B*. A workspace id is like a private password that keeps the rooms in the same workspace isolated from each other. | |
| 2. **Organisation** – keep each class, project or experiment separated without spinning up extra servers. | |
| 3. **Zero-config sharing** – the Frontend stores the workspace ID in the URL hash (e.g. `/#d742e85d-c9e9-4f7b-…`). Send that link to a teammate and they automatically connect to the *same* namespace – all existing video feeds, robot rooms and AI sessions become visible. | |
| 4. **Stateless Scale-out** – Transport Server holds no global state; deleting a workspace removes all rooms in one call. | |
| Typical lifecycle: | |
| * **Create** – Frontend generates `crypto.randomUUID()` if the hash is empty. Back-end rooms are lazily created when the first producer/consumer calls the REST API. | |
| * **Share** – click the *#workspace* badge → *Copy URL* (handled by `WorkspaceIdButton.svelte`) | |
| > Practical tip: Use one workspace per demo to prevent collisions, then recycle it afterwards. | |
| --- | |
| ## 🧠 Inference Server – Session Lifecycle | |
| 1. **Create session** | |
| `POST /api/sessions` with JSON: | |
| ```jsonc | |
| { | |
| "session_id": "pick_place_demo", | |
| "policy_path": "./checkpoints/act_so101_beyond", | |
| "camera_names": ["front", "wrist"], | |
| "transport_server_url": "http://localhost:8000", | |
| "workspace_id": "<existing-or-new>" // optional | |
| } | |
| ``` | |
| 2. **Receive response** | |
| ```jsonc | |
| { | |
| "workspace_id": "ws-uuid", | |
| "camera_room_ids": { "front": "room-id-a", "wrist": "room-id-b" }, | |
| "joint_input_room_id": "room-id-c", | |
| "joint_output_room_id": "room-id-d" | |
| } | |
| ``` | |
| 3. **Wire connections** | |
| * Camera PC joins `front` / `wrist` rooms as **producer** (WebRTC). | |
| * Robot joins `joint_input_room_id` as **producer** (joint states). | |
| * Robot (or simulator) joins `joint_output_room_id` as **consumer** (commands). | |
| 4. **Start inference** | |
| `POST /api/sessions/{id}/start` – server loads the model and begins publishing commands. | |
| 5. **Stop / delete** as needed. Stats & health are available via `GET /api/sessions`. | |
| The Frontend automates steps 1-4 via the *AI Session* modal – you only click buttons. | |
| --- | |
| ## 🌐 Hosted Demo End-Points | |
| | Service | URL | Status | | |
| |---------|-----|--------| | |
| | Transport Server | <https://blanchon-robothub-transportserver.hf.space/api> | Public & healthy | | |
| | Inference Server | <https://blanchon-robothub-inferenceserver.hf.space/api> | `{"status":"healthy"}` | | |
| | Frontend (read-only preview) | <https://blanchon-robothub-frontend.hf.space> | latest `main` | | |
| Point the *Settings → Server Configuration* panel to these URLs and you can play without any local backend. | |
| --- | |
| ## 🎯 Main Use-Cases | |
| Below are typical connection patterns you can set-up **entirely from the UI**. Each example lists the raw data-flow (→ = producer to consumer/AI) plus a video placeholder you can swap for a screen-capture. | |
| ### Direct Tele-Operation (Leader ➜ Follower) | |
| *Leader PC* `USB` ➜ **Robot A** ➜ `Remote producer` → **Transport room** → `Remote consumer` ➜ **Robot B** (`USB`) | |
| > One human moves Robot A, Robot B mirrors the motion in real-time. Works with any number of followers – just add more consumers to the same room. | |
| > | |
| > 📺 *demo-teleop-1.mp4* | |
| ### Web-UI Manual Control | |
| **Browser sliders** (`ManualControlSheet`) → `Remote producer` → **Robot (USB)** | |
| > No physical master arm needed – drive joints from any device. | |
| > | |
| > 📺 *demo-webui.mp4* | |
| ### AI Inference Loop | |
| **Robot (USB)** ➜ `Remote producer` → **joint-input room** | |
| **Camera PC** ➜ `Video producer` → **camera room(s)** | |
| **Inference Server** (consumer) → processes → publishes to **joint-output room** → `Remote consumer` ➜ **Robot** | |
| > Lets a low-power robot PC stream data while a beefy GPU node does the heavy lifting. | |
| > | |
| > 📺 *demo-inference.mp4* | |
| ### Hybrid Classroom (Multi-Follower AI) | |
| *Same as AI Inference Loop* with additional **Robot C, D…** subscribing to `joint_output_room_id` to run the same policy in parallel. | |
| > Useful for swarm behaviours or classroom demonstrations. | |
| > | |
| > 📺 *demo-classroom.mp4* | |
| ### Split Video / Robot Across Machines | |
| **Laptop A** (near cameras) → streams video → Transport | |
| **Laptop B** (near robot) → joins joint rooms | |
| **Browser** anywhere → watches video consumer & sends manual overrides | |
| > Ideal when the camera PC stays close to sensors and you want minimal upstream bandwidth. | |
| > | |
| > 📺 *demo-splitio.mp4* | |