Kinenix Hub Guide

The Kinenix Hub (kinenix-hub) receives worker heartbeats and execution telemetry, stores them in a database, and serves the web dashboard with AI-assisted failure summaries. It is planned to also dispatch jobs to workers (see the roadmap). It was called the Orchestrator before; see Renamed from Orchestrator.

1. Starting the Hub

Install it once. From PyPI (version 0.2.0b1 and later):

pip install "kinenix[hub]"

From a clone of the repository, for development: pip install -e kinenix-core -e kinenix-hub.

# Local only (default): dashboard at http://127.0.0.1:8080
kinenix hub

# Accept remote workers and viewers: set credentials first (sections 2 and 3)
kinenix hub --host 0.0.0.0 --port 8080

The Hub listens on 127.0.0.1 by default, so nothing on the network can reach it until you opt in with --host 0.0.0.0 (or KINENIX_HUB_HOST). The startup banner shows the address it listens on, the URL to open in a browser, the URL workers should use (Worker URL, the machine's LAN address when bound to 0.0.0.0), and whether the worker API key and dashboard password are set.

Saved Settings (First Run)

The first time kinenix hub runs in an interactive terminal, it asks a few questions and saves the answers, so later starts need no flags or environment variables:

1. Allow workers on other machines to connect (binds to 0.0.0.0 instead of 127.0.0.1)
2. Port
3. Worker API key, with hidden input; press Enter to generate one
4. Dashboard password, typed twice; press Enter to keep the dashboard localhost-only
5. Whether kinenix run and kinenix-worker on this machine send their runs to this Hub (saved in ~/.kinenix/kinenix.env, as kinenix connect does)

The answers are stored in ~/.kinenix/hub.env (override with KINENIX_HUB_SETTINGS_FILE), outside the repository. The dashboard password is stored only as a salted PBKDF2 hash; the worker API key is stored as is, because the server must compare it, so the file is made owner-readable only on Linux and macOS. Environment variables and the --host / --port flags always take precedence over the file.

kinenix hub status asks for the dashboard password when the server requires one and KINENIX_HUB_DASHBOARD_PASSWORD is not set.

On Windows, remote workers also need an inbound firewall rule for the port, and the network must be in the profile the rule applies to (run as Administrator):

Set-NetConnectionProfile -InterfaceAlias "<adapter>" -NetworkCategory Private   # only on a trusted home or office network
New-NetFirewallRule -DisplayName "Kinenix Hub 8080" -Direction Inbound -Protocol TCP -LocalPort 8080 -RemoteAddress LocalSubnet -Action Allow -Profile Private

Checking Status from the Command Line

The dashboard is optional. From a second terminal, kinenix hub status shows workers (status, current task, CPU, RAM, last heartbeat) and recent executions with failure summaries:

kinenix hub status                                   # http://127.0.0.1:8080
kinenix hub status --url http://<hub-ip>:8080 --limit 20
kinenix hub logs                                     # steps of the latest execution
kinenix hub logs 54f84bd6                            # steps of one execution, by the ID shown in status
kinenix hub logs --flow "BOT FX"                     # steps of the latest run of a flow
kinenix hub report --flow "BOT FX"                   # short report of the latest run of a flow
kinenix hub report --flow invoice --limit 3          # the last three reports, newest first
kinenix hub report 54f84bd6                          # report of one execution

--flow takes the flow's name, or part of it in any case: invoice finds Monthly Invoice Export. When the text matches several flows, the command lists them so you can type more of the name.

report prints the run report described in hub_data.md: items done and failed, each failure with its item, step, subflow, line, and how it was handled, and the On Error and Finally results. Like query (section 7), it reads hub.db on the Hub machine without a running Hub, or a Hub on another machine with --url. logs and status always ask a running Hub.

logs shows each step's name, action, status, duration, and error, plus the failure and AI summary of a failed run. The dashboard shows the same table from the "ดู Steps" button on each execution. Step output and flow variables are stored in the execution log but are not shown, because they can contain business data. Worker messages outside flow steps (heartbeat problems, service start) stay on the worker; on a Raspberry Pi service, read them with journalctl -u kinenix-worker -f.

It reads KINENIX_HUB_DASHBOARD_USER and KINENIX_HUB_DASHBOARD_PASSWORD from the environment, so set them to the server's values when a dashboard password is configured (section 3).

No CORS policy is configured: the dashboard is served from the Hub's own origin, and workers call the API directly rather than from a browser, so other websites cannot read Hub data through a visitor's browser.

Environment Variables

2. Worker Authentication

The Hub has two independent credentials: an API key for workers (this section) and a username/password for people viewing the dashboard (section 3). Neither grants the other's access.

2.1 Behavior

When the key is unset, the Hub still binds to the network but refuses remote writes, so a forgotten key never leaves the server open. The active mode is logged at startup.

2.2 Setup

1. Generate a random key:

python -c "import secrets; print(secrets.token_urlsafe(32))"

1. Start the Hub with the key:

$env:KINENIX_HUB_API_KEY = "<generated-key>"
kinenix hub

1. On every worker, connect to the Hub. kinenix connect asks for the key, checks it against the Hub (GET /api/v1/auth/worker), and saves the URL and key in ~/.kinenix/kinenix.env only when the Hub accepts them; kinenix run and kinenix-worker on that machine then report to the Hub:

kinenix connect http://<hub-ip>:8080        # or: kinenix connect <hub-ip> (port 8080), --key <key> to skip the question
kinenix connect                             # the Hub in use, where the setting comes from, and whether it answers
kinenix connect --off                       # stop reporting

KINENIX_HUB_URL and KINENIX_HUB_API_KEY in the environment (for example in a worker service's worker.env) and kinenix run --hub URL take precedence over the saved setting; kinenix connect warns when they do. On the Hub machine, kinenix hub show-key prints the key and the kinenix connect command for workers, and the startup banner shows the command too.

Runs started on the Hub machine itself need no step: kinenix hub setup (also on the first start) asks whether to send this machine's runs to this Hub and saves http://127.0.0.1:<port> with the key.

1. Verify from a worker machine:

kinenix-worker ping

ping reports whether the Hub is reachable and whether the key is accepted. Without kinenix-worker, send a heartbeat by hand:

curl.exe -X POST http://<hub-ip>:8080/api/v1/heartbeat `
  -H "X-API-Key: <generated-key>" -H "Content-Type: application/json" `
  -d '{\"worker_id\": \"test-worker\"}'

A 401 response means the key is missing or does not match; a 403 means the Hub has no key configured.

Workers started with kinenix-worker send heartbeats on their own; see the worker guide.

If the Hub rejects telemetry, the worker logs a warning asking you to check KINENIX_HUB_API_KEY. The flow itself still completes; only the telemetry upload fails.

2.3 Security Notes

• Environment variables only: The key is read from environment variables, never from config.json or flow variables, so it cannot be committed to git or written into execution logs.
• Reverse proxy: When a reverse proxy runs on the same machine, every request appears to come from localhost. Always set KINENIX_HUB_API_KEY in that setup.
• Use HTTPS across networks: The key travels in a plain header. Put the Hub behind HTTPS (e.g. a TLS-terminating reverse proxy) when workers connect over untrusted networks.
• Single shared key: All workers currently share one key. Rotating it requires updating every worker.

3. Dashboard Authentication

The dashboard page and its read endpoints use HTTP Basic Auth. Browsers show a built-in login prompt when the page opens and reuse the credentials for the dashboard's API calls, so no separate login page is needed.

3.1 Behavior

3.2 Setup

$env:KINENIX_HUB_DASHBOARD_USER = "admin"            # optional, defaults to admin
$env:KINENIX_HUB_DASHBOARD_PASSWORD = "<strong-password>"
kinenix hub

Open http://<hub-ip>:8080 and sign in. For scripts, pass the same credentials:

curl.exe -u admin:<strong-password> http://<hub-ip>:8080/api/v1/executions

3.3 Security Notes

• Use HTTPS across networks: Basic Auth sends the password encoded but not encrypted on every request.
• Signing out: Browsers cache Basic Auth credentials until the browser is closed; there is no logout button.
• Single account: There is one shared dashboard account. Per-user accounts and roles are not implemented.
• Reverse proxy: As with the worker key, always set the password when a reverse proxy on the same machine forwards requests.

4. Timestamps

• Storage: All timestamps are stored in UTC. Timestamps received from workers are converted using their UTC offset.
• Workers without an offset: Older kinenix-core releases send local time without an offset. These timestamps are treated as the Hub host's local time, which is only correct when the worker and the Hub share a time zone.
• API output: Every timestamp is returned in ISO 8601 with an explicit +00:00 offset. The dashboard converts it to the viewer's local time.
• Existing databases: Rows written before this behavior mix worker local time and UTC. For test data, delete the SQLite file and let the Hub recreate it.

5. Renamed from Orchestrator

The Hub was called the Orchestrator. Old names keep working for now, with a warning asking you to switch:

To upgrade an editable install: pip uninstall kinenix-orchestrator, then pip install -e kinenix-hub.

6. Database

The Hub keeps workers and executions in one SQLite file, ~/.kinenix/hub.db by default (DATABASE_URL selects another file or a PostgreSQL server). Each run is stored with every step and every failure, a short run report, and views for querying; hub_data.md describes the tables and views and gives example queries.

6.1 Upgrades

A newer Hub may need more tables or columns than an existing hub.db has. At every start, the Hub applies the schema changes it has not applied yet and records each one in the schema_version table; existing data is kept. Each change runs in its own transaction, so a change that fails leaves the database as it was, and the Hub stops with the error. Nothing needs to be run by hand. The first start of Hub 0.2.0b4 also fills in the steps, failures, and run reports of the runs already stored, from their logs; on a large database this start takes longer.

6.2 WAL mode and the extra files

The SQLite database runs in WAL (write-ahead log) mode, so workers can write while the dashboard, kinenix hub commands, and other readers read. In this mode SQLite keeps two more files next to the database while the Hub runs:

To back up or move the database (for example to another machine), stop the Hub first. When the last connection closes, SQLite copies the recent changes into hub.db and removes the other two files, so hub.db alone is complete. If hub.db-wal is still there (for example after the machine lost power), copy all three files together; SQLite applies the log the next time it opens the database. Copying only hub.db while the Hub runs can miss the latest executions. Do not keep hub.db on a network drive; SQLite needs local file locking.

7. Querying Runs

kinenix hub query answers questions such as "what failed last night, and where?" from the terminal. It reads the views and tables described in hub_data.md. On the Hub machine it reads hub.db directly, and the Hub does not need to be running. With --url it asks a Hub on another machine, with the dashboard credentials (as status does).

kinenix hub query --list                                 # tables and views, row counts, what each answers
kinenix hub query --describe v_failures                  # columns, their meaning, and examples
kinenix hub query v_failures --since yesterday           # failures since yesterday 00:00, newest first
kinenix hub query v_runs --flow "TIPH Quotes" --status failed --limit 10
kinenix hub query v_failures --filter handled=continued --filter loop_item="R007 HONDA HR-V"
kinenix hub query "SELECT step_name, COUNT(*) AS n FROM v_failures GROUP BY step_name ORDER BY n DESC"
kinenix hub query v_steps --flow Hello --format json     # table (default), json, csv, md
kinenix hub query v_failures --since 7d --output failures.csv
kinenix hub query v_failures --url http://192.168.1.137:8080

The filters work with a view or table name. With SQL, write the conditions in the WHERE clause. A result with more columns than fit the terminal is printed as one block of column value lines per row.

7.1 Safety

• The database is opened read-only, and only reading is allowed: INSERT, UPDATE, DELETE, CREATE, DROP, PRAGMA, and ATTACH are refused.
• One statement per call; a query is stopped after 10 seconds.
• Over HTTP (--url), executions.raw_log reads as empty, because the whole execution log can hold flow variables and step outputs that the API does not show. On the Hub machine the file is readable anyway, so a local query shows it.
• Queries read SQLite only; with another DATABASE_URL the command reports that.

7.2 User Views

Save a query you ask often as a view: put one SELECT in a file named custom_<name>.sql in ~/.kinenix/hub_views/ (setting KINENIX_HUB_VIEWS_DIR). The Hub creates a view with the file's name at every start, so restart the Hub after adding or changing a file.

-- ~/.kinenix/hub_views/custom_failures_by_flow.sql
SELECT flow_name, COUNT(*) AS failures, MAX(failed_at) AS last_failed_at
FROM v_failures GROUP BY flow_name

• Names start with custom_, so Kinenix views (v_) and upgrades never overwrite them. A file with another name is reported and skipped.
• A file custom_<name>.md next to it documents the view: its first paragraph appears in --list, the whole file in --describe.
• A view may read Kinenix views and other user views.
• A file that fails (an SQL error, more than one statement) is logged with its file name at start, and the Hub keeps starting.
• The folder is the source of user views: when a file is removed, the Hub drops its view at the next start. Views hold no data, so nothing is lost.

8. How Machines Connect, and Troubleshooting

8.1 Who connects to whom

kinenix run / kinenix-worker  --HTTP-->  Hub (port 8080)  <--browser--  dashboard viewers
   (any machine)                          (one machine)                 kinenix hub status/logs/report/query --url

• Workers and kinenix run always connect to the Hub; the Hub never connects to them. The only address to set is the Hub's.
• 127.0.0.1 means "this machine". It is right on the Hub machine itself (what kinenix hub setup saves for local runs) and wrong on any other machine, which must use the Hub's LAN address or name, for example http://192.168.1.137:8080 or http://hub-host.local:8080.
• The Hub accepts other machines only when setup allowed it (KINENIX_HUB_HOST=0.0.0.0; the banner says Listening on 0.0.0.0:8080 (all interfaces)). Other machines then need the worker key to send runs and the dashboard password to read.
• A dashboard opened at http://127.0.0.1:8080 on another computer works only through a tunnel such as a VS Code Remote-SSH port forward; without one, open the Hub's LAN address.

8.2 Where a machine finds its Hub

In this order, the first one set wins:

1. kinenix run --hub URL (one run)
2. KINENIX_HUB_URL and KINENIX_HUB_API_KEY in the environment, including a worker service's ~/.kinenix/worker.env and a bundle's .env
3. ~/.kinenix/kinenix.env, saved by kinenix connect or kinenix hub setup

kinenix connect without arguments shows which one is used and whether the Hub answers. A run with no Hub ends with Hub: not connected in its log.

8.3 Symptoms