Added documentation to creating new devices and protocols

2 files changed, 379 insertions, 0 deletions

diff --git a/docs/developer-new-devices.md b/docs/developer-new-devices.md
new file mode 100644
index 0000000..cc03a35
--- /dev/null
+++ b/docs/developer-new-devices.md
@@ -0,0 +1,233 @@
+# Adding a new device type
+
+A "device" is anything that reads and/or writes channels — NI-DAQ hardware, an
+Arduino, a generic serial instrument, or something entirely new. This guide
+walks through building one from scratch. Related but different: a *protocol*
+is the wire format a serial device speaks (see
+[developer-new-protocols.md](developer-new-protocols.md)) — you only need
+this guide if you're adding a new kind of hardware, not a new serial format
+for the existing `SerialDevice`.
+
+## The interface
+
+Every device subclasses `BaseDevice` (`devices/base_device.py`):
+
+```python
+class BaseDevice(ABC):
+    def __init__(self, device_info: DeviceInfo): ...
+
+    @abstractmethod
+    def connect(self) -> bool: ...
+    @abstractmethod
+    def disconnect(self) -> None: ...
+    @abstractmethod
+    def read_channels(self) -> Dict[str, float]: ...
+    @abstractmethod
+    def write_channel(self, channel_id: str, value: Any) -> bool: ...
+    @abstractmethod
+    def get_config_widget(self): ...
+```
+
+Plus two dataclasses you build up front:
+
+```python
+ChannelConfig(
+    channel_id, name, unit="", min_value=0.0, max_value=100.0,
+    enabled=True, color="#00d4ff",
+    writable=False,   # True = this channel accepts write_channel() calls;
+                       # drives the READ/WRITE badge in Configuration > Channels
+    extra={},          # anything protocol/device-specific you need later
+)
+
+DeviceInfo(
+    device_id, name, device_type,   # device_type is a short slug, e.g. "serial"
+    description="", manufacturer="", model="", version="1.0.0",
+    icon="⚙", channels=[...],       # built once in __init__, before super().__init__()
+)
+```
+
+## Minimal skeleton
+
+```python
+# devices/my_device.py
+from typing import Any, Dict
+from devices.base_device import BaseDevice, ChannelConfig, DeviceInfo, DeviceStatus
+
+class MyDevice(BaseDevice):
+    DEVICE_TYPE = "my_device"   # matches the key you'll add to profile.py's factories
+    ICON = "⚡"
+
+    def __init__(self, device_id: str = "my_0", simulate: bool = True, **kw):
+        self.simulate = simulate
+        channels = [
+            ChannelConfig("temp", "Temperature", unit="C", min_value=-40, max_value=150),
+            ChannelConfig("heater", "Heater", writable=True, min_value=0, max_value=1),
+        ]
+        info = DeviceInfo(
+            device_id=device_id, name="My Device", device_type=self.DEVICE_TYPE,
+            icon=self.ICON, channels=channels,
+        )
+        super().__init__(info)
+
+    def connect(self) -> bool:
+        self.status = DeviceStatus.SIMULATED if self.simulate else DeviceStatus.CONNECTED
+        return True
+
+    def disconnect(self) -> None:
+        self.status = DeviceStatus.DISCONNECTED
+
+    def read_channels(self) -> Dict[str, float]:
+        return {"temp": 23.5}   # only include channels you actually have fresh data for
+
+    def write_channel(self, channel_id: str, value: Any) -> bool:
+        if channel_id == "heater":
+            return True   # actually drive the hardware here
+        return False
+
+    def get_config_widget(self):
+        from PyQt6.QtWidgets import QLabel
+        return QLabel("No configuration needed.")   # see below for a real example
+
+    def get_save_config(self) -> dict:
+        return {
+            "device_type": self.DEVICE_TYPE,
+            "device_id": self.info.device_id,
+            "name": self.info.name,
+            "simulate": self.simulate,
+        }
+```
+
+Drop the file in `devices/` — `DeviceRegistry._discover()` (`devices/device_registry.py:25-39`)
+scans that directory with `pkgutil` on every startup and registers any class
+subclassing `BaseDevice`. No registration step, no import to add anywhere.
+
+## Things that aren't obvious from the abstract methods
+
+**`connect()` sets `self.status`, not just a bool.** `DeviceStatus` is
+`DISCONNECTED | CONNECTING | CONNECTED | ERROR | SIMULATED`. The rest of the
+app (poll loop, control-widget staleness check, status dot in Configuration >
+Devices) reads `self.status`, not the return value. Every device follows the
+same pattern: `SIMULATED` if `self.simulate`, else `CONNECTED` on success /
+`ERROR` on failure.
+
+**`read_channels()` return value is sparse, not padded.** `core/acquisition.py`'s
+poll loop only records a value for a channel if it appears in the dict you
+return — omit a `channel_id` on a given tick (sensor didn't respond, channel
+disabled) and it's simply skipped that tick, not zeroed.
+
+**`write_channel()` is only ever called for channels with `writable=True`** —
+enforced both by `ControlEditorDialog`'s target picker (`ui/control_editor.py`)
+and `ControlWidget._write()`'s guard (`ui/control_panel.py`). You don't need
+to re-check `writable` yourself, but do return `False` for any `channel_id`
+you don't recognize rather than raising.
+
+**Auto-naming.** If your device connects over a port the OS can describe
+(serial, USB), prefer that over a hardcoded string —
+`ArduinoLayer.describe_port(port)` (`api_layers/arduino_layer.py`) returns
+the same description Windows Device Manager / Linux udev shows, so
+`devices/serial_device.py`'s `_auto_name()` and `devices/arduino_device.py`'s
+constructor both do:
+```python
+desc = ArduinoLayer.describe_port(port)
+name = desc if (desc and port in desc) else f"{desc} ({port})" if desc else "My Device"
+```
+(the `port in desc` check matters — some OS-reported descriptions already
+embed the port string, e.g. `"USB Serial Device (COM7)"`, so blindly
+appending `(port)` again produces a doubled-up name.)
+
+**Profile persistence.** Implement `get_save_config() -> dict` (not part of
+the abstract interface, but every built-in device has one — it's how
+`core/profile.py` reconstructs your device on load). Always include
+`device_type` and `device_id`; everything else is whatever kwargs your
+`__init__` needs to rebuild the same device. Then register a factory in
+`core/profile.py::ProfileManager.apply()` (`_DEVICE_FACTORIES` dict, around
+line 271):
+```python
+try:
+    from devices.my_device import MyDevice
+    _DEVICE_FACTORIES["my_device"] = MyDevice
+except Exception:
+    pass
+```
+This is the one place you *do* have to add a manual registration — device
+auto-discovery finds the class for the "Add Device" dialog, but profile
+restore needs to know which class a saved `device_type` string maps back to.
+(If you're shipping a device as a plugin instead of a core device, skip
+editing `profile.py` — call `ProfileManager.register_device_factory()` from
+your plugin's `on_load()` instead; see `docs/plugin-development.md`.)
+
+## Real config widget example
+
+`get_config_widget()` is shown in the "Hardware / Backend" tab of
+`ui/config_dialog.py::DeviceConfigDialog` when the operator clicks
+"Configure" on your device's row. Simplest real pattern, adapted from
+`devices/arduino_device.py`'s `ArduinoConfigWidget`:
+
+```python
+class MyConfigWidget(QWidget):
+    def __init__(self, device: MyDevice):
+        super().__init__()
+        self.device = device
+        form = QFormLayout(self)
+
+        self.sim_chk = QCheckBox("Simulation Mode (no hardware)")
+        self.sim_chk.setChecked(device.simulate)
+        form.addRow(self.sim_chk)
+
+        apply_btn = QPushButton("Apply & Reconnect")
+        apply_btn.clicked.connect(self._apply)
+        form.addRow(apply_btn)
+
+    def _apply(self):
+        was_on = self.device.status in (DeviceStatus.CONNECTED, DeviceStatus.SIMULATED)
+        if was_on:
+            self.device.disconnect()
+        self.device.simulate = self.sim_chk.isChecked()
+        if was_on:
+            self.device.connect()
+```
+
+If your device needs to change its channel list at runtime (different pin
+count, different protocol), give it a `reconfigure(**kwargs)` method that
+rebuilds `self.info.channels` and re-creates its backend layer — see
+`SerialDevice.reconfigure()` for the fullest example (handles five different
+backend layer types depending on `parse_format`).
+
+## Making it reachable from the Add Device dialog
+
+Auto-discovery registers your class with `DeviceRegistry` so it's
+constructible, but the "+ Add Device" dialog (`ui/add_device_dialog.py`)
+has its own hardcoded `_PANELS` dict and `_on_add()` if/elif chain — a brand
+new *core* device type needs a small panel class added there (same shape as
+`SerialPanel`/`ArduinoPanel`/`NidaqmxPanel`) plus a branch in `_on_add()`
+that constructs it. This is deliberately not auto-wired, since the dialog
+needs to know what per-type fields to show before any instance exists.
+
+If you're building this as a plugin instead, you don't touch
+`add_device_dialog.py` at all — extend `ui.add_device_dialog._PANELS`
+directly from your plugin's `on_load()`, per `docs/plugin-development.md`.
+
+## Multiple devices sharing one physical connection
+
+If several device instances can legitimately share one serial port (this is
+why `AnalogInputDevice`/`DigitalIODevice` exist as separate classes that
+both talk to one Arduino), don't construct your transport layer directly —
+route through `api_layers/port_registry.py`'s `port_registry.get_layer(port,
+baud, simulate, extra_pins)` singleton, which returns the same instance for
+repeated calls with the same `(port, baud)` and reference-counts
+`release()` calls so the port isn't closed while another device is still
+using it.
+
+## Checklist
+
+1. Subclass `BaseDevice` in a new file under `devices/` — implement the 5
+   abstract methods.
+2. Add `get_save_config()` and register a factory in
+   `core/profile.py::ProfileManager.apply()` (`_DEVICE_FACTORIES`).
+3. If runtime reconfiguration matters, add `reconfigure(**kwargs)`.
+4. If it needs to show up in "+ Add Device", add a panel to
+   `ui/add_device_dialog.py`'s `_PANELS` + `_on_add()`.
+5. If multiple instances might share one physical port, go through
+   `port_registry` instead of opening the connection yourself.
+6. Run the app — `DeviceRegistry` picks the class up automatically, no
+   import to add.
diff --git a/docs/developer-new-protocols.md b/docs/developer-new-protocols.md
new file mode 100644
index 0000000..724bdc7
--- /dev/null
+++ b/docs/developer-new-protocols.md
@@ -0,0 +1,146 @@
+# Adding a new serial protocol
+
+A "protocol" here means a wire format `SerialDevice` (`devices/serial_device.py`)
+can speak to a serial instrument — SCPI, Mark-10, Modbus RTU, CML are the
+existing ones, living in `api_layers/protocols/`. This is narrower than
+[developer-new-devices.md](developer-new-devices.md): you're adding a new
+option to the existing "Serial / UART" device type's Protocol/Format
+dropdown, not a whole new device class. If your hardware isn't
+serial-based, or doesn't fit the "poll a channel map" shape at all, write a
+new `BaseDevice` instead.
+
+## The interface
+
+Every protocol subclasses `BaseProtocol` (`api_layers/protocols/base_protocol.py`),
+which already handles the connection, the background poll thread, and the
+simulate/real split — you only implement three methods:
+
+```python
+class BaseProtocol(ABC):
+    @abstractmethod
+    def _poll(self) -> Dict[str, float]:
+        """Read the instrument once, return {channel_id: value}."""
+
+    @abstractmethod
+    def _simulate(self, t: float) -> Dict[str, float]:
+        """Return plausible fake values at elapsed time t (seconds)."""
+
+    @abstractmethod
+    def write(self, channel_id: str, value) -> bool:
+        """Send a command. Return False for unknown/read-only channel_id."""
+```
+
+`BaseProtocol.__init__(port, baud, poll_interval=0.1, simulate=True)` gives
+you `self._ser` (a `pyserial` `Serial`, only when not simulating — opened by
+the base class's `connect()`, don't open it yourself), `self._io_lock` (hold
+this around every `self._ser` read/write — the poll thread and any
+synchronous `write()` calls share the same serial handle), and `self.read()`
+which any caller uses to get the latest `_poll()`/`_simulate()` result
+(cached, not blocking).
+
+## Two real shapes to copy from
+
+**Fixed channel set, gauge reports its own units — `api_layers/protocols/mark10.py`.**
+The instrument always exposes the same channels; you don't need per-channel
+configuration. `_poll()` sends `?\r`, parses `<value> <unit>` with a regex,
+tracks the unit so it can be exposed via a `current_unit` property (this is
+what lets the Mark-10 gauge's live lb/kgF/N/ozF switch propagate through to
+plot axis labels — see `devices/serial_device.py:145-152` and
+`core/acquisition.py`'s `channel_unit_changed` signal). Good template when
+your instrument has a small, fixed, well-known channel set.
+
+**Configurable per-channel queries — `api_layers/protocols/scpi.py`.**
+Channels are a list of small dataclasses (`SCPIChannel(channel_id, name,
+query, unit, scale, write_cmd)`) the *operator* fills in per-instrument,
+since SCPI query strings vary by model. `_poll()` iterates the list, sends
+each query, parses the numeric response. Good template when the channel map
+can't be known ahead of time — the corresponding config panel
+(`_SCPIPanel` in `serial_device.py`) is an editable table, not static text.
+
+Pick whichever shape matches your instrument; most new protocols are one or
+the other.
+
+## Wiring it into SerialDevice
+
+Five places in `devices/serial_device.py`, all keyed by a short format
+string you choose (e.g. `"mymeter"`):
+
+**1. Register the format** (`_FORMAT_LABELS`, line ~41 — human label → key)
+and add the key to `_PROTOCOL_FORMATS` (line 53, marks it as an
+instrument protocol rather than a generic key:val/csv/json stream) and
+`_STACK_IDX` (line 77 — index into the config-widget stack, see step 4):
+```python
+_FORMAT_LABELS["My Meter"] = "mymeter"
+_PROTOCOL_FORMATS.add("mymeter")
+_STACK_IDX["mymeter"] = 5   # next free index
+```
+(these are module-level dict/set literals in the real file — in practice
+you edit the literals directly rather than mutating at import time, but the
+effect is the same.)
+
+**2. Channel configs** — add a branch in `_build_channel_configs()`
+(around line 289, `elif fmt == "scpi": ...`) returning the
+`List[ChannelConfig]` for your format, same pattern as the Mark-10 branch
+(fixed list) or SCPI branch (built from user-entered channel dicts).
+
+**3. Layer construction** — add a branch in `_make_layer()` (around line
+364) that imports and instantiates your protocol class:
+```python
+elif fmt == "mymeter":
+    from api_layers.protocols.my_meter import MyMeterLayer
+    return MyMeterLayer(self._port, self._baud, simulate=self.simulate)
+```
+
+**4. Config widget panel** — add a `_MyMeterPanel(QWidget)` class (near
+`_Mark10Panel`/`_SCPIPanel`, line ~601+) — static info text if your channel
+set is fixed (copy `_Mark10Panel`), an editable table if it's configurable
+(copy `_SCPIPanel`). Instantiate it alongside the others in
+`SerialConfigWidget._build()` and add it to `self._proto_stack` (line ~501)
+at the index you picked in `_STACK_IDX`.
+
+**5. Save/restore** — if your protocol has operator-configured state (like
+SCPI's channel list), add a branch to `get_save_config()` (~line 561) and
+accept the matching kwarg in `SerialDevice.__init__`/`reconfigure()`. Fixed-
+channel protocols like Mark-10 need nothing here (`elif fmt == "mark10":
+pass  # no extra config`).
+
+## Things that aren't obvious from the abstract methods
+
+**`_poll()` runs on a background thread** (`BaseProtocol._poll_loop`) — don't
+touch Qt widgets from it. Return the dict; `SerialDevice.read_channels()` on
+the main-thread-adjacent acquisition poll picks it up via `self.read()`.
+
+**Units live on the *device's* `ChannelConfig`, not the protocol layer**,
+if you want live unit changes to reach the plot. The protocol layer just
+exposes whatever it parsed (e.g. `current_unit` property); `SerialDevice.
+read_channels()` is what copies it onto `ChannelConfig.unit` each read —
+see the Mark-10 branch at `devices/serial_device.py:145-152` for the exact
+pattern, including the comment explaining why this alone isn't sufficient
+(the unit-change-to-plot-axis propagation is a separate mechanism in
+`core/acquisition.py`).
+
+**Discard vs. keep unit suffixes.** `api_layers/arduino_layer.py`'s generic
+key:val/csv parser strips anything after a numeric value ("0.04 V" → 0.04)
+and throws the unit away — that's correct for the generic Arduino stream,
+but if your protocol's whole value is "parse a unit that can change live",
+don't reuse that parser; write your own regex like Mark-10's `_RESP_RE`
+that captures the unit instead of discarding it.
+
+**Write-only "action" channels** (zero a gauge, cycle units, enable a
+motor) are just channels with `writable=True` and no corresponding read —
+see Mark-10's `zero`/`cycle_units` or CML's `ME`/`MD`/`ST` channels in
+`_build_channel_configs()`. `write()` dispatches on `channel_id` same as any
+other write.
+
+## Checklist
+
+1. New file `api_layers/protocols/my_protocol.py`, subclass `BaseProtocol`,
+   implement `_poll()`, `_simulate()`, `write()`.
+2. Register the format string in `serial_device.py`'s `_FORMAT_LABELS`,
+   `_PROTOCOL_FORMATS`, `_STACK_IDX`.
+3. Add branches in `_build_channel_configs()` and `_make_layer()`.
+4. Add a config panel class, wire it into `SerialConfigWidget._proto_stack`.
+5. If there's operator-configured state, add it to `get_save_config()` /
+   `__init__` / `reconfigure()`.
+6. Simulate first (`simulate=True` is the default) — `_simulate()` needs to
+   produce plausible values before you ever touch real hardware.