Skip to content
nxtinno

Case study

ESP32 edge AI acoustic sensing node

Designed and built a field-ready ESP32-S3 edge device that runs acoustic ML inference locally, reports compact LoRaWAN telemetry, captures labelled training data, and gives operators a local web console for deployment and recovery.

  • ESP32-S3
  • ESP-IDF
  • TensorFlow Lite Micro
  • LoRaWAN

Context

A field sensing use case needed local acoustic detection in places where streaming raw audio to the cloud was impractical because of bandwidth, connectivity, privacy, and operating constraints.

Challenge

The device had to listen continuously, run a compact ML model, communicate over constrained long-range radio, collect recoverable training data, support staged hardware bring-up, and remain serviceable by non-developer operators in the field.

Constraints

  • real-time audio capture and inference inside ESP32-S3 memory and CPU limits
  • compact telemetry suitable for LoRaWAN payload and duty-cycle constraints
  • durable training capture when network upload is unavailable
  • local operator UX without depending on cloud connectivity
  • safe updates for firmware, models, sidecars, and configuration after deployment

Approach

  • Built an ESP-IDF firmware architecture around on-device log-mel feature extraction and INT8 TensorFlow Lite Micro inference.
  • Designed detection state handling with thresholds, confirmation frames, cooldowns, confidence tracking, and event history so telemetry represented operational events rather than raw model spikes.
  • Implemented SD-first labelled audio capture with background sync, preserving local data as the source of truth in weak-network environments.
  • Created a local web console, e-ink status views, RGB status feedback, GPS/battery telemetry, and OTA/artifact update paths for field operation.

Architecture

An ESP32-S3 firmware stack captures I2S microphone audio, derives log-mel features, runs INT8 TensorFlow Lite Micro inference, manages detection events, writes labelled clips to SD, reports health and detections over LoRaWAN, and exposes provisioning, status, storage, training, and update workflows through a local web console.

Production RAG and agent workflowSource systemsdocs · data · APIsIngestionnormalise · chunkRetrievalhybrid · filteredModel + toolsMCP · scoped accessGuardrailseval · approvalDependable outputobserved · attributed
Diagram summary: source systems feed an ingestion and chunking pipeline; retrieval and context assembly ground the model, which acts through scoped MCP tools; evaluation and guardrails observe every step before an answer or action reaches the user.

Key decisions

  • local inference instead of raw audio streaming
  • SD-first training capture with sync as a background layer
  • compact LoRaWAN event and health payloads over general-purpose remote access
  • operator-facing device UX treated as part of the product, not a debug surface

Details are anonymised. Additional context is available under NDA.

Related expertise

AI systems and agentic workflowsSoftware architectureProduct engineering

Curious what AI could actually do for your business?

Bring your questions, including the ones that feel too basic. In 30 minutes we go through how you work today, pick the task with the most to gain, and sketch what testing it would involve. No pitch, no obligation.

Book a free intro call

Prefer email? hello@nxtinno.com