How an Edge TPU stock predictor fixed its own training bugs

Late last year, a software engineer set out to replace gut instinct with data when deciding whether to sell Google RSUs. The experiment began with a modest goal—train a lightweight AI to predict short-term stock movements—but quickly spiraled into a hardware and software detective story that uncovered four critical bugs in the pipeline. What started as a quest for a personal trading assistant became a masterclass in edge AI inference pitfalls.

The Hardware Paradox: Train on x86, Run on 2-Watt TPU

The project leveraged existing hardware: a Coral Dev Board, a discontinued single-board computer packing an Edge TPU coprocessor directly on the SoC via PCIe. Unlike its USB-bound cousin, this board offers ultra-low idle power (around 2 watts) and millisecond-scale inference, making it ideal for always-on financial monitoring. The engineering team wisely separated workloads: heavy training ran on an RTX 3050 laptop, while inference and sentiment analysis moved to the Coral.

This separation made sense because the Edge TPU excels at inference but lacks support for training operations like backpropagation. The Coral isn’t designed for model development—it’s a deployment platform for pre-quantized models. That distinction became crucial when the team faced a cascade of compilation failures.

Conv1D Over LSTM: Why Architecture Choices Matter on Edge TPU

The frozen operation set of the 2019-era Edge TPU dictated model design. Common sequence models like LSTMs and Transformers failed because they relied on unsupported operations such as BatchMatMul or LayerNorm, forcing parts of the graph onto the host CPU. The result? Latencies ballooning from 2.5 milliseconds to nearly 300ms.

By contrast, a 1D convolutional network (Conv1D) mapped cleanly to the TPU’s CONV_2D primitive. The model used ReLU6 instead of regular ReLU, and relied on native pooling layers like GlobalAveragePooling1D. This architectural pivot wasn’t just about performance—it was about survival in the Edge TPU ecosystem.

Fifty-Two Features, Three Horizons: The Input Fabric

The model ingested 60 days of historical data, normalized across 52 engineered features grouped into seven categories:

• Price and volume trends (5): normalized close price, OHLC ratios, volume deviation
• Return and volatility signals (6): daily, weekly, and monthly returns, log-returns, realized volatility
• Momentum oscillators (11): RSI variants, Stochastic K/D, Williams %R, MFI, CCI, ROC
• MACD family indicators (4): line, signal, histogram, delta
• Trend and moving averages (12): comparison to 5/10/20/50/100/200-day MAs, Bollinger Bands, ATR, ADX, directional indicators
• Volume-based signals (4): OBV, volume ratio, CMF, volume momentum
• Market context (10): SPY returns, VIX z-scores, relative strength, calendar effects, proximity to 52-week highs/lows

The model produced three outputs for each of three time horizons (1-day, 3-day, 5-day): predicted log-return and up-probability. This dual-output design allowed the system to act as both a price forecaster and a directional classifier.

Four Silent Bugs That Almost Killed the Project

The first bug surfaced when the model consistently predicted $314.74 with 0.00% change—implying no movement ever. The culprit? A misplaced scaler that silently refit on live data instead of crashing. The fix was simple but brutal: fail fast. The inference pipeline was updated to raise a clear error if the scaler file was missing, preventing silent poisoning of inputs.

The second issue revealed how small layer choices cascade. Using Flatten instead of GlobalAveragePooling1D caused only 5% of operations to run on the TPU. The fix was a one-line swap that enabled full TPU execution.

The third bug exposed a deeper constraint: BatchNormalization introduced quantization artifacts. By inserting a DEQUANTIZE node, it split the graph, leaving the TPU to handle only the first convolutional layer. The solution was radical—remove batch norm entirely. Since inputs were pre-scaled using RobustScaler, the network didn’t need normalization for stability. Replacing use_bias=False with use_bias=True kept activations bounded within ReLU6’s 0–6 range.

The final bug was subtle: reading quantization scales from the wrong tensor field. The model expected per-layer scales, but the code pulled per-channel weights, crushing continuous price data into a narrow INT8 range. Correcting the tensor access restored meaningful input representation.

From Zero to 2.5ms: The Final Pipeline

After fixing these issues, the model compiled cleanly, with every operation mapped to the Edge TPU. Inference latency dropped to 2.5 milliseconds—fast enough for real-time monitoring. The dual-model system now delivers both price forecasts and directional probabilities, though the author cautions that results are "instructive and sometimes humbling."

This project underscores a growing reality in edge AI: success depends less on model complexity and more on hardware-aware engineering. As more teams deploy quantized models on microcontrollers and TPUs, the lessons from this Coral Dev Board experiment will likely echo across industries from finance to robotics.