Welcome to the first issue. I'm starting this newsletter at zero, so if you're reading this you were one of the very first to sign up. Thank you. The deal: one piece roughly every two weeks about what happens when code touches the real world. This is the first one.
This week I published the piece this whole project exists for: a map of the gap between "something physical happens" and "a number appears in your database". If you've only ever worked on the database side, that gap is invisible. It's also where most data quality problems live.
The short version
A single temperature reading passes through six stages on its way to your application, and every one of them changes it:
The sensor itself. A class B PT100 is allowed to be off by ±0.3°C at 0°C, widening to ±1.3°C at 300°C. Add ten years without recalibration and slow drift nobody noticed. Your data has error bars before it leaves the pipe.
The 4-20mA loop. The analog signal gets scaled and quantised, and sometimes bumped by electrical noise from a motor drive nearby. If the transmitter's configured span doesn't match reality, every reading is scaled wrong.
The PLC scan cycle. The PLC samples every 10-100ms. Events faster than that are invisible, and the timestamp usually means "when we asked", not "when it happened".
The historian. It only stores a value when it changes by more than a configured threshold. A flat line in your data can mean "nothing changed" or "we stopped recording small changes", and you can't tell which from the data alone.
The cloud pipeline. Buffering on connection drops, resampling on-change data to fixed intervals, local time versus UTC. Each one quietly transforms the data.
Your application. The number in your database is not "the temperature at that time". It's the best available approximation, given all of the above.
What to do about it, condensed: document the uncertainty in your schema, keep alert thresholds wider than your sensor's error band, never interpolate silently, validate range and rate-of-change at ingestion, and ask the operators which sensors always read a bit high. That last one is the most valuable data quality documentation there is, and it's never written down.
Read the full piece on the blog. It walks through each stage properly, with the numbers and the failure modes.
Behind the scenes
Honest status report: the blog is a week old, this newsletter is younger than that, and I start a three year industrial automation degree this autumn on top of my day job building IoT in the energy sector. I'm writing all of this down as I learn it, including the parts I get wrong. That's the format.
If anything in here matched your own experience, or contradicted it, just reply. I read everything.
Marius