Raspberry Pi Data Pipeline: Collect, Store and Visualize IoT Data

Collecting sensor readings from IoT devices is easy — the real challenge is making sense of them over time. A single temperature reading tells you the room is 28 °C. A week’s worth of readings tells you your HVAC is underperforming every afternoon between 2 PM and 5 PM. That kind of insight requires a proper data pipeline: sensors feed into a time-series database, which feeds into a visualization layer that turns raw numbers into actionable charts and alerts.

In this tutorial we will build exactly that on a Raspberry Pi using InfluxDB (time-series database), Telegraf (metrics collector), and Grafana (visualization). By the end you will have a live dashboard showing sensor data from your IoT devices, with historical queries and configurable alerts — all running locally on your Pi with no cloud subscription required.

Architecture Overview

The stack follows a classic data pipeline pattern:

1. Sensors/Devices — DHT11 temperature/humidity sensor, BMP280 pressure sensor, or any IoT data source (MQTT, HTTP, CSV files).
2. Telegraf — A lightweight agent that collects metrics from dozens of sources (GPIO, MQTT brokers, system stats, HTTP APIs) and writes them to InfluxDB.
3. InfluxDB 2.x — A purpose-built time-series database optimized for high-frequency writes and time-range queries. Far more efficient than MySQL for sensor data.
4. Grafana — A visualization platform that connects to InfluxDB (and 50+ other data sources) to render beautiful, interactive dashboards.

All three components run as systemd services on your Raspberry Pi. The entire stack consumes roughly 300–400 MB of RAM under load — well within the capacity of even a 2 GB Pi 5.

Hardware: Sensors and Wiring

We will use two sensors commonly available in India to demonstrate the pipeline:

• DHT11 — Temperature (0–50 °C, ±2 °C) and humidity (20–90 % RH, ±5 %) via single-wire digital protocol.
• BMP280 — Barometric pressure and temperature via I2C — great for weather stations and altitude tracking.

DHT11 Wiring

Add a 10 kΩ pull-up resistor between VCC and DATA.

BMP280 Wiring (I2C)

Installing InfluxDB 2.x on Raspberry Pi

InfluxDB 2.x has ARM64 packages that install cleanly on Raspberry Pi OS (64-bit, Bookworm recommended):

# Add InfluxDB repo
curl https://repos.influxdata.com/influxdata-archive.key | gpg --dearmor 
  | sudo tee /etc/apt/trusted.gpg.d/influxdata-archive.gpg > /dev/null
echo 'deb [signed-by=/etc/apt/trusted.gpg.d/influxdata-archive.gpg] 
  https://repos.influxdata.com/debian stable main' 
  | sudo tee /etc/apt/sources.list.d/influxdata.list

sudo apt update && sudo apt install influxdb2
sudo systemctl enable --now influxdb

Navigate to http://your-pi-ip:8086 in a browser. The setup wizard will guide you through creating an organization, bucket (database), and initial admin user. Note the operator API token shown at the end — you will need it for Telegraf.

Creating Your First Bucket

In the InfluxDB UI go to Load Data → Buckets → Create Bucket. Name it iot_sensors and set a retention period (e.g., 30 days — InfluxDB auto-deletes older data). For long-term archives increase to 365 days or unlimited.

Installing and Configuring Telegraf

Telegraf is the collection agent that bridges your sensors to InfluxDB. It supports 300+ input plugins.

sudo apt install telegraf

Generate a base config and customize it:

telegraf config > /etc/telegraf/telegraf.conf
sudo nano /etc/telegraf/telegraf.conf

InfluxDB Output Plugin

Find the [[outputs.influxdb_v2]] section and configure it:

[[outputs.influxdb_v2]]
  urls = ["http://localhost:8086"]
  token = "YOUR_INFLUXDB_TOKEN"
  organization = "zbotic"
  bucket = "iot_sensors"

Reading DHT11 Via Exec Plugin

The simplest way to feed DHT11 readings to Telegraf is via a Python script called by the exec plugin. First install the Adafruit DHT library:

pip3 install adafruit-circuitpython-dht

Create /usr/local/bin/read_dht11.py:

#!/usr/bin/env python3
import adafruit_dht, board, time
dht = adafruit_dht.DHT11(board.D4)
try:
    t = dht.temperature
    h = dht.humidity
    print(f'dht11_sensor temperature={t},humidity={h}')
except Exception:
    pass
dht.exit()

sudo chmod +x /usr/local/bin/read_dht11.py

Add to telegraf.conf:

[[inputs.exec]]
  commands = ["/usr/local/bin/read_dht11.py"]
  timeout = "10s"
  data_format = "influx"
  interval = "60s"

Reading BMP280 Via I2C

pip3 install adafruit-circuitpython-bmp280

Create a similar exec script for BMP280 and add another [[inputs.exec]] block. Telegraf will call both scripts on their defined intervals and write measurements to InfluxDB.

Enable and start Telegraf:

sudo systemctl enable --now telegraf
sudo journalctl -u telegraf -f  # watch for errors

Installing Grafana and Building Dashboards

sudo apt install -y adduser libfontconfig1 musl
wget https://dl.grafana.com/oss/release/grafana_10.3.1_arm64.deb
sudo dpkg -i grafana_10.3.1_arm64.deb
sudo systemctl enable --now grafana-server

Access Grafana at http://your-pi-ip:3000 (default login: admin/admin, you will be asked to change it).

Adding InfluxDB as a Data Source

1. Go to Connections → Data Sources → Add Data Source → InfluxDB.
2. Set Query Language to Flux.
3. URL: http://localhost:8086
4. Enter your organization, token, and default bucket (iot_sensors).
5. Click Save & Test — you should see a green success message.

Creating Your First Panel

In a new dashboard click Add Visualization. In the query editor use Flux:

from(bucket: "iot_sensors")
  |> range(start: -24h)
  |> filter(fn: (r) => r._measurement == "dht11_sensor")
  |> filter(fn: (r) => r._field == "temperature")

Select a Time Series visualization type and you will see your temperature history as a line chart. Add another panel for humidity, pressure, and system metrics (CPU temp, RAM usage — Telegraf’s [[inputs.cpu]] and [[inputs.mem]] plugins handle these automatically).

Setting Up Alerts and Notifications

Grafana’s alerting engine can send notifications when sensor values breach thresholds — crucial for a real monitoring system:

1. In your temperature panel click the Alert tab.
2. Set condition: temperature > 35 FOR 5m
3. Add a notification channel (Telegram bot, email via SMTP, or Slack webhook).

Alternatively, InfluxDB 2.x has its own built-in alerting with Tasks (scheduled Flux scripts that can write alert states to a separate bucket and notify via webhooks).

Expanding the Pipeline: MQTT, Multiple Nodes, and Remote Sensors

Once the basic pipeline works, the architecture scales beautifully:

MQTT Integration

Install Mosquitto as an MQTT broker on the Pi. Configure ESP32/ESP8266 sensors to publish readings to MQTT topics. Add [[inputs.mqtt_consumer]] to Telegraf — it subscribes to your topics and auto-writes to InfluxDB. You can now monitor dozens of wireless sensors from a single dashboard.

Multiple Raspberry Pi Nodes

Install only Telegraf on remote Pi devices (no InfluxDB). Configure their Telegraf outputs to point at the central Pi’s InfluxDB IP. All nodes report to one database and appear as tagged measurements (use the tags section in telegraf.conf to tag by location: kitchen, bedroom, outdoor).

Downsampling for Long-Term Storage

Raw 60-second readings accumulate fast. Use InfluxDB Tasks to downsample to hourly averages after 7 days and daily averages after 30 days. This keeps the database small while preserving long-term trends.

Frequently Asked Questions

How much storage does InfluxDB need for typical IoT sensor data?

It depends on write frequency and number of sensors. A rough estimate: 1 sensor writing 4 fields every 60 seconds generates about 25 MB per month of raw data (before compression). InfluxDB’s TSM storage engine compresses time-series data by 10–20x, so 25 MB of raw data becomes roughly 1–2 MB on disk. A 32 GB microSD card handles thousands of sensors for years, and an NVMe SSD on Pi 5 gives practically unlimited capacity.

Can I run InfluxDB and Grafana on a Raspberry Pi 3 or 4?

InfluxDB 2.x requires a 64-bit OS (ARM64). The Pi 3B+ and Pi 4 can run 64-bit Raspberry Pi OS. However, InfluxDB 2.x has a minimum recommendation of 1 GB RAM — the Pi 4/4 GB is comfortable, Pi 3/1 GB will be tight. Consider InfluxDB 1.x (32-bit compatible, lower RAM footprint) for older hardware.

What is the difference between InfluxDB and Prometheus for IoT?

Both are time-series databases but with different philosophies. Prometheus uses a pull model (it scrapes metrics from endpoints at intervals) while InfluxDB uses a push model (agents write data in). InfluxDB also supports higher write throughput and native SQL-like queries via its Flux language, making it better suited for IoT sensor pipelines. Prometheus + Grafana is more popular in Kubernetes/cloud infrastructure monitoring.

Is this setup secure if I expose Grafana to the internet?

By default, no — Grafana and InfluxDB have no HTTPS. For internet exposure: put Nginx in front of Grafana with a Let’s Encrypt SSL certificate, enable Grafana authentication (OAuth or built-in), restrict InfluxDB to localhost only, and use fail2ban to block brute-force login attempts. For home use on your LAN, the default setup is fine.

Can I import pre-built Grafana dashboards for Raspberry Pi monitoring?

Yes. Grafana’s dashboard repository at grafana.com has hundreds of community-built dashboards. Dashboard ID 10578 (“Raspberry Pi Monitoring”) and ID 15765 (“InfluxDB 2.0 System Metrics”) are popular starting points. Import via Grafana UI → Dashboards → Import → paste the ID.

Build your IoT data pipeline today with sensors and Raspberry Pi boards from Zbotic.in — India’s go-to store for electronics components with expert support and fast nationwide shipping.