Real-time Monitoring System Using IOT & ML

Introduction

A prominent SCADA manufacturer in Australia approached our team with a unique challenge. Their electric panels, scattered throughout the nation, collected vital data such as pH levels, water levels in tanks, groundwater levels, etc. The objective was to upload this data in real-time to cloud storage, analyze it for predicting droughts and water quality, and visualize the insights on a secure dashboard accessible by city mayors and the national water department.

Challenges

Remote Panel Locations: Many panels were in areas with intermittent or low-quality internet connectivity.

Data Identification: Determining the origin of data based on the state, city, town, and area code.

Cost-Efficient ML Processing: Cost optimization for machine learning models running almost continuously.

Real-Time Data Conversion: Transforming device sensor readings into a standardized format.

Data Security: Ensuring encrypted and secure data transmission.

Real-time Data Visualization: Displaying live data to decision-makers in government.

Solutions

Optimized Data Transmission

Compression: Before transmission, sensor readings were compressed to minimize data size.

Delta Encoding: Only the delta (change) from the previous reading was transmitted, minimizing repetitive data.

Device Identification: Each device was equipped with a unique identifier (GSM Module & SIM), ensuring traceability of data.

Backend Infrastructure

FastAPI: Asynchronous REST API endpoints handled incoming device data efficiently.

PostgreSQL: A relational database with extensions for geospatial analysis stored and managed the incoming data.

Celery: Asynchronous task queue managed data processing, ensuring the system remained responsive even under load.

Data Conversion

Devices converted readings into a standardized JSON format.

Backend systems parsed this JSON data seamlessly, streamlining the data ingestion process.

Security

mTLS: Mutual TLS ensured encrypted and authenticated communication between devices and servers.

RBAC: Role-Based Access Control was implemented on the dashboard, ensuring users accessed only authorized data.

Machine Learning

PyTorch: Used for drought prediction and water quality analysis.

AWS Lambda & EC2 Spot Instances: These on-demand processing solutions ensured cost-effective and scalable ML operations.

Real-time Data Visualization

ReactJS with Tailwind: Created an aesthetically appealing dashboard.

D3.js/Recharts: These libraries allowed dynamic visualization of live data.

WebSockets: Integrated with FastAPI to push real-time updates to the frontend dashboard.

Results

Upon implementation, the system has been able to successfully:

Provide real-time insights into water levels, groundwater status, and other vital parameters.

Predict droughts and water quality issues, aiding in proactive measures.

Offer an intuitive, real-time dashboard for mayors and the water department, improving decision-making.

Ensure data integrity, security, and confidentiality throughout the pipeline.

Conclusion

Through a blend of cutting-edge technologies and thoughtful strategies, we were able to transform raw sensor data from remote locations into actionable insights for decision-makers in Australia. This project stands as a testament to the power of collaborative innovation and the immense potential of combining IoT with data analytics.