Documenting the journey of building a robust data logging and visualization solution.
You might remember my previous project where I built a miniature weather station. In that setup, I displayed live sensor readings using an MQTT broker app on Android. (If you missed it, you can check out the details here: Solar Weather Station with MQTT and the MQTT client I used: MQTT Client on Google Play).
While this initial approach worked for live data, I quickly ran into a couple of significant limitations:
- Constant Internet Dependency: The system required a continuous internet connection to view the readings.
- No Historical Data: Crucially, the MQTT setup didn't store any past data, making trend analysis impossible.
To overcome these challenges, I needed a more robust solution: a web application with a dedicated data logger.
What's a Data Logger and Why Did I Need One?
In my previous setup, sensor data was published to an MQTT broker and displayed instantaneously. However, this data wasn't being saved anywhere. To address this, I needed to:
- Capture the Data: Subscribe to the MQTT topics.
- Store the Data: Save these readings into a persistent database.
For this, I developed a Python script that subscribes to the relevant MQTT topics and logs the incoming sensor data into an SQLite database. Each entry is timestamped with the current UTC time, ensuring that the logged data is traceable and clear for analysis.
Building the Web Application: Tech Stack and Features
With the data logging mechanism in place, I built a web interface using:
- Backend: Flask (a Python web framework)
- Frontend: Basic HTML, CSS, and JavaScript
This web application provides several key features:
- Average Sensor Readings: Displays summarized information for various sensors.
- Historical Data Visualization: Allows users to view past data, classified by minutes, hours, and days.
- Data Export: The entire database can be downloaded as a CSV file for offline analysis or use by customers.
Navigating Deployment Challenges: From VMs to App Services
I tried deploying the system in a VM in Azure, but I faced several issues:
- Lack of HTTPS out-of-the-box: Securing the application required manual SSL certificate configuration.
- No Friendly URL: Access was via a public IP address, which isn't user-friendly.
- Security Concerns: Managing security on a VM can be complex.
- Manual Management: Updates and maintenance were time-consuming.
- Performance Issues: Lower-tier (SKU) VMs often froze under load.
After discussing these issues with friends, (Thank you Tharindu 😉) the clear recommendation was to containerize the application using Docker and deploy it as an Azure App Service. This approach offered a much smoother path.
Azure App Services provides various ways to run web applications, including options for static web apps, web apps with databases, and WordPress sites. Given my need for a dynamic application with features like persistent storage and a separate, continuously running data logger, I opted for the Web App service.
To streamline deployment, I packaged the entire project, including the Flask application and the Python data logger, into a Docker image. This image was then pushed to GitLab's Container Registry and made public, allowing Azure App Services to easily pull and deploy it.
You can select your container from different sources when setting up the App Service.
Finally, the system is running!
The Result: A Secure and Scalable Web App ✨
Dashboard LinkI'm thrilled to say the system is now running smoothly! The final website is accessible, secure, and much easier to manage.
This project was a fantastic learning experience. I now have a secure (HTTPS-enabled) application with minimal maintenance overhead, thanks to Azure App Services. It also benefits from inbuilt scaling capabilities and protection mechanisms like Azure Front Door (if configured).
Future Enhancements 🛠️
While the current version is a significant step up, there are a couple of features I plan to add next:
- User Authentication: Implement a sign-in page to manage access for different users.
- Integrated Database Solution: Migrate data storage from the SQLite database within the Docker container (which can be difficult to access for direct downloads) to Azure's inbuilt database services. This will make data management and backups more robust and accessible.
0 Comments