LG201: IoT Cloud Server -
Message Routing and
Data Management
So, we have a WebSocket Server it is lightning quick, we know what’s connected to it. Now we just need to relay the messages about and make sure everything is at it seems, make sure the devices are performing the roles we expect of them, and that they are the devices we think they are.
Device Roles, as you’d imagine, describe the roles played by Devices, each role has its own payload which is sent as part of the message and is also hashed and passed as part of the message, to verify its authenticity.
For each device you can set its roles within a project and how often the project will receive data from the device. You can also define what roles a project will consume when you set it up. This will enable project owners to advertise for data providers, sending an invitation to anyone willing to share that type of data.
Users would be able to chat (WSMessageType = 4) to potential data providers, to assess suitability, before adding the device to the project. A social network for data scientists?
Device Types
When designing this system, we have had to consider the kind of connections and the scenarios in which they will be used. The enumerator WSDevices.DeviceTypes represents my current thinking on device types.
- IoT (1) - Sensors, Microcontrollers and single-board computers
- APP (2) - Console Apps, and other non-IoT applications.
- WEB (3) - JavaScript browser-based clients.
- API (4) - a built-in Virtual App to connect to API.
- HOOK (5) - a built-in Virtual App to send messages to WebHooks.
Onboarding Devices
Devices would be set-up by a User, whilst securely logged-in to the WebApp. IoT devices would be able to onboard using an extension of the Captive Portal capabilities used in the last module. A Web Form would HTTP POST the Email Address, Password and DeviceID to the WebApp which would add the device to the users' account. This method ensures Passwords are never stored in code, they are not required once a device is on the system.
Each Device will need to store three variables, DeviceID, DeviceKey and DeviceSalt, and additionally in memory during a communication session, the Handshake which is unique to the session, and an AES Key, if the communication session is being encrypted.
Verify Message Data
Messages received from non-onboarded devices will force the closure of the WebSocket, as will messages received that are not in the correct format, or the correct order for that matter.
{
"messageID" : "0cd93295-40d3-431b-ba48-002b619ff6f7",
"typeID" : 1,
"deviceUID" : "5e9d6471-8038-4347-9d23-563dbe1e9098",
"payload" : "KY-f7H6Zv-SAkLUA-lXsdeV",
"hash" : "LADeL5A1o7Hfl8sxJb24PSipRAA="
}Above is an example of the WSMessage format, this instance is a Handshake Request, which simply passes the DeviceKey as the payload, and it’s hash as the hash value. The server can re-hash the payload using the devices salt (issued during device onboarding) and check it matches to ensure the message hasn’t been altered in transit.
There are currently five message types stored in the WSMessageType enumerator, Handshake = 1, Response = 2, Command = 3, Chat = 4, Data = 5.
Handshake
Any WebSocket messages sent by a device will be ignored until they have sent a Handshake to the server. Once a handshake(1) is confirmed a Response(2) is sent to the device. This contains the Handshake which is a key that must be used in all communication, in either direction, from that point onwards. All payloads must also be hashed and sent with the messages.
The entire onboarding, handshake and communication process is illustrated in the illustrations in the right-hand column of this page.
As The Rock said, “Know your role”…
Indeed… but we need to know all our device’s roles, in fact, all the roles of all the devices.
What do we need to do when we receive a message:
- Receive a valid message from a device, find its Role.
- Check which projects the device has that role for.
- Check if the correct period has passed since the last message was sent.
- Send to all devices in that project that accepts that kind of data, if they are connected.
- If it’s message type 5 then timestamp and store the data in a database.
We are currently achieving this by iterating through Lists in Linq within the WSManager class. This works great, but a SQL Query may be computationally superior when we have higher volumes of data and devices. This will be the busiest method in the system. We have added Analytics to record its use, this will be expanded to test execution time and we will investigate possible savings.
Are we safe? We’re pretty good…
JavaScript is problematic, the Key and Salt are private, if we use them in JavaScript they will be viewable in the browser. When we create a JavaScript client on our WebApp we know who the user is as they have logged in and there is a Session Variable containing their details. We know this is safe, we will disable cross-domain JavaScript for now.
Going forward we could require users viewing JavaScript clients on other servers to log-in from a web form or pass the Key using server-side code, to receive the Handshake. It would also be necessary to use HTTP POST commands from server-side code to create the Payload-Hash before sending the Web Socket message.
JavaScript aside… We’re good!
The only time the account password is used is in the initial POST request from the device to the server, it is not saved onto the device. As routing and functionality are managed by the server the device cannot be modified to perform other tasks on the network.
The DeviceKey and DeviceSalt are saved to the device during onboarding, messages cannot be created or modified anywhere else.
Even a fully compromised device, poses no risk to infrastructure or user account data.
If a user has lost a device or it is behaving unexpectedly, it can be disabled. Change of network, multiple instances etc. could trigger email alerts and devices simply disabled from the WebApp.
Physical IoT devices are at risk because they are often distributed into the real world. Devices that can perform one-way encryption of source code, such as the ESP32 would be used.
Any device created will need to perform the required Hashing Function or call the API to do so.
SQL or NoSQL, that is the question…
Now all this real-time data and communication is pretty cool, but Data is King, so we need to store all this beautiful information we’re gathering somewhere. Relaying commands and messages between devices, storing sensor data, just by connecting a couple of ESP32’s for a few days whilst testing I amassed over 100,000 records. This could get real big real quick.
So, what are we storing? Let’s consider a single payload from a single message…
{"RoleID":2,"Data":{"humidity":85}}Let’s add a little more info to that, where is it going, when did it go, and what is it from…
{
"id":"461cec38-26a8-4bfc-bcfa-064abc992cc5",
"projectID":1,"roleID":2,
"deviceUID":"ESP32-E40082286F24",
"sent":"2020-08-10T13:31:10.2853476+00:00",
"data":
{
"humidity":85
}
}When it comes to storing sensor data, as Van der Veen, Van der Waaij and Meijer state in Sensor Data Storage Performance: SQL or NoSQL, Physical or Virtual, “A database that performs well on single writes and multiple reads would be the ideal candidate for storing sensor data.” [1]
https://ieeexplore.ieee.org/document/6253535
We already have a really powerful database so we could just store the data in there… SQL Server Azure can handle it, no problem.
But, hang on, I’m taking a structured JSON string and storing it in a flat database structure. NoSQL Databases were designed to store structured data like this, I’ve never used one, but I’m sure this is exactly what they are designed for. We have CosmosDB in Azure we can put an instance of that up for a few pence a day, is that a better route?
Unfortunately, the research highlighted above didn’t find a clear answer to this, but interestingly noted,
“The presence of an index has a tremendous effect on performance both for MongoDB and PostgreSQL. If no index is present, they need to perform scans to reach the data the client asks for. This is a very slow operation, especially for large data structures.”
And there is our answer… indexing.
The advantage of relational databases is the relationships, if we chuck sensor data in a table it just a big lump of JSON or text, we can’t index that, we don’t know what it’s going to contain, it will be different for every role.
Hadi Fadlallah, states in his article, ‘SQL Server JSON functions: a bridge between NoSQL and relational worlds’,
“ If we need to filter our queries based on a property value within the JSON column, and we need that this property is indexed, we should add a computed column based on this value. Then we must create an index for this column.”
https://www.sqlshack.com/sql-server-json-functions-a-bridge-between-nosql-and-relational-worlds/
This also highlights another issue, we spent ages locking this database down, you can only communicate with it through Stored Procedures, we’re not going to open it up to queries and we don’t want to write Stored Procedures for each Role and search unindexed data.
If I had any doubt left the official documentation for Azure Cosmos DB states “In Azure Cosmos DB, every container has an indexing policy that dictates how the container's items should be indexed. The default indexing policy for newly created containers indexes every property of every item, enforcing range indexes for any string or number. This allows you to get high query performance without having to think about indexing and index management upfront.”
https://docs.microsoft.com/en-us/azure/cosmos-db/index-policy
Sold! Let’s do this!
We create a Singleton DatabaseClient in Startup.cs and pass it to WSCosmosClient.cs in the Class Library to handle the requests, links to code below….
References
1. J. S. van der Veen, B. van der Waaij and R. J. Meijer, "Sensor Data Storage Performance: SQL or NoSQL, Physical or Virtual," 2012 IEEE Fifth International Conference on Cloud Computing, Honolulu, HI, 2012, pp. 431-438, doi: 10.1109/CLOUD.2012.1