Transforming Agriculture: Lumo CTO Shares His Secrets for Architecting Remote Device Connectivity

Watch Webinar

In this webinar Lumo CTO, Cihan Ucar, explains how Lumo (https://lumo.ag) revolutionized smart agriculture to save growers time, money, and water by leveraging MQTT and HiveMQ. Learn about the challenges faced, architectural decisions made, and the impressive ROI achieved with their connected irrigation system. 

Cihan will offer real-world insights and practical advice including:

• How Lumo architected their solution on AWS using MQTT, Kafka, and Postgres 

• The challenge and solution for connecting remote, low-power devices

• How they cut several months of development time and sped time-to-market

Webinar attendees will receive the Lumo case study and our new Asset Performance Management Playbook. Register now. 

Key Takeaways

• Webinar guest speaker, Lumo CTO Cihan Ucar, has over 20 years of experience in technology startups, with expertise in IoT and water sustainability.

• Lumo is a California-based smart irrigation technology company building products to help farmers save time, money, and water. Its mission is to ensure water and food security for future generations.

• Lumo's product is an integrated cloud management platform for efficient and sustainable irrigation. It uses smart valves that are connected, battery-operated, and solar-powered, eliminating the need for electricity or wiring in farms.

• Lumo’s system provides offline support and monitors water flow, allowing for data-driven decision making.

• Key problems Lumo solves include labor shortages for manual irrigation, need for precision in water deployment, and reducing water waste and overuse.

• Challenges in agricultural IoT include harsh environmental conditions (heat, cold, dust, etc.), power constraints requiring battery/solar solutions, connectivity issues in remote areas, and large areas to cover, requiring mesh networking.

• Lumo chose the MQTT protocol for its low latency, responsiveness, and suitability for low-power devices. 

• Lumo selected HiveMQ Cloud, citing benefits like peace of mind, ROI, and not having to maintain infrastructure themselves.

• Lumo's architecture includes Apache Kafka. HiveMQ Kafka Extension is used to seamlessly move data between MQTT and Kafka.

• Timescale is Lumo’s chosen time series database, where they keep data for 6 months, with cold storage for longer-term archiving.

• For Lumo, key considerations when choosing vendors are meeting project requirements, reliability, cost, quality of support, company longevity, and product roadmap.

• Cihan’s advice for IoT architects underscored the importance of getting the communication protocol right.

Transcript

Introduction

Anadelia Fadeev: 00:00:03.207 Welcome. Good morning, good afternoon, and good evening to everyone. Thank you for joining our webinar today. In today's presentation, we will hear from Lumo on how they revolutionized smart agriculture to save growers time, money, and water by leveraging MQTT and HiveMQ. Before we get started, a couple of logistics. At the bottom of your screen, you should see the Zoom toolbar. There you can find the controls such as the chat, Q&A, and more. So if you have any questions throughout the presentation for our speakers, please submit them using the Q&A section. For discussion, please use the chat. And if you're experiencing any problems, please send me a direct message. I'm here to assist you. We will be recording this session today, and the recording will be sent out to everyone following the webinar. And now I am going to pass things over to our HiveMQ speaker, Mark Herring, to kick things off. Thank you for joining us, and I hope you enjoy the session.

Mark Herring: 00:01:02.088 Thank you, Anadelia. And super excited to be with you here today. And even more excited about my guest, Cihan, who we go back a little bit. We've been at a couple of different companies, worked together, and Cihan's going to share some of his experiences. But just to set the context of what we're trying to do here — Cihan's got a wealth of information. And yeah. I'm hoping to extract some of that and have some fun along the way. But it's going to get a little technical, hopefully, and down into the weeds of why things like Kafka, why things like MQTT. But before we get to that, maybe I'll just hand it over to you, Cihan, to do a little bit of introduction about yourself and about Lumo. What does Lumo do?

Cihan Ucar: 00:01:44.890 Thanks, Mark. And thanks, everybody, for attending. My name is Cihan. I'm the CTO at Lumo. I've been around for a while building technology startups for a little bit over two decades; I believe about 25 years now. I squeezed in a few water-based water protection startups, and Lumo is one of them, a few AI companies, and a few FinTech, and a few, I think, small-business HR template sort of companies. But I like IoT. That's my expertise mostly. And my sub-expertise is about water. I like to build sustainability startups. So again, thanks for having me. I'm really looking forward to having fun and technical conversations. So for the participants, please ask the most difficult questions you can ever imagine.

Lumo Overview

Mark Herring: 00:02:37.327 Love it. So Cihan, do you want to sort of — I know you had a sort of slide deck to go a little bit about Lumo and sort of the background there. Do you want to walk through that?

Cihan Ucar: 00:02:45.074 Yeah. Sure. I guess it's visible now.

Mark Herring: 00:02:48.845 Yes. It's perfect.

Cihan Ucar: 00:02:49.556 So we are Lumo. We are a California-based smart irrigation technology company. We are building products for farmers who actually grow our food to sustain this world and humanity. And our product saves a tremendous amount of time, money, and water for them. We have a mission. Like I said, we want to make sure our next generations have water and food security. And obviously, our objective is to be the number one irrigation management system and the large system of record for freshwater use globally. I'll put this into context. So basically, again, this is me. This is my background. But this is the product we build. We are building an integrated cloud management platform to make irrigation extremely efficient and sustainable. We do that by actually building a smart valve, modernizing an infrastructure that this generation ignored quite a bit. And by doing that actually, we save people quite a bit of time. So a lot of manual irrigation labor is gone, and a lot of cost savings come into play. So you would be surprised, Mark, actually how we grow our food right now. My generation that grew up in the city probably thinks that food grows in the supermarkets because it's very easy for us. We just go, swipe our card, and we have fresh produce. There is a lot of operational complexity, almost similar to building a jet plane, a lot of supply chain, a lot of operations.

Cihan Ucar: 00:04:35.191 So we like to modernize this. And we are one of the actually — let's call it the second-generation wave because the first-generation wave of products tried this with cables, controllers, with a lot of, actually, complexity. But with the modernization of the technology, now we are tapping into that modernization to build really, really cool products. I'll talk a little bit about the product later, but our product is unique. We build our own hardware, own software. So basically, this is a smart valve. You buy a bunch of them. You replace your valve with our valves. They are connected to each other. They are battery operated, solar-powered. So we are solving problems like you don't have electricity in your farm — no problem. And you don't have to wire cables because it gets into your way. Like I said, there's a lot of operational complexity like driving tractors and all sorts of things. No cabling, no electricity needed. These guys have offline support, and we monitor water flow quite a bit. We capture a lot of data. So basically, it's a decision-maker product. You can effectively apply and deploy water. But also, you can collect all that data, like you see in an iPhone screenshot, to make decisions. So it's a lot of cool complexity, and I'm hoping to talk about that in the technical section.

Solving Critical Problems for Farmers

Mark Herring: 00:06:05.450 Love it. Cihan, let me ask you a quick question on how do people — I'm a farmer, not really, but let's pretend. How do I know I have this problem?

Cihan Ucar: 00:06:15.735 So you know it. First of all, most of the farmers are generational farmers. We have a customer that is actually a 23rd generation winemaker. So they were making wine for Medici families, maybe even for Jesus. [laughter] But basically, let's assume you're the first. You just bought a farm and you're first generation. That complexity includes nurturing the soil, planting whatever you're growing, let's say in this case, wine. Right now, we pretty much focus on California, wine growers. You have to deal with a lot of fertilizers. Obviously, plants grow with water. You have to deal with a lot of pumps and plumbing and all sorts of things. Well, who's going to water your plants? You have to find labor. And that labor, you're not going to hire a bunch of, I would say, PhD students to go turn those valves manually for you in the middle of the day or the night.

Cihan Ucar: 00:07:22.693 So the no.1 problem here is actually the labor shortage, especially in California. Also, again, this is difficult. This is a difficult job. You either do it at night, or you do it under 120 degrees in the middle of the day. It's number one. You can't find that labor. And if you find, you're relying on manual labor to deploy water to your plants. Now, if you're in high-value crops like, let's say, wine or strawberry, you want precision. You want to say, "Hey, I want," let's say, "this block, this much acre to be watered by this much water." An example would be, "Hey, I want to deploy 10,000 gallons to my," let's say, "Chardonnay, and I want to deploy 5,000." Or you can say time-based, right? "Hey, from 2:00 AM to 6:00 AM, I want to water my plants." So you're going to send somebody at 2:00 AM. First of all, you can't find that labor, but if you find it, you have to pay overtime. And is it a sustainable labor cost? It's actually not.

Cihan Ucar: 00:08:26.404 So we are solving all those problems on the time front, and obviously, money-saving. But the third one is obviously one of the most important ones. When you cannot control, when you don't have the tools to deploy how much water, as a farmer, you tend to just use as much water as possible. You don't want to under-water. People flood their farms. They redirect rivers, or they pull insane amounts of water from underground reserves. Well, that freshwater is limited. If we keep doing that, our grandkids probably won't have any proper underground water. So we solve all those problems. In fact, the company, our CEO, actually had this problem. He's a tech guy. He bought a farm. He was doing this, and he quickly realized — and he realized that, "Okay. Nobody does this." So the problem is actually very common. People get this. The challenge here is not that people do or don't realize the challenges. The problem had been so far — they didn't have trust in technology. Because so far, again, the first generation of products to solve this problem failed. It failed because, again, they approach the problem from a technology standpoint. There was this disconnect. You can't just take technology and apply it to sometimes a real problem. You have to get the context. You have to ask the question, "Hey, what's the problem? What's the challenge?" Then you go develop. That's actually our approach, and we take pride with this approach.

Early Start in IoT

Mark Herring: 00:10:04.575 Interesting. Very, very interesting. So take a step back maybe, before Lumo came along. And I know you sort of had your slide of sort of your backgrounds, but what excited you about IoT? Why IoT? And maybe we start the journey there. Obviously, Lumo's where we are now, but the journey started before.

Cihan Ucar: 00:10:28.126 Yeah. Yeah. I've been doing IoT for about a decade, 10 years. So actually, my background is industrial engineering, but I was always a computer guy. I was fascinated with data. I don't have kids, but if I have a kid, probably the first one would be named Data. The second one would be Efficiency. That's a negotiation with my wife at this point going on.

Mark Herring: 00:10:50.891 Good luck on that one. Yeah. It's [crosstalk] on that. Yeah.

Cihan Ucar: 00:10:54.231 Yeah. I'm coming from an engineering background, and funny enough, actually my grandparents were farmers. And my dad used to build hardware for the Turkish military, big components and military ships and everything. So I grew up playing with mechanical parts, putting those together. Believe it or not, I used to fix our TV when I was 11 years old. I managed to fix a lot of things. So I went into software, industrial engineering background. I learned how to make things efficient, how to put things together. And I spent about 10 years in software building a lot of, actually, I would say, SaaS products. Started with travel industry, modernized it quite a bit. And then I quickly got the startup bug, cofounded, if you want, CRMs, ERPs, integrations. But the one that I enjoyed the most after I moved to the States in 2010 was this tiny startup where we actually helped small businesses. Basically, we were helping mom-and-pop shops get their licenses with like two clicks, preparing all their forms, or let's say an immigrant person will apply for a green card, for example. Instead of paying a lawyer $5,000, they would just come to our products, subscribe, and get all those forms, provide all the information at this clean PDF, and be done.

Driven by the Challenge of Product Building

Cihan Ucar: 00:12:24.782 So I really got this mission of helping people as much as possible. And then that company was acquired by Intuit. Intuit is a big, obviously, corporation. I worked at QuickBooks product quite a bit. And then my CTO at that time at that startup actually recommended me to one of his friends. That person, Henry, one of our co-founders actually at Lumo — we still work together — he's the inventor of my first IoT startup. It was called Flo, Flo Technologies. It was a water monitoring and conservation device. Basically, I'm sure audience is familiar with Google Nest or a car alarm. It was like a water alarm. So when I met Henry — he's an extremely knowledgeable gentleman, in his late 70s, extremely, extremely, I would say, sharp person. When he showed me his invention, I was like, "Okay. That's it. This is the thing that I want to work." By the way, this is 2015, late 2015. This notion of IoT is not that common. Technical people, let me say this. There is no AWS IoT. IoT probably is used by like 5 to 10 people in the world. And I think people that knew MQTT was maybe like over maybe like 10,000. And I would say like 99% of them probably were IBM people.

Cihan Ucar: 00:13:54.568 So I loved the idea of actually building a product and being the first person. I was the first hired person there. Henry and Henry's son were the co-founders. I just got that back. Immediately, I'm like, "Okay. I'm tired of —" again, no disrespect to, obviously, corporation, but I'm tired of making corporations more rich. I didn't love the fact that people in the United States had to pay even more money to pay their taxes. So I'm doing like a flip. And honestly, Henry explained that to me. Coincidentally, I'm a backpacker by heart. Two weeks before I joined Flo, I was walking in High Sierra, California on a dry lakebed, actually, sand up to my knees, the famous John Muir Trail. Anyway. So everything aligned, universe calling me. And fun fact, Cihan means universe in Turkish. So I took it as a [crosstalk] from universe. I joined. And the amount of data and the engineering challenges was just fascinating. Honestly, I probably learned five times more in my last 10 years compared to my first 15 years of career. And for technical people, challenges are the ones that drives us. Long story short, that was my bug. And then Flo was acquired. And then I just kept going, like AI, IoT, and I think I'm just —

Mark Herring: 00:15:25.827 I mean, Flo or even Lumo. I mean, it seems similar. You've got this remote device. Why MQTT? Because I mean — you mentioned MQTT in passing, but obviously, there's other tech out there. So let's start there with why MQTT? And I assume that's what we are still today. But yeah. Let's start there.

Principle-Driven Development (PPD)

Cihan Ucar: 00:15:48.075 Great question. So for our Flo use case, being real-time was the no.1 requirement. I mean, I like this statement, principle-driven development. I don't even know if it's a thing. I just use it, PDD. When we sat down with Flo people, we said, "This product will protect people even from a drip a minute. Everything has to be real-time." So obviously, I was very comfortable with HTTP and response request stuff. I did not know about MQTT when we were having this conversation. So I started to look around. Obviously, no.1 requirement was being real, real-time. Now, the notion of real-time, obviously, is — to me, it's like 100 milliseconds or less. So we chose MQTT because of the low latency. I don't think there is any protocol — again, unless [inaudible] was developed, and I'm not sure. Nothing can beat MQTT in terms of low latency and responsiveness and bidirectional. So in my opinion, it was the right protocol, and it was proven immediately.

Cihan Ucar: 00:17:03.082 We had two competitors at Flo, and we were the first acquired one. We actually got acquired in about like 3 years. And I always say that. The marketing team loved it. We dunked on our competitors in our demos in responsiveness. And our CEO, a super smart guy, former CEO — he said it very clearly, "If I want to turn off my water, and if I click on my mobile app, I'm going to see that valve turning and closing it. So in under like 200, 300 milliseconds, less than half a second, you could actually protect your home." That was why. Also, not to mention that these devices are very low-power. They are not like random high-GPU computers. The amount of data, the processing data, they all come into play. And not to mention the security, obviously. MQTT protocol is just like a suite protocol. And in my opinion, it's highly, highly, highly optimized for these kinds of cases.

Mark Herring: 00:18:09.770 Interesting. And then I know — sorry. Go ahead.

Cihan Ucar: 00:18:13.399 No. No. I would say 50% of the decision was nothing came close to real-time.

How MQTT Fits into Lumo

Mark Herring: 00:18:19.602 Because I was going to ask — because Lumo seemed even more complex in terms of energy consumption. And you said there's batteries on your solar. How does MQTT fit into that? And yeah. Tell us a little bit more of the pain of — I guess, yeah, the pain that you also encountered at Lumo where things are a little further from the house, right? So yeah.

Cihan Ucar: 00:18:42.934 Right. Correct. Yeah. There are pros and cons, but Lumo product, from the engineering standpoint, is actually a much more difficult product. Why? Flo product, the hardware was a little bit better. Also, it was a consumer product. Lumo, this one is more like a B2B use case. Also, it's sitting outside. So Lumo products are sitting in farms under sometimes like 130 Fahrenheit or 60 Celsius. It snows on them. It hails on them. It's like —leave your computer outside for 24 hours, and you will see, actually, what happens, right? There is dust. There is mud. There is all sorts of things. Also, one of the main things is the power budget. Flo was powered. We could wire electricity. It was constant power in this case. Obviously, technical people understand this. There's no power; there's nothing. So our current product is, obviously, by design, battery-operated and solar-charged. Number one pain point in farming is — in most of the farms — electricity is either far away or it's nonexistent.

Cihan Ucar: 00:19:58.495 Remember, farmers choose their farms very carefully, depending on what they are going to grow. Some people grow pinot noir, right? They like to go higher, maybe on the north side. Some people choose differently. The second one in terms of difficulty — the connectivity. Like I said, farms are far away from the city. In certain cases, there is no reception. So how are you going to actually comment to say, "Okay. Turn on the water," or "Turn off the water." Or how are you going to collect the data? So the second was the connectivity challenges. And the third one  — we deploy a mesh product. Those farms can be as big as hundreds of acres, thousands of acres. So how are you going to cover, actually, in such a large mesh? So think of your home network. Let's say a typical 2,000-square-feet home. You want to deploy Wi-Fi routers. The current modern routers with their APs, access points, you have to deploy four or five so they can talk to each other, right? In this case, you're talking about a mile in between. Sometimes you're talking about actually 10 miles.

Lumo as Close to Real-Time as Possible

Cihan Ucar: 00:21:11.037 Now, industry solves these challenges with amazing protocols like LoRa, LoRaWAN and different things. But it's not the cookie-cutter to say, "Oh, take LoRa and apply." I mean, let me give you a tip for those that are actually willing to do some sort of similar application. Your communication protocol is what makes or breaks you. So a lot of people can say, "Yeah. But LoRa can go up to 50 miles and kilometers." Yeah. Sure. But the bandwidth in LoRa is so low — it is horrendous. It's a great application. If you measure this little bit, one, zero, whatever, every 2 days, five weeks, that's fine. But in our case, Lumo is also actually as close to real-time as possible. In this application, we are as real-time as a minute right now, and we are going to make it much, obviously, better. So the distance was one of the most difficult challenges. And these are not just flat places, right? There is water in between hilltops, trees. So we had to do a lot of testing. And I'm talking about antenna, communication, penetration depth, battery level, how much actually you need to sleep because the irrigations don't happen all the time — right? — and how much heat. So it's like playing chess at any time. I'm actually very proud of our product. And this is a startup, obviously a startup, with a lot of resource constraints. So yeah.

Why Lumo Runs on HiveMQ Cloud

Mark Herring: 00:22:49.668 Interesting. Yeah. So we got a question that came in while we're talking there on your MQTT infrastructure. Are you running that on-premise? Are you running it as a cloud service? And didn't ask this, but, and why maybe? Yeah.

Cihan Ucar: 00:23:02.948 Yeah. Yeah. We are running on HiveMQ Cloud. You guys actually take care of us at Lumo. Before, at Flo, we ran on-prem. And the only reason for that, you guys didn't have a cloud application. So I'm a firm believer of, like I said, principle-driven development. In early-stage startups like us, pace is everything. We operate with a sense of urgency and go-to-market pace. Usually, in startups, people that go to market earlier with a higher-quality product wins in the long term. So I wanted to use our limited engineering resources to build our own product versus maintaining HiveMQ Cloud promise and everything. Like I said, this is a principle. I use managed services on AWS, and I could actually build this product with a handful of engineers. Right now, we have a little bit more awesome folks, but the original team was about four people and me on the engineering side. And we could commercialize this product. And it's because the ROI on the cloud is just insane. And I'm a firm believer of this. I think times change. There are a lot of amazing experts on infrastructure. I like them to handle this for me.

Cihan Ucar: 00:24:23.299 And the second one is peace of mind. I don't have to have an infrastructure engineer, and I don't have to think about, "Oh my God, will my MQTT broker go down at 4:00 AM, and set up international staff and monitoring?" In the first 2 years of startups, I have a lot of problems to deal with and challenges, both technical and mental. It's an emotional rollercoaster. So just having that peace of mind, actually, affects me positively when I face those other challenges too. And like I said, honestly, ROI is probably 50x. And what I would do on-premise, this is my — look, architecture is like supporting a football team, a soccer team. There's no right or wrong. If everything goes well like we hope, and if we get to a scale where we need to customize things and it's more cost-efficient, I would take it on-prem, no problem. So I'm not scared of that. It's just that you have to be context-aware of where you are as a startup.

MQTT vs MQTT SN

Mark Herring: 00:25:27.604 Excellent. Another question came in, which I think is relevant here as well. Did you consider MQTT SN, sort of for sensor networks, or is MQTT — I don't know — classic?

Cihan Ucar: 00:25:36.684 Yeah. I actually talked to Christian, HiveMQ CEO, about that. By the time you guys were implementing, that's actually something in our queue. We will definitely look into it. It's a good use case for applications like us.

Latency Achieved Using MQTT

Mark Herring: 00:25:52.172 Interesting. Very interesting. And then I think the last one while we are on MQTT is I think you mentioned 100-millisecond latency and then a minute. So what sort of latency are you getting now in your deployments?

Cihan Ucar: 00:26:06.625 Yeah. We actually are pretty much real-time. So if I wanted to send a command to one of our units, for example, to say, "Turn on, turn off," or "Apply the schedule," it's actually real-time. When I'm saying a minute, that's our data aggregation point. We collect data every minute. But right now, our latency is under a second. That's not a problem. And even actually with a few layers, I think you will ask me the question about what kind of architecture.

Mark Herring: 00:26:40.811 Yeah. Exactly.

Lumo Architecture Explained

Cihan Ucar: 00:26:44.053 In my opinion, in IoT, data and data integrity is the most important thing. If you cannot send the data or receive the data, you're just done. Therefore, your first layer in your infrastructure that your sensors connect to is your most — in this case, HiveMQ for us — is the most critical point. If you can get your data to your MQTT broker, the current applications like HiveMQ or your competitors, at least to a point, they are resilient enough. You guys don't lose the data. And then after that, obviously, some people like to take that data and put it into S3, Postgres. I use Kafka as a backburner. What I say is if I can put my data into my Kafka broker, I'm good because I have a lot of resilience and everything. So all those things end to end, from sensors to HiveMQ to Apache Kafka and downstream — right? — data processing, and we use time series databases like Timescale, we are pretty much, honestly, in seconds. And for our agriculture application, a few seconds actually are nothing. People are used to hours. Right now, we're at minutes, so people think this is rocket science. They pretty much love Lumo at this point. You can check the logos on our website. We work with some of the largest wine growers in the world right now. Again, it's [crosstalk] —

Mark Herring: 00:28:10.514 That's fantastic.

Cihan Ucar: 00:28:11.186  —startup.

Why Lumo Chose Kafka

Mark Herring: 00:28:12.040 Yeah. So okay. So we've got the data coming in through MQTT on the cloud. Why Kafka? Why did you put Kafka in there? Yeah. What's the architectural reason for that?

Cihan Ucar: 00:28:24.407 Yeah. I think this is one of my best decisions as an IoT architect, let's say. I've been using Kafka since 2014. Why Kafka? Because I look at the sensor data as a stream. Obviously, the most important thing in IoT data is the timestamp and the actual value. In our case, let's say, water flow rate. It could be 40 GPM, gallon per minute, time stamp. MQTT brokers are great for real-time to take the data, but they are not a traditional queue, or they are not an unlimited resilient queue. So Kafka actually provides — it's not technically a queue. It's a log. It's a committed log. It's a very reliable piece of infrastructure that you can push your data, and it provides that peace of mind for you. You can take that data and archive to S3. You can take that data, process in your backend applications, and push into your monitoring systems, which we do. Everything I'm saying, actually — what we do today. You can take it. You can check data integrity. You can massage the data. You can correct data. And you can put into your Timescale or time series database — we use Timescale — to actually power your applications. In this case, web applications, mobile applications.

Cihan Ucar: 00:29:51.054 Also, let's say something happened in your backend or AWS is down, let's say. If your data arrives to Kafka, you're going to recover that data. Obviously, you have to configure it properly. In this case, you can set a retention of, let's say, 30 days in your Kafka topics. Let's say your background data stream application, your custom code failed, or your engineer checked the bug, and you have to reprocess everything for the last, let's say, 7 days. No problem. You can just go change the offset. So it just creates a back-off sort of systems. And there's a lot. If you can design your MQTT-Kafka connection properly, you can actually guarantee the data order, which is one of the most important things you need to do. And you can do deduplication and all sorts of things. One of the actually sweet parts, I would say, it's extremely fast. For technical people, don't be scared of Kafka. Now, there are managed services, and the producers, consumers, the libraries are so modern right now. And one thing — we use HiveMQ Kafka Extension. Honestly, it's one of the best things in the world because I actually wrote that extension myself in 2016. And your CEO knows this was my first request to HiveMQ. We wrote it. It took a lot of engineering time to get it, right? Right now, you just say, "Hey, if data comes to this MQTT topic, just send it to this topic." It's not like a confirmation of topics. I know that pain, so I'm smiling because it's probably the same sequence. And one thing you can do with Kafka or anything — look, it doesn't have to be Kafka. It could be RabbitMQ. It could be SQS.

Mark Herring: 00:31:56.874 Kinesis. Yeah.

Cihan Ucar: 00:31:57.941 Kinesis, whatever, which I definitely don't recommend, but people use it for whatever reason. One thing, in my opinion, you should do, Kafka has a very powerful ecosystem. When I was using it the first few years, there was no Kafka Connect or nothing. Right now, you can use Kafka Connect. You can put a connector there, like a Sink, Kafka S3 Sink, and you're going to keep your data science team very happy. What you can do, your data science will say, "Hey, don't change the data. Don't massage the data. Just give me whatever comes from the sensors," because they are going to run algorithms. They are going to do anomaly detection, right? But if you get a random incorrect reading — your sensor broke — you cannot show it to your customer. So you can just basically plug in Kafka Connect S3 Sink and deploy a configuration. In about five minutes, you're going to start archiving your data to S3 as a cold archive, right? In five minutes, you can configure, let's say, Kafka Connect Elasticsearch Sink, and you can immediately actually push it to Elasticsearch or whatever, Timescale, for monitoring and for data.

Cihan Ucar: 00:33:14.582 I like that kind of, like I said, plug-and-play architecture. And I like to focus on my own logic, building on product. And I almost think it's a waste of our brains to redo things over and over and over. Again, that S3 plugin is available for 5 years. That Kafka Connect is a mature thing. And not to mention, you can run a lot of logic. You can put Apache Flink, tap into Kafka. Just be careful. I made a mistake when I used this. Again, very rookie mistake 10 years ago. Make sure to use your partition keys properly so the data order is guaranteed. Last thing you want is to have to reorder all those things at your consumers. That's a lot of fun. Your customers will see, let's say, your water pressure is going up. Instead of 40, 41, 42, they will see 40, 42, 41. Those are rookie mistakes that we have made. [laughter] It's cringey. But yeah. We have done all those mistakes.

Ensuring MQTT Client IDs Are Unique

Mark Herring: 00:34:22.853 That's why we have you because people don't have to make the same rookie mistakes.

Cihan Ucar: 00:34:26.226 Yeah. Oh, and then one more rookie mistake. I'm sure nobody will do among attendees, but just make sure your MQTT client IDs are unique. I've seen this in two different IoT startups. One of them was me 10 years ago, and one of them was one that I was consulting. All of a sudden, clients lose their visibility. It may actually lead to data loss. I know it's the most basic thing, but just make sure your MQTT subscribers, publishers, just use unique IDs, either their MAC ID combined with CP ID or something like that. Trust me, a lot of people just ignore that.

Mark Herring: 00:35:04.541 Okay. So we've gone from the MQTT to the cloud — right? — cloud and Kafka. You mentioned going into a time series database, Timescale. How long you’re keeping historical data for, right? And then you also talked about cold archive. But yeah. Let's talk about length of keeping that data.

Using Timescale Time Series Database

Cihan Ucar: 00:35:22.939 It's definitely based on your requirements. So cold archive, infinite. Obviously, you have to retain your archive. In our use case, we keep it for six months in Timescale. At Flo, it was actually much shorter, 30 days because the whole notion of time series is data aggregations. So your data is usually at a very high frequency. It could be 1,000 readings per second. It could be a reading per second. Everyone's use case is different. But at Flo, we were doing 1,000 readings per second. We use different databases. We use Timescale. Your retention should be determined by your product use case. I highly recommend keeping it short. In Timescale notion, they do a lot of good stuff. I know the team actually behind it, and honestly, they are awesome people. If you learn how to use the database, you can use their chunks, and you can keep your database extremely efficient. A, you're going to pay less storage fee. Your compute — obviously, you don't have to scale to massive clusters. But most importantly, your queries will be extremely efficient.

Cihan Ucar: 00:36:39.888 Again, I'm coming back to this principle-driven development. My third principle is: data is great, but data is bad. So the data you need is great. Everything has to be data-driven. But stale data is worse. That's going to just kill you. I made that mistake too. In another database, we didn't clean up our data for about 2 and a half years. And it was a cloud product. I mean, InfluxDB, as I'll just say it — it's not their fault. And we just kept upgrading the CPU and the storage. And they came back to us and say if you don't clean up your data, the queries are just becoming massive. CPU times are becoming massive. They said, "We cannot serve you. We have to let you go." And that was like, "Okay. More data is not good." And it was so unnecessary because we had the cold storage anyways. A lot of startup people will ignore that, including myself, because — right? — you don't have enough data. You don't care. It's a technical debt. Anybody doing this — don't cross six months. Just go set up a clean-up policy or something. Also, don't wait for your cold storage. Honestly, set it up day first. Otherwise, you have to migrate data, right? You have to always keep it for six months. Don't set yourself for failure. These are hard-learned lessons.

Thoughts on AI for Decision-Making

Mark Herring: 00:38:07.554 I mean, obviously, today with AI or ML, clearly, we're wanting to make better decisions on the data. And I'm sure Lumo's got the same problem going, "Hey, it's great that the farmer thinks about the stuff, but hopefully, the computer can do some thinking." What's your thoughts on that and how you're approaching that side of the world?

Cihan Ucar: 00:38:27.408 Yeah. Great question. We actually use quite a bit of, I would say, decision-making to help people. I wouldn't say we apply typical AI. We are going towards there, but we have a few features already in production, commercial. One of them is Flow Vision. Basically, right now, people come and create their schedules, right? "Hey, I want to deploy water from 2:00 AM to 4:00 AM." Now, we are actually telling them, "Hey, this is what you intended, but this is what happened," and we are just showing them recommendations. This is your theoretical rate, but this is the reality. So we are going towards more, like, creating effective schedules for them. So for context, effective schedule means determining how much water they need. It's monitoring it. Pushing irrigations tonight. Why? Because at night, evaporation is less, right? And making sure, actually, their combined schedules are not conflicting with each other.

Cihan Ucar: 00:39:32.020 So we are trying to find the most optimum savings for them. Also, you're living in California, so you know energy costs are different, peak hours, off-peak hours, and some of the irrigation can be moved to off-peak hours to reduce the energy consumption. Now, people can say, "Wait, how do you spend energy?" Most of the people run those massive pumps to pull water, either from their lakes, rivers, underground reserves. It's a lot of, obviously, carbon production, and it's a lot of energy. So as a farmer, you keep an eye on those costs. So this is where we are working. And we got the execution. We can deploy the water. Now, we are trying to help them with integrations and those decision-making, like what the schedule should be and how you can utilize water in the most effective way. In general, AI — again, I'm a tech guy. My last startup where I was a cofounder was an, actually, AI company. We tried very hard, and still going on, to help people in the recruitment business to apply AI to improve diversity. And it's a little bit of a difficult use case to explain. But basically, we applied AI to help people get recruited in a more fair way. So we were tapping into video and transcription and trying to extract insights and analysis and using AI to shorten those footage and decision-making processes. So everybody's actually saving time, right? An interview doesn't have to be 40 times one-hour recording. It was an interesting use case.

Cihan Ucar: 00:41:24.627 I'm actually pretty excited about AI, but mostly AGI. And I heard today the former OpenAI chief scientist, Ilya, just announced a new AI company. I just read it on Twitter. I'm pretty excited about AI as it is now. ChatGPT can even write a poem. Don't tell my wife, hopefully. [laughter] I asked ChatGPT to write a poem for her on her birthday, and it did a great job, honestly. I changed it a little bit just in case I don't get caught. I am very, very excited about AGI. When the AI gets to the point where it can act like a human and is capable or is sentient, I think the world will change. I will always say this. Turing machine — computers change the world. AGI will change the world, in my opinion. And even now, typical AI is changing the world. I don't buy the hype where a lot of people think like, "Oh, it can only do very tedious secretary work." That's not correct. There is a lot of technology behind the scenes accessible to give us hope to change the world. That includes autonomous cars. That includes solving very complex diseases. I know a lot of AI models are being used in cancer research. Some of my friends actually work there. So it's unlocking a lot of value that would take humans almost generations. My concern is about fair application and safe application, but I know it's a general concern. Again, Ilya made an announcement today. I think his company is called something like Safe. Safe.

Mark Herring: 00:43:17.533 Interesting. Yeah.

Cihan Ucar: 00:43:19.631 It's not a hype. It's a reality. Don't be one of those people that deny the technology. Sure. You can write a poem like I did. You can generate a song. That's the fun part. AI is not done. In 10 years, 20 years, the world will be a very different place, in my opinion.

Why Lumo Chose HiveMQ

Mark Herring: 00:43:37.689 Yeah. Totally agree. So maybe just wrapping some of this stuff up and thinking about — obviously, you mentioned you use us, which is great. But how do you choose a vendor? How do you choose that I'm not going to go and run this myself? I'm not going to — because clearly, MQTT — many options there. Kafka — many options we talked about — time series databases. But yeah. Maybe as a general point, how do you choose a vendor?

Cihan Ucar: 00:44:00.768 Yeah. No.1 is requirements. So I think it's fair to mention that I procured thousands of software in my career in 25 years. HiveMQ is just one of them. The number one is requirements. The product that I'm buying or I'm getting has to check maybe all, if not most, of the requirements. That's no.1. And I usually check the boxes. And then I look at — this is my personal order. I look at the reliability. Is that product reliable? I like to get a trial. I like to talk to a customer or a few customers if I can. Once I see, "Okay, this product is reliable, and it's going to work all the time," then I look at the cost. Obviously, if it's not something I can afford or as I scale, if it's going to break the bank, I just can't take it. The fourth one, this is my, again, favorite one, support. If every product or partner, the way I like to call, you will need support at some point, especially if you're serious about it, if you're going to scale it.

Cihan Ucar: 00:45:19.219 I actually met HiveMQ on this topic. Obviously, when I decided on the MQTT, I called IBM. They got back to me in 40 days. This is no joke. Again, I'm in Los Angeles, based in Los Angeles. And their team from Canada got back to me in 40 — actually, it was like 41 days or something. And I'm like, "Okay. This can't happen." So I'm googling around, and I came up with HiveMQ. I just cold-called your CEO. I don't know how I found it or LinkedIn. And again, time zone difference, right? I probably messaged about 4:00 PM. At my midnight, I got a message, "Hey, this is my cell number. When can we talk?" That's how I met you since then. Honestly, support, your support and the quality of support, but also the responsiveness of support is what makes me call HiveMQ a partner. And again, for the audience, this is the second time I'm becoming a customer. I'm a paid customer. And just to let you know, HiveMQ is not paying me anything to say these things. But I like you a lot.

Mark Herring: 00:46:28.798 We should have started the whole pitch with this. So that's true. Yeah.

Cihan Ucar: 00:46:31.845 And the longevity. This is actually something I really care about as well. For example, you cannot pay me enough right now for me to use Terraform. Now, tech people know why. Because they got acquired by IBM, and God knows what's going to happen to them. I look at the longevity. I like to talk to people. I ask this question to Timescale. I asked this question to Influx back in days. You name it. I work with Aiven. They have a great, actually, Kafka offering. If you guys want to use Kafka really quick — Aiven is like Confluent, but I prefer Aiven. Longevity — don't be shy to ask, "Hey, okay." A good example was Lenses, Lenses.io. They have a great product. They got acquired. And they just disappeared. In fact, I had recommended them to a few people. One of my friends couldn't get a response for three months. This is no joke. I think now it's settled.

Cihan Ucar: 00:47:30.666 Just be careful because some companies, when you're just about to scale, they will get acquired, or they will go bankrupt. They won't be able to raise money. So longevity of the company is pretty important. And the last thing, obviously, I like — this is rare, but I usually ask for a roadmap. I may have asked HiveMQ like 100 times, "Hey, when are you going to release this Kafka extension? Hey, why don't you build this Kubernetes-based one?" And you guys were fair. And many times, your CTO actually hopped into a call with me, maybe three times during my Flo tenure, 3 years, and he guided us or my architects, "Hey, Cihan, have you considered this or this or this?" Again, I asked about MQTT. I sent to Christian, your CEO. He told me your private road map, "Hey." Now, I'm going to go ask him, "Hey, is this implemented? Can I just use it in the next three months?" So this is my personal list. Again, you can choose, but requirement and reliability must-have, obviously, for everybody. Support, cost, efficiency, support, and longevity, I would say.

Why Lumo Chose Timescale Over InfluxDB

Mark Herring: 00:48:51.054 This question came up, and you don't have to answer it. I'll ask it, but you can defer it, right? So why aren't you using InfluxDB anymore? Do you want to answer that, or should we defer it?

Cihan Ucar: 00:48:59.350 Sure. Sure. I mean, there's a very fair question. I like to use the best products. InfluxDB went through a few tough decisions. I actually literally went to their InfluxDays in San Francisco, so I have a good relationship. There's nothing wrong with them. In my opinion, Timescale is a much, much, much superior product and built by a much better team. When I met Timescale people, actually, person, their CEO, it was in IoT World 2017, '18 in Santa Clara. It was this tiny booth. But when I saw the product, I'm like, "Okay. This product will just go and be a much better product." So I didn't just jump. Again, InfluxDB went through this — their query language was different, and then they came with something else, Flux, I think, and SQL. There was a lot of migration, and there were high cardinality problems. But the main reason, again, that was not on InfluxDB.

Cihan Ucar: 00:50:02.851 Engineers know SQL by heart. In my opinion, that's the most — it's my love language, I would say. Learning Timescale for my entire team of maybe 15 engineers, this is no joke, half a day. And we could migrate in 2 days. And we use Postgres, the power of Postgres and the extensions. It's just like, for me at least, it was a no-brainer. I haven't followed Influx. I'm sure they are still a great product. Everybody should list their requirements and approach. But in my opinion, nothing beats Postgres ecosystem. It's so powerful. Also, think about hiring and training, right? Any engineer you hire these days will have Postgres experience. If not, they will have SQL experience. Learning curve is just like this. And remember, I'm in the game of pace. So if I'm going to spend three months training an engineer versus 2 days, that's a no-brainer decision for me.

Final Words of Advice 

Mark Herring: 00:51:07.382 Awesome. So we're going to have — so if you have questions, put them in the chat. We'll get to those. One last question, Cihan, and it's just more of a wrap-up — in terms of you talked about your PDD, principle design development. Love that. But what's your one bit of advice for an architect sitting in this going, "Okay. I have similar problems. Maybe I'm not in the water sector, but still have remote monitoring problems?" So what advice would you want to leave with the audience?

Cihan Ucar: 00:51:36.463 Yeah. I would say, "Don't get paralyzed." This is like playing chess, right? You have to play the game to think about the move at like the 40th move or 30th move. Don't chase perfection and go step by step. You can iterate on a lot of things. However, you've got to get a few things right, and I'll just count them, okay? Get your protocol right. In this case, if you're going to use MQTT, just make sure that's a good protocol for you. The second — get your security fundamentals right. Adding security to a bad architecture is insanely difficult. In fact, it's going to probably create a redo or vulnerabilities. You have to decide how you pass the data. And the third, this is a little bit more on the hardware side, but choose your hardware carefully. You can iterate on the software. You cannot iterate on the hardware. And design for failure. No matter what you do, your network will fail. Your hardware will fail. Your software will fail. If you're just relying on everything working 100%, you're going to 100% fail in IoT. Mark my words on that. That's why I call it principle-driven development. Again, get your communication protocol right, get your security model right. Even though you don't have to implement it, you have to.

Cihan Ucar: 00:52:56.294 And a quick practical advice, if you can — I hate passwords. In my opinion, even before the AI revolution, we have to do a password revolution. Everybody should stop using passwords. Everybody should move to passkeys, meaning use cryptography, right? And that problem is not — I mean, we don't have a problem until quantum computers become available, that cryptography is resolved. But that's a massive problem. Implement your security model. If you can, implement a public key infrastructure using client certificates with cryptography because you're going to use a username and password on MQTT, and guess what? How are you going to deploy it to your every sensor? If you deploy every username — if your username and password is not unique, well, guess what? One of them will be hacked, and you're going to have a massive problem. I'm sure you guys heard of this baby monitor being hacked, doorbells. Same problem.

Cihan Ucar: 00:54:00.014 So when I implemented public infrastructure, obviously, we didn't go to DigiCert. Well, I asked. They said $800,000. My team implemented it in 2 days. In fact, now there are amazing public infrastructures using cryptography. Smallstep is one of them. There is one on the Java side, EJBCA, I guess. But on API, most of the IoT architects here need to implement and integrate with their manufacturing processes. Just be careful. If you're using China, this is a very hypothetical situation, but US elections always affect tariffs, integrations. For those who have not been to China, Google is blocked. If you're using GCP, good luck, right? So there's a lot of practical approach, but don't leave security to the last minute. You're going to fail. In fact, a very practical approach, when you raise money or when you do diligence, they are going to rip you apart if your security is not on par. And I will say use a reliable MQTT broker. I would not use — I've seen this. I advise a lot of startups. Nothing against Mosquitto. I love the guys behind it. I use Paho. But they just put a Mosquitto broker because if you google it, it's going to come first, right? It's just setting it up — it’s like two minutes. It's not secured. It's not clustered. Remember what I said. Your MQTT broker is what's going to make or break you. That's your no.1 touch point for your sensors. If you lose your MQTT broker, you're losing your connectivity. You're done.

Cihan Ucar: 00:55:45.663 And last practical tip — I think this is the last one — don't wait to implement your over-the-air updates. That's part of your product. In fact, when I see people not thinking about their over-the-air updates as the first thing, I immediately like to think like, "Okay. This person has no idea about IoT." Like your CI/CD in your cloud, you have to set up your OTA and make sure your success rate is about 99%. Your product can be awesome. It could be amazing. If you cannot release code to update the firmware to your sensors, you're done. You're done. Unless you have this military application — right? — you're going to send a commandos parachuting into your ships and updating the USB. I don't think that's the case.

Mark Herring: 00:56:35.625 A little expensive. Yeah.

Cihan Ucar: 00:56:37.148 And don't implement your OTA, guys. I've done those mistakes. There are a lot of amazing OTA frameworks. From the top of my head, Mender is one of them. Golioth is one of them. Depends on if you're using a Linux, RTOS. Maybe one more I can squeeze. That's another hard-learned lesson. Keep your binary size as low as possible. I've seen a lot of people just choosing a Linux distribution. Just because you can't run a full Ubuntu distribution in your connected device doesn't mean you should run it. There's things like Yocto, for example. You can just go choose this package, that package — right? — optimize it. We could actually reduce at Flo. Now, our binaries are very low because we are more experienced. But at Flo, I think we started from 600 megabytes. We reduced it to almost like — I believe it was like 4 megabytes or 5 megabytes. It increases your success rate quite a bit. So yeah. Those are critical things.

Wrapping Up

Mark Herring: 00:57:50.104 Excellent. Well, appreciate your time, Cihan. I know we're sort of wrapping up here. I'm going to pass it back to Anadelia, but I wanted to at least thank you so much for sharing this stuff again. And Anadelia, yeah, what's the next steps? How do people get access to all that cool stuff that we talked about?

Anadelia Fadeev: 00:58:05.291 Yes. Absolutely. Thank you all for joining us today. If we did not get to your questions, we will be following up with you via email. You will also receive the link to the recording of today's session. And once again, thank you so much to our speakers for joining us today, and we will see you at the next one.

Mark Herring: 00:58:23.570 Great. Thanks all.

Cihan Ucar: 00:58:24.891 Thank you.

Mark Herring: 00:58:24.951 Bye-bye.