Introducing: Soracom Orbit
Written by: Takashi Oguma
Published: July 22, 2020
I am delighted to announce the release of Soracom Orbit today. As an engineer in Soracom’s Tokyo office, I’ve been involved in the development of this new IoT service for the past six months, so today is a big one for me.
Soracom Orbit is a microservice that lets you run your own programs to transform data on the fly as it passes between devices and the cloud. Data format mismatching between device and cloud has always been one of the more irritating challenges in IoT architecture. SORACOM Orbit offers a way to solve this problem with relatively little effort.
Data formatting challenges affect every deployment differently. From the device side, best practice is usually to minimize the number of bytes stored in the device memory and the amount of data transferred. But in most cases, data is then parsed and expanded into an object or other data structure to be consumed by cloud services. We hope to minimize the effort required to write custom parsers and create an enjoyable developer experience. Orbit introduces another achievement towards our goal of creating vital data services, which are “Connectivity Agnostic” and “Cloud Agnostic.”
With this announcement, I decided to have a little fun and share how Orbit came to be. I hope you enjoy the story even though it may be a little on the long side.
A year in the making
Last July, at Soracom Discovery 2019, I exhibited a proof-of-concept plant cultivation system. That system used SORACOM Funk in combination with a binary parser. And from that experience, I began to think, “I want a better binary parser.”
From there, I made a list of what it would take to create a better binary parser.
- The ability to read part of the data and change the processing of the rest.
- The ability to use arithmetic operations and also mathematical functions such as sin() / cos() / tan() and sqrt() and various utility functions. For example, with location coordinates, you could perform more complex operations with trigonometric functions and insert the calculated values into the session payload.
- The new parser would not be limited to a binary data format, but also various other text formats such as CSV. Even devices that output GPS position information as comma-separated character strings can easily connect to cloud services.
- The option to use data such as tags tied to a SIM. If you put SIM name or IMSI in the data, it will be convenient for visualization and data analysis.
To be honest, current binary parsers are already very convenient, very simple, and easy to use, but human desires are endless. The list above is not a pain to implement in your own code, but if these were standard in a binary parser, wow!
Picking the preferred language
The question became, how can we extend the current binary parser to support these infinite number of possible use cases? When I thought about it, it was easy to imagine that the current method of expressing conversion rules using a proprietary DSL (Domain Specific Language) would soon hit the limit. Even if we develop a more complex and high-performance DSL, it will force our customers to learn new notations, and it will be difficult for us to provide and maintain it. Therefore, I thought that if we provided a mechanism to have the conversion code written in some existing programming language, we would be able to avoid the need to have our custom DSL while ensuring a certain degree of freedom in processing content.
Alas, no matter which programming language we choose, there would be customers who are unfamiliar with it. For example, the Lua language is common for such plugin-like purposes (Nginx and Wireshark). But unfortunately, many engineers aren’t so familiar with Lua and would limit the customers who could implement the service. However, supporting multiple languages also seems like a heavy burden on us. So I pressed on to find a solution that would allow customers to write data processing code freely in their preferred language. It was at this time I discovered WebAssembly!
About WebAssembly
WebAssembly (Web Assembly, or WASM for short) is a standard designed to execute applications on a Web browser quickly. As the name implies, it is written in an assembly and is a W3C standard. So, why does WebAssembly have anything to do with the “great binary parser” I was building? Actually, the first sentence of the specification of WebAssembly (V1.0) has the following.
- WebAssembly (abbreviated Wasm) is a safe, portable, low-level code format designed for efficient execution and compact representation. Its main goal is to enable high-performance applications on the Web, but it does not make any Web-specific assumptions or provide Web-specific features, so it can be employed in other environments as well.
I thought that if customers could write code in their preferred language that supported WASM output and execute it on the SORACOM platform, various problems would be solved. Continuing the journey, I tried WASM code and wrote a WASM processing system available on GitHub.
Pitching the Idea
And the beginning of this year (2020), the next-product contest was held in the Japan office. I submitted four ideas, one of which was the “great binary parser” based on WebAssembly. I thought since the “great binary parser” is a service located on the boundary between the Soracom platform and the customer’s device, I applied for the contest under the codename Frontier. Later the results were counted, and Frontier won!
The development started with workshops to introduce the team to WebAssembly. Unfortunately, work on “Frontier” did not move significantly forward until Nori, Soracom’s growth hacker, and scrum master suggested we prioritize the project for launch at our annual conference, Discovery 2020.
Proof of Concept (PoC)
At the time, I remember it being around April and thinking to myself the deadline for Discovery is in July, would it be ready by then?
While working on PoC, my previous project work was taken over by my teammates Moz and Tadashi. While each of them already had a full workload, they showed an inexhaustible number of talents to push through the additional tasks. It is no exaggeration to say that Orbit would not be a reality if it was not for their efforts. Thank you for the support moz+++++ tadashi+++++!
As development continued, I began to grow more confident in the timeline. A significant part of this added confidence came from choosing WASM for WebAssembly. Then, officially, we decided to proceed with announcing the new service at Discovery.
From Codename to Product Name
At the beginning of June, it was time to decide on the new service’s final product name. There are some cases where the applicable use cases overlap or partially replace Funk. So will Frontier be adopted as it is because it is a sibling of the service name starting with F? After some internal discussion and voting, I finally chose Orbit from the pool of winning ideas.
Feels like the first time
Through Orbit’s development, I was reminded of the excitement I had developing some of our foundational services, such as the user console, API, and Beam. Even though I wrote this from my perspective, there were so many others who contributed to the project and will continue to implement new features and bug fixes. Thank you to everyone who played a role in Orbit. I am deeply grateful to work in an environment surrounded by such supportive colleagues.
Summary
I hope you did not mind reading the story of how we created Orbit. I felt it was important to share how products are developed at Soracom, even though this article may have been longer than usual blog posts.
In summary,
- Dogfooding is important. After actually using the binary parser and Funk, I felt the need for something like Orbit.
- Be curious. At first, I thought WebAssembly had nothing to do with me, but once I understood the specification, it was clear there was no better option.
- Building something new inside an organization requires a team. The members of Soracom are too kind!
Please give Orbit a try and give us your thoughts and feedback!
