Troubleshooting When Your IoT Deployment Goes Down
Written by: Steve Hardy
Published: October 21, 2021
We’ve all been there. Your remote deployment has gone down, and your team is in panic mode attempting to diagnose and fix the issue. Although “high-reliability” solutions are the bread and butter of the cellular IoT world, they will still contain many potential points of failure between multiple pieces of hardware, software, and communications infrastructure. To keep your uptime as close to 100% as possible, and minimize on-site troubleshooting time, it’s important to have a standard troubleshooting plan for when a device goes down.
When troubleshooting cellular connectivity issues, the most effective way to check for potential problems is to test the device directly. In many cases, running a series of simple tests can help you narrow down what sort of problem you’re facing.
In other situations, interacting directly with the modem can also help you understand how it is configured and verify its connection status. This is typically done by accessing the modem’s serial interface and sending commands directly to the modem to check how it behaves.
System Status
A good first step is to check for reports of service issues with the cellular provider. In some cases, a connectivity issue may be related to a temporary service issue, such as network maintenance by the local network operator. In these situations, normal connectivity should resume once the issue has been addressed. One commonly used crowd-sourced resource for outage detection is Downdetector.
If you have deployed with Soracom, you can refer to Soracom’s System Status page for known service issues, or check the local network operator’s website for local outage reports.
Basic Deployment Troubleshooting
In some cases, a connectivity issue may be related to the device setup or configuration, such as a loose antenna or a missing setting.
If you are using a Soracom reference device or starter kit, you can also check out our dedicated tutorials and guides for how to set up your device:
If you have finished verifying your device setup but are still having connectivity issues with your Soracom SIM card, there are also advanced diagnostics available in the User Console. The Diagnostics documentation can be consulted for more information on these features.
Device Troubleshooting
IoT devices are produced by many manufacturers at a range of price points, resulting in varying levels of quality. While some expensive or advanced devices may contain more sophisticated troubleshooting tools in their software, or contain embedded SIM cards, the following tests can generally be replicated across all devices.
- Reboot the device – If the device resumes normal operation after rebooting, this may indicate that there was a temporary issue, such as a software error.
- Move the device to a new location – If the device resumes normal operation after moving, this may indicate that the signal strength was poor.
- Try the SIM in a different device – Moving the SIM to your smartphone is often a quick and easy way to perform this test. If the new device successfully connects using the same SIM, this may indicate a problem with the previous device, such as configuration error or hardware failure.
- Try a different SIM in the same device – If the device resumes normal operation after switching to a different SIM, this may indicate a problem with the previous SIM, such as physical damage.
- Factory-reset the device – If the device resumes normal operation after performing a factory reset and reconfiguration, this may indicate that there was an issue with the previous configuration.
- Manually start a network scan (if applicable) – If the device provides a function to scan for available networks and resumes normal operation after successfully completing a network scan, this may indicate that the device encountered a scenario where it could not complete a network scan by itself.
- Manually select a network to connect to (if applicable) – If the device provides a way to choose a specific network to connect to and resumes normal operation after selecting a particular network, this may indicate that the device is unable to select and connect to a network automatically.
Advanced Troubleshooting of your Deployment
If you are able to directly access your modem’s serial interface, you can use AT commands to check the modem configuration and connection status.
While every modem will have a serial interface, not every device provides a way to access this interface. If you are unsure whether or not your device allows you to access the modem’s serial interface directly, please contact the device manufacturer for more information.
The process for connecting to a modem’s serial interface is different for every device. In some cases, you may be able to connect directly using a terminal application and, if necessary, an appropriate hardware serial interface adapter. In others, you may need to compile a program, prepare a debugging environment, or use custom wiring in order to enable or access the modem’s serial interface. The manufacturer of your device can typically provide additional information.
No matter the process for connecting to your modem’s serial interface, you will also need to have its AT command manual available. This is typically a PDF file published by the manufacturer of the modem that will correspond to the model number of the modem in your device.
For examples of AT command usage and further information Soracom has put together a handy guide on our Advanced Troubleshooting page.
Soracom Troubleshooting
If you are using any of the following Soracom features, you can also try temporarily disabling these options, as they may affect your device’s ability to connect:
- Disable IMEI Lock – If you have enabled IMEI Lock for your SIM and your device resumes normal operation after disabling IMEI Lock, this may indicate that IMEI Lock was configured for a different device, or the local network operator did not report the IMEI of your device, causing Soracom to block the connection. For more information, refer to the IMEI Lock documentation.
- Disable Expiration – If you have enabled Expiration for your SIM and your device resumes normal operation after disabling Expiration, this may indicate that Expiration was configured in a way to disable the SIM, causing Soracom to block the connection. For more information, refer to the Expiration documentation.
- Disable CHAP Authentication – If you have enabled CHAP Authentication for your SIM and your device resumes normal operation after disabling CHAP Authentication, this may indicate that your device was not properly configured to use your CHAP Authentication settings, causing Soracom to block the connection. For more information, refer to the CHAP Authentication documentation.
- Disable Virtual Private Gateway – If you have a Virtual Private Gateway (VPG) and have configured your SIM to connect using the VPG, and your device resumes normal operation after disabling the VPG, this may indicate that your VPG has a configuration or routing issue which prevents your device from successfully communicating with servers or cloud endpoints.
- Open a Support Ticket – If you need further assistance after running through your troubleshooting list you can always open a support ticket to get in touch with one of our Customer Reliability Engineers. With no tiered support system, you will immediately be connected to an engineer that has the technical knowledge to begin an investigation and provide solutions.
Editors Note: The author of this piece also wanted to credit Soracom Product Manager Felix Hsieh for his role in the creation of this article.
………………..
Got a question for Soracom? Whether you’re an existing customer, interested in learning more about our product and services, or want to learn about our Partner program – we’d love to hear from you!
