Disaster recovery: come what may

Keeping mission-critical networks up and running in the face of nature’s fury is one of our industry’s biggest challenges. Sam Fenwick looks at how operators and vendors work to give first-responders the service they depend on when they need it most

This year saw the US take more than its fair share of punishment, with Hurricanes Harvey, Irma and Maria barrelling in from the Atlantic to wreak havoc on the mainland and Puerto Rico. Keeping two-way radio service going during and after such storms is challenging to say the least. However, public safety organisations and their vendors were prepared for the worst, as Steve Gorecki, Motorola Solutions’ director – media relations, explains.

“When we heard about those hurricanes, we had our sites ready, we put equipment in position; with Irma, we staged equipment including batteries and generators in Florida’s neighbouring state, Georgia, prior to the hurricane so that when the storm passed it would be ready to go.

“As storms come by you learn from each one. From Katrina, we learned it was not just about interoperability, but operability. You need generators but you also need fuel. When these storms happen, there’s a run on the gas stations so we had fuel trucks staged in Georgia to be able to provide fuel for generators in Florida. With Harvey in Texas and Irma, our systems continued to operate – a prime example of LMR’s resiliency and the importance of redundancy.”

Gorecki adds some microwave dishes became misaligned, but Motorola Solutions’ systems continued to operate. “The only area where there was some need for repairs to entire sites was in the Florida Keys, where we had a convoy in Georgia; within days it had everything back up and running there.”

He adds that the exception was in Puerto Rico where some smaller networks in different services had some issues. “With Puerto Rico, we were a little hamstrung in that we had to wait until they allowed transportation of the needed equipment. The radio communications equipment, whether it was for a small agency, a commercial whatever, did not take priority over transporting water and food.”

Gorecki explains that when its customers are in the path of the storm, his company monitors their systems to make sure they are operating and identify any problems with sites. It also opens up a 24/7 hotline and provides equipment.

Greg Holcomb, division manager and E911 co-ordinator, director of public safety communications technologies, Lake County, Florida, gives an operator’s perspective. He says hardened infrastructure is first and foremost “making sure we’ve got diverse routes and multiple network connectivity. We designed our towers and loading with hurricanes in mind, with additional sidearms added to microwave dishes to prevent them from twisting in heavy-gauge winds.”

Holcomb adds that all their sites have a minimum of one hour’s battery back-up, rising to two hours for critical sites. All sites have their own generator sets; he says there is a need to make sure his organisation has “vendors and suppliers in place that can get us fuel. Diesel fuel’s [availability] becomes a major issue during these type of events so all of our sites are either on natural gas or propane, which is a little bit more readily available during emergencies.”

Redundancy is also important – while Lake County has a 700/800MHz Astro P25 system as its main trunking system, “on top of that we have 800MHz national mutual aid channels, so there are independent receivers and transmitters above and beyond the trunking system. We have a VHF and UHF system for redundancy and back-up, and a 700MHz overlay, an independent overlay that spreads throughout the state and runs independently of the trunking systems.”

Holcomb explains that the national mutual aid channels allow outside agencies to turn up and immediately start assisting in the disaster relief effort, “even though their radios may not be programmed for your particular system”.

The county’s police officers also have mobile data terminals, with the majority being on Verizon, which adds an extra level of redundancy.

Lake County usually gets about 72 hours’ notice of it being in “a potential cone of danger”. As soon as they receive notice, Holcomb and his colleagues start preparing and performing preventative maintenance. Once the storm hits, there is the increased load on the system to manage. During such events, first-responders are told by the dispatcher and by text message that “emergency procedures are in effect”, which require that only critical information is communicated over the air. Holcomb adds that the system, when busy, queues users, notifying their radios when a channel becomes available. He says the number of PTT calls per day during Hurricane Irma rose to almost 70,000 at its height, roughly double that of a normal day.

Frank Pauer, Hytera Mobilfunk’s deputy chief sales officer, says when TETRA sites go down, it’s often due to flooding of the buildings housing the base stations. That is why he recommends mounting base stations where the antennas are located – on building roofs.

His colleague, Hauke Holm, the company’s CTO, explains that Hytera’s network management systems detect any unexpected behaviour in a network and can automatically activate more carriers in a particularly busy area to increase capacity and ensure service. “All of our base stations are equipped with additional carriers, but due to interference issues and the limited spectrum available, they can’t be enabled all the time, so that needs to be managed.”

The same system also gives operators recommendations on how to optimise communications. “It also shows you if there are too many groups in operations, so it helps the operators to decide that it might be better to have joint forces in one group rather than individual groups in the disaster area, reducing demand for radio resources.”

EE recently used a helikite to provide LTE coverage during a mountain-biking event in Wales

Let’s go fly a helikite
While the traditional model – using vehicle-mounted base stations to provide temporary coverage to patch holes created by sites going down – still holds sway, one approach that seems to be gaining traction is the use of small aerial vehicles, either helikites or unmanned aerial vehicles (UAVs), with the latter acting as a quick-to-deploy solution and the former used to provide medium-term coverage.

As far as helikites are concerned, one of the first developments in this area was a proof-of-concept project, Project ABSOLUTE, which ran from 2012 to 2015 and involved Allsopp Helikites and BAPCO. Among other tasks, it involved the design and development of a lightweight battery-powered LTE-A amplifier and associated antennas that could be integrated into an aerial platform and provide a high-capacity bearer with sufficient range for use by multiple first-responders. The project was demonstrated at a site 40km from Paris with multimedia communications across all locations – incorporating text messaging, one-to-one and one-to-many voice communications, teleconferencing, real-time imagery and image transfer – over the ABSOLUTE network. Information exchange in the form of voice communications over the PSTN (public switched telephone network), and access to corporate databases and social media sites and other internet-based sources of information, was also demonstrated.

This isn’t just about providing temporary coverage for first-responders. Flying at 1,000ft, a helikite has radio line of site of 45 miles, giving it the ability to span more than 6,362 square miles. According to the abstract of a paper covering the project and written by Sandy Allsopp, “such a 4G airborne base station will allow anyone with a mobile phone to be connected to the network if required. Thousands of victims will be able to report their circumstances to the emergency services, with their smartphones automatically providing their exact position to first-responders.”

This use-case was demonstrated by Project ABSOLUTE, which during its final demonstration allowed someone playing the role of a member of the public in need of assistance to communicate with a remote HQ. They were directed to an application, which enabled them to populate information that automatically highlighted their geographical location and distributed it across the network.

EE, the mobile network operator responsible for providing the UK’s first-responders with the Emergency Services Network, demonstrated a similar concept at the Kia Oval in February – a helikite supplied by Allsopp Helikites and equipped with a 5W small cell from Nokia. During the event, EE demonstrated a VoLTE call over the helikite’s small cell, using satellite backhaul. Data speeds weren’t shown, but EE has tested up to 50Mbps in the field.

The company is still proving the concept and evolving the technology. However, it has used it during planned commercial events. For example, EE recently used a helikite to broadcast a 360-degree live stream from the Red Bull Foxhunt downhill mountain-biking event in Wales. The ‘air mast’ was flown 300 feet above Machynlleth, on the edge of Snowdonia National Park, to boost 4G coverage and capacity. A team of Ericsson field engineers worked in support of the EE Special Projects Division.

On the UAV side, EE also demonstrated a drone equipped with a base station and antenna at the Oval event, while AT&T has used an LTE-equipped Flying COW (Cell on Wings), which can stay in the air for an extended amount of time to provide Puerto Rican consumers with temporary service; it can span 40 square miles with cellular coverage and support up to 8,000 users simultaneously.

The need for simplicity
Karim El Malki, Athonet’s CEO, says that his company, which in addition to its wider cellular offerings provides drone-, vehicle- and backpack-mounted LTE systems, has worked hard to put “leading edge” technology such as video broadcasting into the hands of emergency workers.

“But I always like to [stress] simplicity as well. We really put a lot of effort into making the solution easy to use for the IT team and the emergency responders and removing the need for telco competence. We don’t assume that they have any specific knowledge, therefore we’ve created a web-based GUI through which all operations can be made, including the management of the system, its configuration, viewing potential and alarms.

“We’ve also made our systems really easy to deploy. You just switch them on and the networks get up and running in a pre-configured basis and they can be easily adapted. We’ve also done this in a virtualised environment in some cases.”

El Malki adds that for Athonet, network slicing is already here and the company can allow one of its networks to be operated separately by different teams, giving each team the quality of service and network resources they require. He says that this is a more cost-effective approach than having separate networks for each type of responder.

Athonet’s rucksack-mounted LTE system, the PriMo Rucksack, and its drone-mounted small cell, which supports voice over LTE calls, are “attracting a lot of interest”, and El Malki says the main reason for this is “there are situations where the bigger vehicle-transported units just cannot reach certain areas. A number of countries such as Italy have areas like valleys where it is very hard to bring a vehicle in an emergency, while a person could get there. We’ve had several earthquakes unfortunately and in several of these cases this type of technology would have been fundamental. And those areas where there is no pre-existing coverage, the digital divide areas, that’s the worst position you can be in,” he explains.

Hytera Mobilfunk’s Pauer says his company is looking at LTE mesh networking that can cover “even a large area in a fairly fast amount of time, because you don’t need backhaul for each base station”. He adds that meshing technology allows the coverage area to be easily extended just by adding more nodes, be they vehicles, drones or people with rucksacks, on an ad hoc basis. He adds that Hytera’s LTE mesh base station weighs about 5kg, making it light enough to mount on drones. The base station was showcased at CCW and will be commercially available around Q2 2018.

One of the considerations with mesh networking is the cumulative latency seen when voice and data have to travel through many hoops to reach their destination. However, Pauer stresses Hytera’s view that TETRA is the key critical technology for voice, while Holm says that Hytera designed its voice-over-LTE service to be very latency-tolerant. “Of course it is annoying for the users, but it does not interfere with the service – it keeps going even if you have a huge voice delay.”

Speaking of TETRA, Pauer says Hytera’s DIB-R5 outdoor system is a full base station in a single IP65-rated 10kg box and can get up and running within five minutes of being installed. He claims that it consumes half as much power as similar systems from Hytera’s competitors. This, he says, makes a huge difference when it comes to trying to power the unit with solar panels. While the DIB-R5 is already available in Asia, it will be making its European debut at PMRExpo.

He notes the importance of the control room and its supporting infrastructure in disaster recovery “to also make the right decisions. One of the key strategic things in our portfolio are the control room applications. Providing our customers an end-to-end solution helping with situation awareness while having a quick and easy-to-use solution is the philosophy of Hytera’s product strategy and an area where we keep investing heavily. There’s no point having front-end infrastructure if you don’t have a control room that allows you to make the right decisions using all the data you’re collecting.

“It’s not just enough to put an LTE base station up using public safety spectrum, you must make sure that in the back end, the networks are converged (through our SmartOne dispatcher solution, for example), allowing the operator to make the right decisions in a disaster scenario.”

Pauer also highlights body-worn video cameras’ value during natural disasters. “It takes time to describe a situation using voice, while a remote speaker microphone video capture can immediately show the condition of critical infrastructure, such as dams. The control room can instantly see the exact situation a police officer is in, while describing it through voice always leaves a lot of room for interpretation. With video, the control room can immediately capture the seriousness of the situation.” This is even more valuable in situations where the observer on the ground might not be able or experienced enough to accurately assess the extent of the damage.

Given the impact of climate change and statistics showing that the number of disasters has more than quadrupled to around 400 a year since 1970, operators can expect their networks to be put to the test more frequently. It’s a good thing then that so many minds are focused on ensuring first-responders get the service they need, even when our power over nature is made to look very small indeed.