August 2019 Outages Explained
Photo Credit: Hannah Whisenant.
Only a very small piece of the NWPB system for audio delivery.
If you were listening the first week of August, you may have heard the audio dropouts that were plaguing Northwest Public Broadcasting. Here is a recap of what happened. At the bottom of this recap is a list of solutions NWPB is working on to reduce some points of failure in our system.
Tuesday, August 6th
A failed power supply in our equipment room took all NWPB stations off the air. The engineering staff in Pullman rerouted the power to get the equipment powered again and get NWPB back on the air.
10 minutes later, NWPB went off the air again. It was then discovered the T-1 was no longer connecting, this was purely coincidence and not related to the power outage. The T-1 Line is a high-speed dedicated landline that carries all audio and digital commands to our satellite up-link site in Denver, Colorado.
After about 10 to 15 minutes, the back-up connection to the satellite up-link was established via the internet.
Wednesday, August 7th
While on the backup internet link, our telecommunications (telecom) provider tested the line, and surmised the problem was with a different telecom carrier on our T-1 Line. No resolution.
Then, in the late morning, there was a disconnection in our system in the Pullman building that affected the automation system. All the Classical stations went off the air. The connection between the audio databases shut off. It was rebooted and the Classical stations were restored. That was resolved, thanks to our IT department.
At this point, NWPB is still operating on backup audio to Denver. And with it, another issue arose. In Denver, in their system, a small piece of equipment called a “silence sensor” was malfunctioning, occasionally knocking the Classical stations off the air. The sensor was switched off, and was supposedly fixed Wednesday afternoon.
Thursday, August 8th
In the morning, that same silence sensor knocked Classical stations off the air, again. This time, the team in Colorado completely rewired the path and bypassed the sensor on the Classical channel and took it out the system completely on Classical. That resolved the issue on Classical.
Still no resolution from any telecom provider about the T-1 line. Our engineering team tries to escalate the ticket as far as possible as the outage has been almost 72 hours at this point.
Friday, August 9th
Friday around 1:00 AM, the silence sensor malfunctions again, this time on the News service. The silence sensor is completely removed and bypassed for all audio streams, which seems to resolve the issue.
All throughout this process, NWPB is airing ID’s in an abbreviated fashion, in order to remain FCC compliant with station identification.
Late afternoon on Friday, the telecom company finally isolates the problem to a central matrix. This is after four days of malfunction with their equipment.
They decide they will return to the issue on Monday.
Monday, August 12th
Telecom troubleshooting begins, but it is a slow resolution as two telecom providers must work together and there is a lot of miscommunication between them.
Tuesday, August 13th
Finally, exactly one week after the T-1 line was interrupted, the telecom companies finally fix their issue that took us off the air. According to their email, it was a “DSX Panel” which stands for a “Digital Signal Cross-Connection Panel”. It’s basically a newer version of a switch panel that telephone operators used to connect different lines.
Thank you for your patience during this time.
The Future:
Every time there is a major issue, steps are taken to resolve it, and hopefully prevent it in the future. While the DSX Panel issue was not within NWPB’s control as it happened outside our building and outside the organization, there are steps that will be taken to address some underlying issues with the current system.
What was discovered during this process is several projects IT and Engineering will accomplish to fortify NWPB’s systems.
- Purchase and install a more robust power supply and upgrade it with sensors so if it loses power, it will switch our equipment to a backup electrical system.
- Upgrade the T-1 line to a newer system. Apparently, the T-1 system is antiquated and its infrastructure is gradually less and less maintained by telecom companies. This newer system is currently used at many large broadcast organizations and NWPB will be moving in that direction.
- Placing the automation file servers in a room directly connected to the larger system, bypassing the failed connection that happened Wednesday, August 7th.
- Beginning a schedule of reboots and maintenance on all equipment. There are over a hundred pieces of equipment, and they all run 24/7, every day of the year. With that in mind, it is extremely reliable, however, all equipment needs a breather occasionally.
These steps will not fix all the issues involved in a highly technical system. However, it will improve the systems to make them less vulnerable to certain issues.
All these projects will incur costs to update and upgrade. The business of broadcast dictates keeping up with current technology, regardless of whether the public knows what it is or not. What you hear is programs. But you’re hearing it because of all the infrastructure and technology NWPB invests in – and must continue to invest in – to remain on the air for you.
Help invest in the future of this station. You can find more information by clicking here.
