Fencewire, the full simulation.
A rural ISP whose switchboard is overwhelmed during every winter storm as hundreds of subscribers call about the same outage. The case followed on this page, the Lolo Peak relay outage, on a Tuesday before dawn in December, during a snowstorm.
A rural ISP whose switchboard is overwhelmed during every winter storm as hundreds of subscribers call about the same outage.
Every outage explained to subscribers before their first call.
- Fencewire delivers broadband access to fourteen thousand homes and businesses across western Montana valleys, deploying fibre in townships and fixed wireless links from high-altitude relay towers. The Missoula-based firm employs sixty staff, including a nine-person customer service team.
- When a snowstorm cuts power to a relay site, hundreds of subscribers lose their connection simultaneously. The switchboard becomes flooded, and advisers spend the day repeating the same explanation. Meanwhile, field engineers only discover the full scale of the fault through incoming calls.
- American internet service providers must file broadband deployment data with the FCC biannually, on 1 March and 1 September, on a location-by-location basis. The federal programme assisting low-income households with broadband bills ended on 1 June 2024, prompting many Fencewire subscribers to alter their plans.
- An agent monitors network alerts around the clock. When a relay goes offline, it identifies the affected subscribers and alerts the on-call engineer, including the probable cause.
- It prepares notifications for subscribers enrolled in service alerts and updates the chat interface with the latest incident status. The network engineer reviews and signs off every message before dispatch.
What remains with the teams
The network engineer directs all remediation work and signs off every customer communication. Customer support retains sole authority over goodwill concessions, and the compliance officer certifies every FCC filing.
- Subscribers notified before their first call
- Inbound calls logged during an outage
- Time from alert to engineer dispatch
- Corrections requested on FCC filings
Broadband coverage filing
Twice a year, the agent reconciles the list of served addresses, relays and subscriptions, then flags discrepancies with the FCC public map.
The compliance manager checks and certifies the filing.
New connection requests
For each request, the agent checks whether the address is served by fibre or wireless, and prepares the technical site visit.
The technician decides on feasibility on site.
Pre-storm relay inspections
Before each forecast snow event, the agent checks battery levels and generator fuel for each relay.
The network manager decides on each relay climb.
Post-outage subscriber churn
The agent identifies subscribers who cancel after an outage and prepares a monthly review of cancellation reasons for customer service.
The customer service manager decides on each goodwill gesture.
the Lolo Peak relay outage, on a Tuesday before dawn in December, during a snowstorm
A relay outage in the middle of a storm, before and after.
- The alarm
- The relay alarm gets lost among dozens of others during the night.
- The agent spots the relay outage and identifies the affected subscribers.
- First workstream
- On-call duty
- The on-call engineer learns of the outage from the first morning calls.
- The agent alerts them within minutes, with the probable cause.
- First workstream
- Subscribers
- Hundreds of subscribers call about the same outage, overwhelming the switchboard.
- Subscribers who opted to be notified receive a message approved by the engineer.
- First workstream
- Live chat
- Advisers repeat the same answer all day long.
- The agent responds in the chat with the outage status and hands over specific requests.
- First workstream
- Reporting
- The coverage filing is prepared in a rush, a few days before the deadline.
- The agent reconciles addresses and subscriptions well before the deadline.
- Workstream 2
- Connection request
- The technician travels on site only to discover the address is not covered.
- The agent checks network coverage before scheduling the visit.
- Workstream 3
- Monitor
- It monitors relay and link alarms day and night.
- Identify scope
- When equipment goes down, it identifies affected subscribers using the network map.
- Alert on-call staff
- It alerts the on-call technician with the probable cause and site access details.
- Draft the notification
- It drafts the message for subscribers who opted to be notified.
- The network engineer approves every message.
- Respond
- It responds in the live chat with the outage status and hands over any specific requests.
- Customer advisers handle specific requests.
- Track
- It tracks the repair until service is restored and drafts the resolution message.
- The network engineer approves the message.
- Archive
- It maintains a history of each outage, including affected subscribers and its duration.
- Report
- Each month, it prepares a review of outages and the calls received during each one.
- Network monitoring
- Alarms from radio relays and the fibre network.
- Read-only
- Network map
- Radio relays, links, fibre sections, and served addresses.
- Read-only
- Subscriber software
- Subscriptions and contact preferences.
- Read-only
- Website live chat
- Interactions with subscribers.
- Outage status responses
- Subscriber messaging
- Messages intended for subscribers who opted to be notified.
- Sent after approval
- Managing Director
- Sets the agent rules with us and chooses the next workstream.
- Network engineer
- Decides on each repair and approves every message sent to subscribers.
- Customer service
- Handles specific requests and decides on goodwill gestures.
- Compliance manager
- Reviews and certifies every FCC filing.
- The FDE
- Leads the engagement and writes the agent rules with the team. They are accountable for every agent.
- Network agent
- Monitors alarms and identifies affected subscribers.
- Subscriber agent
- Drafts messages and responds in the live chat.
- Compliance agent
- Prepares coverage filings.
- Audit agent
- Compares detected outages with actual incidents weekly and flags recurring errors to the FDE.
- The agent announces an outage that does not exist.
- No message is sent without approval from the network engineer.
- A subscriber facing difficulties receives no answer.
- The agent hands over to an adviser as soon as an inquiry goes beyond outage status.
- A message is sent to a subscriber who did not request it.
- Only subscribers who opted to be notified receive messages.
- The FCC filing contains an error.
- The agent flags discrepancies, and the compliance manager checks and certifies the filing.
- Subscriber data is leaked.
- Addresses and contact details remain within the Fencewire system, and all access is logged.
- We measure the baseline over one month, such as notified subscribers, calls received during outages, technician dispatch times, and corrections to the previous filing. The network engineer co-writes the alerting rules with us.
- The agent monitors alarms alongside the on-call team for two weeks, and we compare its alerts against actual incidents.
- Messages to subscribers are sent after approval by the network engineer, and the agent responds in the live chat.
- We compare against the baseline and choose the next workstream with the Managing Director.
- Subscribers notified before their first call
- Share of affected subscribers who receive the message before calling.
- Calls received during an outage
- Calls per outage, compared to the baseline month.
- Time between the alarm and technician dispatch
- Time gap recorded between monitoring systems and the on-call rota.
- Corrections requested on the FCC filing
- Corrections and challenges following the filing, compared to the previous submission.