Watchbill, the full simulation.
A company that monitors US SMEs’ systems day and night, whose analysts are swamped with alerts. The case followed on this page, a suspicious login to an accountant's account, at a financial client, in the middle of the night.
A company that monitors US SMEs’ systems day and night, whose analysts are swamped with alerts.
Every alert reaches the analyst with its case file.
- Watchbill monitors the workstations, servers, email accounts and cloud access of US SMEs, from New York and Chicago. Its seventy people, mostly analysts, work in shifts round the clock. Some of its clients are financial firms subject to the State of New York’s cybersecurity rules.
- Detection tools produce thousands of alerts per week, and most turn out to be harmless. At night, two analysts sort the queue by hand and gather the details of each incident across several consoles.
- For a financial client, time is critical. They must notify the New York regulator within 72 hours after establishing that an incident has occurred, including when the incident affects a provider such as Watchbill.
- An agent takes each alert as soon as it drops. It groups those that relate to the same event and adds the context of the affected workstation and account. Classification follows the rules written with the analysts.
- For each probable incident, the agent prepares the case file, with the timeline, affected systems, indicators observed and open questions. For a financial client, it adds the facts the client will need if they must notify the regulator.
What remains with the analysts
Analysts decide each response, from isolating a workstation to blocking an account, and the agent never acts alone on a system. The client alone decides to notify the regulator.
- Time from alert to initial handling
- Alerts processed by analysts each night
- Incident case files complete on first review
- Time taken to deliver the file to the client
Monthly reports
Each client receives a monthly report detailing incidents during the period, pending patches, accounts to monitor and analyst recommendations.
The analyst responsible for the client reviews each report before it is sent.
Tuning detection rules
The agent identifies the rules that generate the most unactioned alerts and suggests adjustments, each accompanied by the alerts that justify it.
The detection engineer accepts or rejects each adjustment, and no rule is deactivated without them.
Emails reported by employees
Client employees forward suspicious emails to a dedicated address. The agent analyses each one and replies to the employee. Any malicious email is immediately escalated to an analyst.
The analyst decides whether to block the sender for the entire client.
Security questionnaires
Watchbill clients themselves receive security questionnaires from their own customers. The agent prepares the responses based on each client's documentation.
The client's head of security signs off each response.
a suspicious login to an accountant's account, at a financial client, in the middle of the night
A night-time alert, before and after.
- The alert
- The alert arrives amidst hundreds of others, and the night analyst sees it forty minutes later.
- The agent groups it with the preceding failed logins and places it at the front of the queue.
- First workstream
- The context
- The analyst opens four consoles to find out who the accountant is and where the login is coming from.
- The case arrives with the accountant's login history and the context of their role.
- First workstream
- The decision
- Lacking information, the analyst hesitates and waits until morning to call the client.
- The analyst decides to block the session and reset the password, and the agent logs their decision.
- First workstream
- The case file
- The report to the client is written the next day from memory.
- The case file is maintained throughout the night, with the timeline and remaining open questions.
- First workstream
- The regulator
- The client discovers the incident late and struggles to gather the facts for the New York regulator.
- They receive the facts they need to decide, within their 72-hour window, whether they must notify the regulator.
- First workstream
- The following month
- The client does not know what was blocked on their systems during the month.
- Their monthly report covers each incident and pending remediation.
- Workstream 2
- Receive
- It takes each alert from detection tools as soon as it arrives.
- Group
- It groups alerts that relate to the same event.
- Enrich
- It adds the context of the account concerned and the client's alert history.
- Classify
- It classifies each group according to the triage rules written with the analysts.
- The analysts maintain the triage rules.
- Prepare the case file
- For a probable incident, it prepares the timeline, the affected systems, the evidence gathered, and the open questions.
- Alert
- It notifies the on-call analyst and sends them the case file.
- The analyst decides on each response.
- Log
- It logs each analyst decision and each action taken, with the timestamp.
- Prepare the client update
- It drafts the update for the client from the case file, with facts relevant to the regulator when the client is a financial firm.
- The analyst reviews the update before sending, and the client decides whether to notify the regulator.
- Detection tools
- Alerts raised across each client.
- Read-only
- Client directory
- User accounts and their roles.
- Read-only
- Incident management tool
- Case files and the history for each client.
- Case creation and updates
- On-call messaging
- The channel that reaches the on-call analyst, day or night.
- Sending alerts and case files
- Decision log
- The record of each analyst decision.
- Append-only, no deletion
- Analysts
- Decide on each response and maintain the triage rules.
- Detection engineer
- Tunes detection tools and approves each rule change.
- Account managers
- Maintain the relationship with each client and review reports.
- Head of security
- Sets the agent permissions and audits the decision log.
- The FDE
- Leads the engagement and writes the agent rules alongside the analysts. They are accountable for each agent.
- Triage agent
- Groups and classifies alerts.
- Case preparation agent
- Prepares incident case files.
- Reporting agent
- Prepares client updates and monthly reports.
- Auditing agent
- Checks a sample of dismissed alerts and reports any discrepancy to the FDE.
- A real attack is mistaken for a false positive.
- An analyst reviews a sample of dismissed alerts daily, and the triage rules are adjusted with them.
- The agent acts alone on a system.
- It has no action permissions. Blocking an account or isolating an endpoint remains an analyst decision.
- An attacker slips instructions into the content of an alert.
- The agent cannot execute anything, and its triage rules only change with an analyst.
- The client misses their 72-hour deadline.
- The case file is maintained as the incident unfolds, and facts relevant to the regulator are sent to the client as soon as the analyst approves them.
- Client data leaks.
- The agent reads without the ability to export. Every access is logged, and data remains hosted in the United States.
- We measure the baseline over the previous month, including alerts received, response times, alerts handled each night, and delivery times for case files. The analysts write the triage rules with us.
- The agent triages in parallel with the analysts, without sending them anything, and we compare its classifications to theirs.
- Nights switch to the agent for a first group of clients, and an analyst reviews every dismissed alert.
- All clients switch to the agent. We compare against the baseline and select the next workstream with leadership.
- Time between an alert and response
- Gap between the alert and an analyst opening the case file, recorded in the incident management tool.
- Alerts handled by analysts each night
- Alerts that an analyst must examine directly between 22:00 and 06:00, compared to the baseline month.
- Incident cases complete on first review
- Proportion of case files where the analyst did not have to search elsewhere, recorded by them at closure.
- Time to deliver the report to the client
- Gap between the discovery of an incident and sending the update to the client.