Episode 25 — Rate sources and evidence with discipline
In Episode 25 — Rate sources and evidence with discipline, we are going to take a topic that feels administrative at first and show why it is actually one of the most practical skills in threat intelligence. When you are surrounded by alerts, reports, chat messages, and secondhand summaries, your real job is not to collect more information. Your job is to decide what deserves trust, what deserves caution, and what deserves to be held back until it can be verified. A formal rating system gives you a consistent way to make those decisions without relying on mood, urgency, or whoever sounds most confident. The discipline here is not about being skeptical of everything. It is about being precise about confidence, so your stakeholders receive intelligence that has been weighed, not just repeated.
The main idea is that reliability is not a vibe, it is a measurable property that you can evaluate and communicate. In practice, teams often treat reliability as an unspoken assumption, and that creates confusion because different analysts carry different internal thresholds. One person treats a vendor report as nearly certain, while another treats it as a starting hypothesis that must be validated locally. Without a shared system, those differences stay hidden until a decision goes wrong, and then everyone argues about what they thought the data meant. A disciplined method forces you to separate two things that people often mix together: how much you trust the source, and how much you trust the specific information being reported. That separation matters because even the best sources can be wrong sometimes, and even unknown sources can occasionally deliver accurate observations.
A widely used approach for making that separation visible is the Admiralty Code, which assigns a letter and a number to each report or key claim. The letter describes the reliability of the source, while the number describes the credibility of the information content. This pairing is powerful because it gives you a compact way to say, I believe this source tends to be dependable, but I am not yet convinced this particular claim is true, or the reverse. It also helps when information travels across teams and time, because your reasoning does not depend on you being present to explain it. Another analyst can read the rating, see your notes, and immediately understand how heavily to weigh the claim. The important point is not to worship the code itself. The point is to use a shared, transparent language for confidence so decisions are not based on implied trust.
When you grade source reliability using a letter from A to F, you are answering a specific question: based on known history, how likely is this source to be accurate and truthful in general. An A source is considered consistently reliable, often because it has a track record of reporting correctly, providing verifiable detail, and behaving predictably over time. A B or C source might be generally reliable but with limitations, such as occasional errors, incomplete context, or a tendency to report with bias toward certain interpretations. A D or E source may have a pattern of inaccuracies, exaggeration, or inconsistency that reduces confidence before you even look at the content. An F source is typically considered unreliable to the point that you should treat it as untrusted unless independently confirmed. This letter is not a moral judgment. It is a performance assessment tied to how the source behaves.
One of the easiest mistakes to make is to assume that a highly reliable source always provides completely accurate information. That assumption feels efficient, and sometimes it even works, but it is the kind of efficiency that creates expensive surprises. Reliable sources can still be wrong because they can be deceived, they can rely on incomplete telemetry, they can misinterpret technical artifacts, or they can compress nuance into a simplified narrative. They may also have incentives that shape what they emphasize, such as a desire to appear first with a breaking story or to support a product storyline. Even internal sources can be wrong, including a trusted system or a respected teammate, because every system has blind spots and every person has assumptions. Discipline means treating source reliability as a prior, not a guarantee. You start with a level of trust, but you still test the specific claim against evidence and context.
That is where the second half of the code matters, the rating of the information itself from one to six based on probability. This part asks, independent of who said it, how credible is the specific content given what you know right now. A high credibility rating is reserved for information that is confirmed by independent sources, directly observed, or strongly supported by multiple consistent artifacts. A mid level rating might fit information that is plausible and partially supported but still missing key confirmation. A low credibility rating fits information that is unconfirmed, internally inconsistent, or in tension with what you would expect to see if it were true. The value of the numeric rating is that it forces you to make the probability question explicit instead of hiding it behind confidence language. Instead of saying this seems likely, you communicate that you have evaluated it and assigned it a defined level of credibility.
Now picture a realistic moment where this discipline matters, such as scoring a new intelligence report before it reaches senior leadership. Leaders rarely have time to decode raw uncertainty, and they will naturally treat anything presented as an intelligence product as something that has already been vetted. If you pass along a claim without clarifying its reliability, you are effectively transferring your uncertainty into their decision space without warning. A formal rating helps you decide what belongs in the primary message and what belongs in a caveat, a watch item, or a request for additional collection. It also shapes how you write, because you will avoid definitive language when the information rating is low. When your stakeholder asks how sure you are, you will not have to improvise. You will have a documented rating and the reasoning behind it, which is exactly what a mature intelligence function should provide.
It can help to think of these ratings as a quality seal on your intelligence data products, but a seal that comes with a specification, not a vague promise. A quality seal does not mean perfect, it means inspected, categorized, and labeled so others can use it appropriately. In threat intelligence, the seal tells consumers how much weight to put on a claim, how aggressively to act, and how much verification to require before making high impact moves. This also improves collaboration, because analysts can exchange findings without having to negotiate confidence from scratch every time. Over time, the seal becomes part of your team’s trust fabric, because people learn that when a report carries a certain rating, it will be backed by consistent standards. That consistency reduces fatigue and cynicism, because consumers stop feeling like they are being whiplashed by shifting certainty. It also makes it easier to learn from mistakes, because you can review whether the ratings matched reality and adjust your approach.
A common challenge is handling information from a completely new and unknown source, because you do not have history to lean on. In that case, the right move is to be honest about what you do not know and start the source at an appropriately cautious reliability level. You can still use the information, but you should treat it as a lead rather than a conclusion, and you should prioritize verification steps that can either corroborate or refute it quickly. Unknown does not automatically mean wrong, and it does not automatically mean right. It means uncalibrated. The disciplined approach is to look at internal indicators of quality, such as whether the source provides specific, falsifiable details, whether the claims align with known technical constraints, and whether the narrative avoids the telltale signs of rumor. You then update the source letter only after repeated performance, not after one lucky hit.
This discipline has a direct operational payoff because it prevents teams from acting on low quality or unverified rumors. Rumors spread fast in security because they often carry urgency, and urgency can hijack attention even when evidence is thin. When your team has a formal rating habit, a rumor gets labeled quickly as low credibility, and that label creates a barrier against impulsive action. It does not mean you ignore the rumor. It means you treat it as a hypothesis that triggers targeted validation rather than broad disruption. Without ratings, teams can waste hours chasing stories that collapse under basic scrutiny, and that cost is not just time. It is lost trust, because stakeholders notice when intelligence products swing wildly and then quietly retract. The goal is to move fast on what is solid and move carefully on what is fragile, and you cannot do that if everything looks equally important.
Consistency in rating is what makes the system useful to other analysts, because it teaches them what a given rating means in practice. If one analyst uses A for almost everything and another analyst uses A only for the rarest cases, the rating becomes noise rather than guidance. Consistency does not require perfection, but it does require shared definitions and calibration conversations over time. A good team revisits examples, compares how different people scored them, and aligns on what the scale should represent in their environment. This is especially important when teams grow or when responsibilities shift, because new analysts inherit rating norms whether those norms are explicit or not. When the norms are explicit, the team can onboard faster and avoid silent divergence. When the norms are silent, confusion grows until a decision forces a confrontation. Disciplined rating is as much about team coherence as it is about individual judgment.
Another way to strengthen confidence is to compare two different sources describing the same event and see whether they match in meaningful ways. Matching here does not mean repeating the same headline. It means aligning on technical particulars that would be hard to guess, such as timing patterns, observed behaviors, infrastructure relationships, or consistent indicators that appear across independent telemetry. When sources match on those details, your information credibility rating should rise, even if each source individually has limitations. When sources disagree, the disagreement is itself valuable data, because it tells you where uncertainty concentrates. The disciplined move is to document the discrepancy rather than smoothing it away, and then decide what additional evidence would resolve it. This comparison habit also protects you from echo effects, where many reports appear to corroborate a claim but are actually repeating the same original statement. Independence matters, and you should treat repeated copying as a single source until proven otherwise.
Documentation is the part people skip when they are busy, but it is what turns a rating into a defensible analytic artifact. When you assign a letter and number, you should also record why, in plain language that another analyst can understand later. That reasoning might include the source’s history, the presence or absence of corroboration, the specificity of the claim, and any known limitations in collection coverage. The goal is not to write a long essay, but to capture the key drivers of your confidence so the rating is not arbitrary. Documentation also protects you from hindsight bias, because once an event resolves, it becomes easy to believe the outcome was obvious all along. With documented reasoning, you can review whether your rating process was sound given what you knew at the time. That is how teams improve without becoming defensive. It also makes peer review more effective, because reviewers can critique the logic rather than guessing what you were thinking.
Over longer periods, disciplined teams re evaluate source ratings as performance changes, because reliability is not a permanent trait. A source that was excellent last year might degrade due to staffing changes, shifting incentives, reduced visibility, or a change in methodology. A source that was unreliable might improve if it becomes better resourced or if it tightens its validation practices. If you never revisit ratings, you end up with stale priors that mislead you, and you will over trust sources that have quietly slipped. Periodic review does not have to be heavy. It can be anchored to outcomes, such as how often a source’s claims were later confirmed, how often they were corrected, and how actionable their detail was in your environment. This is also where you separate honest error from systemic unreliability. Everyone can be wrong sometimes, but patterns of avoidable error should move the letter grade over time.
The deeper benefit of all of this is that it makes your intelligence products more stable under stress. When pressure rises, people tend to reach for shortcuts, and source credibility is one of the first places where shortcuts show up. A disciplined rating habit is a guardrail that keeps you from passing uncertainty as certainty, even when the organization wants a fast answer. It also improves communication because you are not forcing consumers to interpret your tone or guess what you mean by likely. You are providing a defined statement of confidence, backed by reasoning, that they can factor into decisions. This discipline also reduces internal conflict because teams argue less about personalities and more about evidence. When the process is visible, disagreements become constructive rather than political. Over time, your stakeholders learn to trust not only your conclusions, but your restraint, which is often the harder skill.
Conclusion: You know the traps so ask a peer to review your work. In the same spirit, reliability is measurable so you can treat source grading as a normal, recurring practice rather than an occasional formality. A practical approach is to choose your top sources, apply the letter and number ratings with documented reasoning, and then review them with a peer so your standards stay calibrated. If you aim to do that by this Friday, the real win is not the date on the calendar, it is the habit you reinforce. You will notice which sources you have been over trusting, which ones you have been ignoring without reason, and where your team’s confidence language needs tightening. Once you start rating discipline consistently, your intelligence products will carry clearer weight, your stakeholders will make better decisions with less friction, and your team will spend less time chasing noise. That is what source rating is really for, not bureaucracy, but better judgment made visible.
