NG Solution Team
Cybersecurity

How can specialized agents improve security alert triage on Databricks?

Databricks processes vast amounts of security logs from diverse sources such as endpoint security tools and cloud activity logs, which are analyzed for potential threats. Traditionally, security teams prioritize alerts by severity, focusing on high and medium alerts while low-severity alerts are addressed as resources allow. However, at Databricks, the Incident Response (IR) team manages alerts of all severity levels, leading to a need for efficient triage methods.

To enhance the handling of low-severity alerts, Databricks implemented specialized agents tailored to specific detection sources. These agents, each focusing on a single source, are designed to provide context-specific triage, improving accuracy and reducing false positives. A dedicated Threat Intelligence agent supports these efforts by providing additional insights when necessary.

The system employs deterministic filtering to suppress benign alerts and enriches context before involving language models (LLMs) for reasoning. This structured approach allows agents to escalate alerts that truly require further investigation, significantly improving the identification of genuine threats. The agents have successfully triaged thousands of alerts, saving analyst time and increasing the likelihood of identifying true positives.

Key findings from this approach include a dramatic reduction in false positives and the identification of suspicious domains and policy violations. The use of Databricks’ own platform tools, such as Spark Structured Streaming and MLflow Tracing, has been instrumental in building this efficient security operation. Future developments aim to enhance these capabilities with natural language processing tools to further empower analysts in threat investigation.

Related posts

Emergency supplies: UK urges households to keep three days’ kit

David Jones

Levi Strauss cyber breach: social engineering led to data extraction

James Smith

Check Point Issues Hotfixes for Actively Exploited Zero-Day

David Jones

This website uses cookies to improve your experience. We assume you agree, but you can opt out if you wish. Accept More Info

Privacy & Cookies Policy