Lead Observability Engineer

Key Responsibilities

• Own the design, deployment, and lifecycle management of the Splunk Enterprise platform, including indexer and search head clustering, forwarders, and knowledge objects.
• Define and implement best practices for data onboarding, parsing, enrichment, and storage to support observability use cases.
• Collaborate with infrastructure, DevOps, security, and application teams to build reliable, scalable observability solutions.
• Develop advanced SPL searches, correlation rules, alerts, and performance dashboards.
• Improve alert quality and reduce noise through smarter event correlation and visualization.
• Drive observability maturity initiatives including logging standardization, automation, and self-service access to telemetry data.
• Evaluate and integrate additional observability and monitoring tools (e.g., Prometheus, Grafana, LogicMonitor, AppDynamics, Dynatrace, etc.) to complement existing capabilities.
• Lead troubleshooting and incident response efforts where visibility and telemetry data are required.
• Mentor junior engineers and influence platform and observability architecture decisions.

Qualifications

• 8–12 years of progressive experience in observability, infrastructure monitoring, or SRE roles.
• Minimum 7 years of direct hands-on experience with Splunk Enterprise at enterprise scale.
• Deep knowledge of Splunk architecture, including clustering, ingestion pipelines, search performance tuning, and index lifecycle policies.
• Advanced proficiency with SPL (Search Processing Language) and dashboarding.
• Experience building and scaling log pipelines using technologies such as syslog, Fluentd, Logstash, Cribl, etc.
• Familiarity with cloud platforms (AWS, Azure, or GCP) and hybrid infrastructure environments.
• Experience working with configuration management and infrastructure-as-code tools (e.g., Terraform, Ansible).
• Excellent collaboration, problem-solving, and communication skills.

Must Have

• Splunk certifications (e.g., Certified Architect, Consultant, or Admin).
• Experience with APM and tracing tools (e.g., OpenTelemetry, Jaeger, New Relic, etc.).