Why join Upscale AI

Upscale AI is building the high-performance infrastructure powering the next generation of artificial intelligence. Backed by over $300M in funding and rapid global adoption, we are scaling systems designed for the world’s most demanding AI workloads.

We focus on first-principles engineering across silicon, systems, and networking—where performance, scale, and execution are critical. Our team is talent-dense and high-performing. We value ownership, technical rigor, and speed, and we offer the opportunity to work on foundational problems with immediate, real-world impact.

If you’re looking to do high-impact work, move fast, and help define the infrastructure behind the future of AI—Upscale AI is where you can produce meaningful work at the frontier—and operate at a high standard.

Upscale is building the engineering team that drives product improvements though hands-on field engineering. The Forward-Deployed Engineer (FDE) owns complex fabric and system issues end-to-end: from first report through root-cause analysis, workaround delivery, and verified fix. You resolve them with the rest of the team, or you drive them to resolution across whatever boundary stands in the way.

This role is new to UpscaleAI and sits at the intersection of support, engineering, and product management. You will work directly with customers running production AI fabrics, reproduce issues in the lab, develop workarounds under pressure, and contribute fixes and diagnostic content back into the product and our AI-driven monitoring agent.

The environment is often unstructured. Problem definitions are incomplete. Documentation may not exist yet. The right candidate sees that as an opportunity, not an obstacle.

Large AI infrastructure operators increasingly expect their vendor's engineering team to function as an extension of their own infrastructure organization. This role is built to meet that expectation.

This is a small team. There will be an on-call component, but we are building a global team to reduce out-of-hours calls. We work with customers directly, so some travel is involved. Remote meeting tools handle much of the collaboration but building strong trust relationships will require time on site.

###

Key Responsibilities

Triage and resolve complex AI fabric issues: including silent packet drops, queue anomalies, NCCL stalls, gray failures, and performance degradation in production environments. You own the problem from intake to resolution.

Reproduce issues in the lab: building test scenarios that isolate L2 and L3 problems (packet loss, latency, retransmits, ECMP behavior, PFC/ECN interactions) and deliver reproducible cases to the development team when code fixes are needed.

Act as the escalation buffer for engineering: resolving issues without engaging development when possible, and packaging clean, reproducible problem statements when development engagement is required.

Train and improve the on-box and off-box AI agents: contributing field-validated detection signatures, classification logic, and resolution recommendations based on real cases. Your field experience directly shapes what the agent can identify and handle autonomously.

Validate AI agent accuracy: reviewing the agent's data collection, anomaly detection, and triage classifications against real-world outcomes. You determine when the agent is ready to advance from data collection to active triage to mitigation

Work with customers directly: understanding their fabric topology, workload patterns, and operational constraints. Communicate findings clearly to both technical and non-technical stakeholders.

Contribute to the product: identifying supportability gaps, proposing diagnostic improvements, and feeding field insights into the product management process. You are an active voice in what the product needs to become.

Maintain and evolve the AI agent's detection capabilities: update detection signatures, retrain classification models, and tune thresholds as customer fabrics scale, new hardware is deployed, and new failure modes are discovered in the field. The AI agent is an evolving system, not a shipped product.

Read and work with source code: engaging with developers at the code level when needed to understand behavior, identify root causes, or validate fixes. You do not need to be a full-time developer, but you must be comfortable in the codebase.

Requirements

5+ years of hands-on experience with high speed data center switching platforms at scale. Experience with AI fabric deployments is a strong plus but not required if switching and troubleshooting background depth is there. People who take ownership. The FDE role requires someone who treats every problem as theirs until it's resolved, regardless of where the root cause sits. If the issue crosses into ASIC behavior, NOS code, optics firmware, or customer configuration, you follow it there. Pointing to another team is not a resolution.

Deep Ethernet troubleshooting experience, including L2/L3 forwarding, ECMP, and packet-level analysis

BGP operations experience, including route reflectors, convergence behavior, and fabric-scale deployments

QoS configuration and troubleshooting: PFC, ECN, DSCP, queue management

Hardware support background: ASICs, FPGAs, chassis-based platforms, pluggable optic modules

Fiber and optical link troubleshooting, including DOM telemetry interpretation

Software support history: working with NOS, firmware, and driver-level issues

Linux proficiency (command line, system administration, log analysis)

Telemetry and monitoring: experience with streaming telemetry, counters, and event-driven diagnostics

Experience contributing to or training ML/AI systems (classification models, labeled data, feedback loops)

Lab skills: ability to design, build, and execute complex test scenarios that isolate specific failure modes

Clear written and verbal communication, including customer-facing interaction

Nice to have

Experience with SONiC or other open network operating systems

Familiarity with AI data center designs: low-latency fabrics, GPU cluster networking, NCCL, RDMA/RoCEv2

Experience with NVIDIA Spectrum switch family, BlueField SuperNICs, or ConnectX adapters

Understanding of Ultra Ethernet Consortium specifications and goals

PCIe architecture knowledge (relevant to NIC and accelerator integration)

Experience with Ixia/Keysight or similar network test equipment and test suites

Exposure to high-frequency telemetry (HFT) or WJH (What Just Happened) event data

Where you fall within that range depends on your experience, skills, and impact—we benchmark against internal levels to keep things fair and consistent.

Equal Opportunity

Upscale AI is building a team that reflects a wide range of perspectives, backgrounds, and experiences. We’re proud to be an Equal Opportunity Employer and consider all qualified applicants regardless of race, color, religion, national origin, sex, sexual orientation, gender identity, disability, or veteran status.

Accessibility & Accommodations

We’re committed to making our hiring process accessible to everyone. If you need accommodations at any stage, just reach out to us at [email protected]—we’re happy to help. Note: This inbox is only for accommodation requests.

Forward-Deployed Engineer, AI Fabric

Why join Upscale AI

Other open roles at Upscale AI(6)