AI Hardware + Software Company (Stealth) | San Francisco, CA | Hybrid
 

About the Role

We are seeking a founding-team Senior Inference Engineer to join our core SpeechLLM lab. You will be responsible for building and optimizing high-throughput, ultra-low-latency inference engines for large language models and foundational speech models. This is a rare opportunity to define the architecture of a next-generation AI system that natively understands and generates speech as part of a hardware-software intelligence platform.
 

You May Be a Good Fit If You

  • Have hands-on experience building and deploying high-throughput, ultra-low-latency inference engines for large language models or foundational speech models.
  • Understand the intricate tradeoffs between latency, throughput, and Time-To-First-Token (or Time-To-First-Audio) in real-time streaming environments.
  • Have practical experience with continuous batching, KV cache management (e.g., PagedAttention), and stateful connections for real-time conversational AI.
  • Possess a deep understanding of GPU architectures (NVIDIA Ampere/Hopper) and the memory hierarchy, allowing you to identify and eliminate hardware bottlenecks.
  • Communicate clearly and collaborate effectively at the intersection of core ML training and backend infrastructure teams.
  • Thrive in fast-moving environments and genuinely enjoy the systems engineering challenge of squeezing every last drop of performance out of a GPU cluster.
 

Strong Candidates May Also Have

  • Frontier Serving Frameworks: Deep, under-the-hood familiarity with modern LLM serving frameworks such as vLLM, TensorRT-LLM, SGLang, or NVIDIA Triton Inference Server (open-source contributions are a strong plus).
  • Real-Time Audio Streaming: Experience handling continuous audio streams over WebSockets or WebRTC, deploying neural audio codecs, and managing chunked audio generation to minimize conversational latency.
  • Advanced Inference Techniques: Hands-on implementation of speculative decoding, lookahead decoding, or chunked prefill techniques.
  • Model Compression and Quantization: Experience with post-training quantization (PTQ) and deploying models in FP8, INT8, AWQ, or GPTQ without degrading audio naturalness or ASR accuracy.
  • Large-Scale Distributed Systems: Experience deploying multi-GPU (Tensor Parallelism) and multi-node inference pipelines and managing autoscaling infrastructure via Kubernetes.
 

What We Offer

  • Meaningful early-stage equity and the opportunity to be a foundational member of a core research and engineering lab.
  • Highly competitive base salary: $200K to $540K depending on experience, plus performance bonus and equity.
  • Top-tier healthcare for employees and dependents, including dental and vision coverage with a generous employer subsidy.
  • 401(k) with company matching.
  • Unlimited PTO plus 13 paid holidays.
  • 12 weeks paid parental leave regardless of gender.
  • Hybrid working model with a minimum of 3 days per week in-office.
  • Top-of-the-line laptop or workstation, annual team offsites, and a fully stocked office.