Kubernetes System Architect

Kubernetes System Architect

Overview

Penguin's ICE Software products are used in the deployment, provisioning, management, and monitoring of some of the largest computational systems in the world. In this role as a Kubernetes System Architect, you will join our remote-first Software Engineering organization as a specialist focused on Kubernetes and container orchestration technologies as part of our cluster management software, ICE ClusterWare.

This role combines deep hands-on experience administering Kubernetes clusters with the ability to architect and guide large-scale, systems-level integrations. The successful candidate will serve as the technical bridge between modern containerized infrastructure paradigms and our cluster management initiatives, bringing critical Kubernetes expertise to enhance our capabilities for AI and High-Performance Computing (HPC) Linux-based environments.

As a member of our ICE Software Engineering team, you will work closely with software engineers, QA engineers, and our scrum master to design, prototype, and implement robust Kubernetes-based integrations. You will also partner with Product Managers, Solution Architects, and Product Architects to ensure Kubernetes-related initiatives align with product strategy and system scalability goals. This position requires both strong technical depth and architectural vision to translate complex container management concepts into actionable, maintainable, and product-ready solutions.

Responsibilities

Architecture & Design

• Define and architect Kubernetes integration strategies within the ICE ClusterWare platform to enable containerized workloads and hybrid cluster orchestration.
•  Design scalable, secure, and resilient Kubernetes-based infrastructure for HPC and AI compute environments.
• Develop architectural blueprints for cluster lifecycle management, service discovery, and workload scheduling across on-premise and hybrid infrastructures.
• Evaluate emerging CNCF ecosystem technologies (e.g., operators, CRDs, service meshes, observability stacks) and guide adoption strategies.
• Provide technical leadership in Kubernetes administration, troubleshooting, and performance optimization.
• Define best practices for all aspects of Kubernetes cluster configuration, scaling, and upgrade strategies.

Systems Integration & Enablement

• Collaborate with software engineering teams to integrate Kubernetes APIs and services into ICE ClusterWare’s management and monitoring subsystems.
• Enable seamless integration of Kubernetes with existing cluster management workflows, job schedulers, and monitoring frameworks.
• Administer and maintain Kubernetes clusters, including cluster creation, upgrades, node management, and scaling.
• Drive consistency in configuration, security, and policy enforcement across multi-cluster deployments.
• Implement observability and reliability frameworks for monitoring, logging, and alerting using leveraging Kubernetes-native tools such as Prometheus, Grafana, and OpenTelemetry.
• Manage and optimize cluster networking, including CNI plugin configuration (e.g., Calico, Cilium), ingress controllers, and service meshes.
• Configure and maintain persistent storage solutions in Kubernetes using dynamic provisioning, CSI drivers, and storage classes.
• Manage authentication, authorization, and access control through RBAC, service accounts, and integration with external identity providers.

Cross-functional Collaboration

• Serve as the internal Kubernetes subject matter expert and mentor for engineering peers.
• Partner with automation teams to ensure system reliability through automation and Infrastructure-as-Code methodologies.
• Partner with software engineers to guide Kubernetes-aware feature design and API development.
• Work alongside Product Architects and Product Managers to align architectural decisions with product roadmap and customer use cases.
• Manage authentication, authorization, and access control through RBAC, service accounts, and integration with external identity providers.

Qualifications

• Bachelor’s degree in Computer Science, Software Engineering, Systems Engineering, or a related technical field—or equivalent experience.
• Minimum 7–10 years of experience in software or systems engineering, with at least 4 years of hands-on Kubernetes cluster administration and architecture experience.
• Deep understanding of Kubernetes control plane, networking, security, and storage subsystems.
• Proven experience designing and operating multi-node, multi-cluster Kubernetes environments in production.
• Strong familiarity with Linux-based environments and cluster management systems.
• Understanding of microservices architectures, container runtime interfaces, and cloud-native design principles.
• Experience with Infrastructure as Code (e.g., Terraform, Ansible, or equivalent) and automation frameworks.
• Ability to translate system-level requirements into practical, scalable Kubernetes solutions.
• Proficiency in at least one scripting or programming language (e.g., Python, Go, Bash, etc.).
• Excellent communication skills, capable of conveying complex infrastructure concepts to software development teams.
• Self-motivated and capable of working independently while maintaining strong team collaboration.

Preferred Qualifications

• Understanding of microservices architectures, container runtime interfaces, and cloud-native design principles.
• Experience with HPC and AI cluster workloads in Kubernetes environments.
• Knowledge of GPU scheduling, device plugins, and high-performance networking within Kubernetes.
• Familiarity with Helm and other deployment automation tools.
• Experience with various Kubernetes distributions and vendor platforms (e.g., Red Hat OpenShift, Rancher RKE2, Canonical MicroK8s, VMware Tanzu, or similar enterprise-managed Kubernetes solutions)
• Kubernetes certifications (CKA, CKAD, or CKS) highly valued.

Location

• This is a remote role in the United States.

Travel

There is no expected travel with this role.

Compensation & Benefits

The base pay range that the Company reasonably expects to pay for this remote position in the United States is $136,000 - $165,000. The pay ultimately offered may vary based on business considerations, including job-related knowledge, skills, experience, and education. The position is bonus-eligible, and there are medical, dental, and vision benefits available. There is a 401k saving plan and other benefits, such as Paid Time Off, Life Insurance, and an Employee Assistance Plan.   

Diversity and Inclusion Statement

We are committed to creating an inclusive environment that embraces differences and fosters belonging for all.

Equal Opportunity Statement                                                                                  

We are an Affirmative Action/Equal Opportunity Employer and strongly committed to all policies which will afford equal opportunity employment to all qualified persons without regard to age, national origin, race, ethnicity, creed, gender, disability, veteran status, or any other characteristic protected by law.