Snowpark enables data engineering and AI/ML workloads in Snowflake by executing user-defined functions in secure sandboxes. Many of these workloads require external connectivity to access cloud APIs, external databases, or feature stores, creating a dependability challenge: how to provide transparent network access while preserving strict multi-tenant isolation and resource fairness. This paper presents Secure Network Access in Snowpark (SNAS), a production architecture for secure external communication from sandboxed workloads. SNAS combines Extended Berkeley Packet Filter (eBPF) packet filtering, Generic Network Virtualization Encapsulation (GENEVE) overlay networks, and distributed egress proxies for policy-driven egress control with low overhead. We describe the design, deployment, and measured production behavior of SNAS, including an eBPF-based bandwidth limiter using the Earliest Departure Time (EDT) algorithm, dual-tier policy enforcement, and safeguards for connection limiting and port exhaustion. SNAS is deployed across all Snowflake regions and supports large-scale production workloads including petabyte-scale data transfer and latency-sensitive external integrations.
翻译:Snowpark通过在安全沙箱中执行用户自定义函数,支持Snowflake中的数据工程与AI/ML工作负载。这些工作负载大多需要外部连接以访问云API、外部数据库或特征存储,由此带来一个可靠性挑战:如何在保持严格的多租户隔离与资源公平性的同时,提供透明的网络访问。本文提出Snowpark中的安全网络访问(SNAS)——一种用于沙箱化工作负载安全外部通信的生产级架构。SNAS结合了扩展的伯克利数据包过滤器(eBPF)包过滤、通用网络虚拟化封装(GENEVE)覆盖网络以及分布式出口代理,实现低开销的策略驱动型出口管控。我们描述了SNAS的设计、部署及实测生产行为,包括基于最早出发时间(EDT)算法的eBPF带宽限制器、双层策略执行机制,以及连接限制与端口枯竭防护措施。SNAS已部署于所有Snowflake区域,支持包含PB级数据传输与延迟敏感型外部集成在内的大规模生产工作负载。