T-Mobile Cuts IPv4 on 10 Million Handsets as IPv6 Hits 94%

IPv6占比达94%，T-Mobile在千万台手机上停用IPv4 该运营商利用DNS64和464XLAT向纯IPv6传输过渡，同时将协议回退逻辑直接推向客户端应用。 Stephan Lagerholm概述了T-Mobile在其移动终端设备中实现超过94%的IPv6普及率的历程，具体做法是通过在蜂窝接口上禁用IPv4，并利用DNS64、NAT64和464XLAT等技术。他解释了客户端转换与苹果应用商店（App Store）的相关强制要求如何促成了Android和iOS设备的大规模迁移，同时仅留下了少量的老旧设备、网络共享和漫游流量这一长尾。Lagerholm还展示了Happy Eyeballs机制下浏览器故障转移行为的试验台测试结果，警告运营商不要为受损的第三方域名提供运营商端DNS变通方案，并探讨了协议决策向上层应用层转移的架构趋势。 2017年春季，T-Mobile在约1000万台iPhone上关闭了IPv4，仅分配IPv6蜂窝接口，且未对最终用户造成明显影响。 (3:22) 截至2018年年中，T-Mobile自治系统发出的出站流量中约有94%优先使用IPv6，其余6%则分布在企业Wi-Fi、老旧手机、网络共享和漫游中。 (5:20) T-Mobile结合使用DNS64、NAT64、464XLAT（RFC 6877）以及RFC 7050前缀发现机制，以保持老旧IPv4目标地址和使用硬编码IPv4字面量应用的可达性。 (15:38) 传输和协议的决策权正积极从网络运营商向更高级别的应用程序API、浏览器以及通过DNS over HTTPS（DoH）的中央云解析器转移。 (20:55)

T-Mobile transitioned its cellular subscriber fleet to an IPv6-only transport by leveraging mobile operating system capabilities, DNS64/NAT64 network translation, and client-side translation frameworks. According to architecture lead Stephan Lagerholm, while over 94% of carrier traffic operates natively on IPv6, complete IPv4 elimination remains impeded by roaming constraints, tethering variations, legacy clients, and application-level fallback behaviors. Mobile Fleet Migration and Flag-Day Deployments (3:22) Mobile operators benefit from a relatively uniform client base compared to wireline networks. T-Mobile began enabling IPv6 after Google introduced native IPv6 and 464XLAT support in Android 4.3 in 2013. Apple provided crucial industry momentum by mandating that all App Store submissions support IPv6-only environments effective June 1, 2016. In spring 2017, T-Mobile deactivated IPv4 on cellular interfaces for approximately 10 million iPhones, moving them to an IPv6-only transport without user disruption. Lagerholm explained that major rollout phases functioned as scheduled cutovers tied to supported operating system versions. Outbound IPv6 traffic recorded by World IPv6 Launch climbed from 16% in early 2014 to roughly 94% by mid-2018. Because leading content providers like Google, Facebook, and Oath/Yahoo deliver content natively over IPv6, and about 96% of T-Mobile handsets run IPv6-only cellular configurations, the vast majority of subscriber traffic bypasses IPv4 transport entirely. Translation Mechanics: DNS64, NAT64, and 464XLAT (15:38) To connect IPv6-only handsets to IPv4 endpoints, T-Mobile deployed paired DNS64 and NAT64 systems. When an IPv6 handset queries DNS64 for a domain lacking authoritative AAAA records, the resolver synthesizes an IPv6 address using the Well-Known Prefix 64:ff9b::/96. Traffic directed to this prefix routes through redundant NAT64 gateways, which strip the synthetic prefix and translate packets to the IPv4 internet. For applications relying on hardcoded IPv4 address literals, devices employ 464XLAT (RFC 6877). Android executes a local client translator daemon (CLAT) that intercepts IPv4 literals, prepends the NAT64 prefix, and transmits standard IPv6 packets to the carrier gateway (PLAT). Devices dynamically discover the network's NAT64 prefix using RFC 7050 queries for ipv4only.arpa. On iOS, Apple handles translation at the application framework level via APIs like NSURLSession rather than running a system daemon, embedding translation logic into higher software layers. The Residual Long Tail of IPv4 Traffic (5:20) The remaining 6% of traffic touching IPv4 stems from specific architectural edge cases. Roaming is a primary hurdle: T-Mobile's global APN roaming configuration defaults to IPv4 because turning on IPv6 requires uniform readiness across global partner carriers. Domestic roaming in areas without native coverage also tunnels traffic over IPv4. Together, Lagerholm estimated roaming accounts for 1% to 2% of residual IPv4 traffic. Hotspot tethering accounts for another estimated 1% to 2%, as tethered iPhones expose only IPv4 to connected clients, whereas devices like the LG V30 offer dual-stack tethering. Other factors include unmigrated corporate campus environments, guest Wi-Fi networks requiring dual-stack fallback for raw IPv4 reachability, and legacy devices running versions prior to Android 4.3 that cannot run CLAT. Application Fallbacks and Operational Realities (12:44) Protocol decision-making has migrated from network operators to the application layer. Active SLA monitoring and Happy Eyeballs algorithms enable applications to autonomously trigger IPv4 fallbacks. In controlled failure testing with iptables, Chrome and Safari quickly shifted to IPv4 and reverted to IPv6 once restored. Internet Explorer and Edge failed over to IPv4 but never returned to IPv6, while Firefox on Android failed to switch mid-session. Furthermore, technologies like DNS over HTTPS (DoH) allow applications to bypass carrier DNS infrastructure entirely. Operationally, T-Mobile scanned mission-critical domains prior to disabling IPv4 on iPhones, finding defective DNS setups at major banks and universities. Lagerholm strongly warned against implementing local DNS overrides or spoofing records to mask external errors, citing security and man-in-the-middle liabilities. Instead, carriers should cite IETF RFC specifications and Apple App Store mandates to compel third-party providers to resolve domain breakages. While T-Mobile demonstrated that an IPv6-only mobile core is commercially viable at scale, full protocol retirement remains unfinished: internal carrier management networks (NMnet) still battle RFC 1918 address exhaustion. Moreover, as applications and centralized cloud resolvers increasingly govern transport selection, network operators face reduced control over traffic paths and failover behaviors.

T-Mobile架构主管Stephan Lagerholm在NANOG会议上阐述了该运营商向纯IPv6演进的历程，指出依托移动端生态的标准化以及DNS64、NAT64和464XLAT等关键架构，T-Mobile已使超过94%的移动流量原生运行在IPv6上，同时协议决策权正由传统运营商网络逐步推向操作系统与应用程序层。 终端生态驱动的单栈迁移与切换 (3:22) 相较于必须应对温控器等各类遗留终端的有线宽带网络，移动运营商面对的客户端生态更为集中，绝大部分用户归属于Android和Apple设备。Google自Android 4.3版本起原生支持IPv6与464XLAT架构，为T-Mobile在安卓终端全面开启IPv6提供了契机；Apple则在WWDC 2015上宣布自2016年6月1日起，所有提交至App Store的应用必须支持纯IPv6环境。得益于这两大操作系统生态的成熟，T-Mobile在2017年春季对网内约1000万台iPhone直接关闭了蜂窝接口的IPv4地址分配，实现了对终端用户透明的纯IPv6单栈运行。 在演进节奏上，Lagerholm说明这一转变采用了分批次的“标志日”（Flag Day）切换模式：在确定特定系统版本成熟后直接开启，并依赖用户换机周期自然提升渗透率。根据World IPv6 Launch组织的数据，T-Mobile自治系统中偏好IPv6的外发流量占比从2014年初APRICOT 37会议上公布的16%，迅速攀升至2018年中的约94%，与Verizon Wireless以及印度Reliance Jio共同成为推动全球超大规模IPv6商用的主要运营商。 DNS64与464XLAT协同翻译架构 (15:38) 为了在纯IPv6蜂窝网络中兼容传统IPv4资源，T-Mobile部署了DNS64与NAT64协同体系。纯IPv6手机在蜂窝网络下仅发起DNS AAAA查询；若权威DNS存在AAAA记录，流量将直接走最优路径进入IPv6互联网。当目标服务仅有IPv4的A记录时，DNS64服务器会将知名公共前缀64:ff9b::/96拼装在IPv4地址前端，合成虚假AAAA记录返回给终端，诱导手机将IPv6报文发往运营商的冗余NAT64网关；网关剥离前缀后将流量转发至IPv4互联网。Lagerholm强调，NAT64网关采购与运维成本高昂，主流内容提供商如Google和Facebook全面拥抱原生IPv6大幅削减了网关负荷，但滞后的小型CDN和云主机仍迫使网络维持翻译能力。 针对硬编码IPv4字面量地址的应用程序，网络引入了RFC 6877定义的464XLAT技术。在Android平台，系统内置clat守护进程，通过RFC 7050向ipv4only.arpa查询动态学习NAT64前缀，在本地将IPv4报文转换为IPv6报文送出。Apple则未采用系统级守护进程，而是要求开发者调用NSURLSession等高级API，在应用层透明执行RFC 7050前缀发现与IPv6地址合成，将原本由运营商承担的双栈路由决策推升至客户端框架内。 协议决策权上移与客户端回落缺陷 (12:44) 随着网络技术演进，流量协议的选择权经历了从运营商网络策略、到操作系统驱动（如464XLAT clat），再到应用程序自决的深刻转移。当前移动应用开始主动监控服务等级协议（SLA），并在探测到性能波动时自主触发Happy Eyeballs机制降级回退至IPv4。Lagerholm指出，随着DNS over HTTPS（DoH）的发展，浏览器正逐步绕过本地运营商解析器而直连云端集中式解析服务，这种协议决策权完全脱离网络控制的趋势令运营商感到担忧。 在针对Happy Eyeballs回落与恢复机制的测试中，不同客户端表现出显著差异。测试人员通过持续刷新图片并利用iptables人为阻断IPv6链路发现：Google Chrome与Apple Safari表现优秀，不仅在IPv6失效时迅速切换至IPv4，更在链路恢复后迅速重新切回IPv6；Microsoft Edge与Internet Explorer能够回落至IPv4，但链路修复后无法自动重返IPv6；Android端Mozilla Firefox则出现严重缺陷，在连接中断后无法降级至IPv4，且仅在初次建连时做单次判断。这种实现缺陷导致移动用户在进出隧道等弱网恢复场景下，应用容易被长期锁死在IPv4状态。 剩余6%流量长尾成因与排障准则 (5:20) 对于仍触碰IPv4的最后6%长尾流量，Lagerholm拆解出五大核心成因：未完成纯IPv6改造的企业与保留双栈回退的访客Wi-Fi；低于Android 4.3的极少数旧设备；偶发性弱网导致的算法锁定；移动热点共享（iPhone作为热点向接入电脑仅提供纯IPv4网络，占1%–2%）；以及漫游限制（占1%–2%）。由于APN漫游配置属于全局属性，在确认全球漫游伙伴均具备IPv6能力前只能维持IPv4，这也导致用户进入无自建网覆盖的美国境内怀俄明州或跨国漫游时，流量必须通过IPv4隧道回传。 在运营保障方面，Lagerholm将IPv6改造比作401(k)养老金投资，主张在内部网络管理系统（NMnet）面临RFC 1918私网地址耗尽时，结合硬件生命周期更替稳步推进升级。对于公网解析异常，他严正告诫运营商切勿在内部DNS实施伪造劫持，否则外部机构变更IP时极易导致运营商沦为意外的中间人攻击者；应依据RFC规范与Apple App Store的硬性准则要求第三方修复错误配置，并借助Twitter和Reddit等社交媒体监控实际用户的应用故障反馈。他提醒同行不必苛求全网100%完美，果断移除IPv4才是推动网络演进的务实做法。 Lagerholm总结认为，T-Mobile的实践证明了超大规模移动网络直接停用IPv4的可行性，但全球漫游配置的全局绑定、热点共享行为差异以及应用层回落算法的碎片化，依然是彻底消除IPv4依赖所面临的技术瓶颈。

[0:10] Thanks everybody. Can you have everybody [0:12] hear me okay? Yeah, it sounds good. [0:14] Well, it's great to be here in Denver, [0:15] sunny day and we're going to talk about [0:17] IPv6 here indoors, so that feels great. [0:20] Some background on me, I'm going to talk [0:22] about how our path to going IPv6 only at [0:26] T-Mobile. [0:27] Uh I was born and raised in Sweden. I [0:29] moved to the US in 2007. [0:31] In 2009, I I remember attending an ARIN [0:34] conference in San Antonio. [0:36] Uh I lived in Dallas [0:38] at that time and and I met with Stan [0:40] Barber, who some of you guys might be [0:41] familiar with and we were talking about [0:43] the deployment of IPv4 versus IPv6 in [0:46] the Europe and versus US and kind of [0:49] came to the conclusion that, you know, [0:50] it's it's we're we're a little behind [0:52] here in the US and maybe we should do [0:54] something. So, we founded the Texas IPv6 [0:56] Task Force uh back in 2009. We were [0:59] inspired by the Rocky Mountain Task [1:01] Force here in Denver, who by the way, [1:03] they're a sponsor and there's some [1:05] events going on tonight if anybody's up [1:07] for a free beer. Uh anybody who's doing [1:09] IPv6. [1:11] Uh [1:11] nevertheless, uh I started work for a [1:13] company called Secure64, also based here [1:15] in Denver, a DNS company and one day [1:18] this sales guy comes, you know, comes [1:20] into my office and says, "Hey, you got [1:22] to talk to this guy Cameron at T-Mobile. [1:24] He's talking about DNS64, NAT64, IPv6. I [1:27] have no idea what he's talking about." [1:30] So, I know you're into um IPv6, so why [1:32] don't you give him a call? So, I did and [1:34] um [1:35] um Cameron, uh who's still with [1:37] T-Mobile, who who's a good friend and [1:39] and somebody I work very closely with, [1:41] he convinced me that, you know, DNS64, [1:44] NAT64 was the right path for a mobile [1:47] service provider to take to transition [1:49] to IPv6. [1:50] And uh we actually Secure64 at that [1:52] point became partners of T-Mobile and we [1:55] had the first commercial DNS64 platform. [1:58] Anyway, fast forward a couple of years, [2:00] I worked for Microsoft doing DNS for [2:02] them for a while and then about 2 years [2:04] ago this opportunity showed up to lead [2:06] the architectural group within T-Mobile. [2:09] So, I jumped on that and, you know, end [2:11] of 2016 when I started, one of the first [2:14] thing Cameron said was [2:15] "By the way, we're turning off IPv4 on [2:18] our iPhones here [2:19] shortly. So, that might be something you [2:21] want to look into and dig into." So, [2:23] it's kind of cool to have that full [2:24] circle, first working [2:26] as a vendor and now kind of have have [2:28] the first-hand view of what we're trying [2:30] to accomplish and and why we're moving [2:32] to IPv6. [2:33] If you want to reach me, you know, [2:35] here's my T-Mobile email address and [2:36] Twitter, LinkedIn and so forth. [2:39] Or feel free to find me in the corridors [2:41] here lately [2:43] after after this talk. [2:45] So, I'm going to talk about our journey [2:47] to become IPv4 independent. [2:49] Uh I'm going to have a little bit of an [2:52] talking about retries, returns and happy [2:55] eyeballs and how that is affecting our [2:57] network. I've been doing a little bit of [2:58] research there lately. [3:00] And then give you guys a background for [3:01] those who who doesn't know NAT64, DNS64 [3:04] and some of the technologies we are [3:06] using like prefix discovery and 464XLAT. [3:10] And then talk about application level [3:12] fallback to IPv4 and what that is and [3:14] then some best practices and some [3:16] learnings that I've had from [3:18] removing IPv4 from our network. [3:22] So, our customer base is pretty easy to [3:24] deal with compared to perhaps, you know, [3:27] a wired service provider where there's [3:29] nest thermostats and all kinds of stuff. [3:32] Uh the bulk of our customer base is [3:35] either Android handsets or Apple [3:37] devices. And Android from starting from [3:40] 4.3 and onwards have had support for [3:43] IPv6 plus 464XLAT. [3:46] And that's when we decided to turn on um [3:49] IPv6 on these Android handsets. [3:52] And for Apple in 2015 at their developer [3:55] conference, they announced that uh [3:58] starting June 1st, 2016, any application [4:01] within the application store needs to be [4:03] not only, you know, [4:05] working on a V6 like a dual stack [4:07] environment, but actually must support [4:10] IPv6 only networking. So, it's been [4:12] tremendously helpful for us to um [4:16] to have those two giants behind us also [4:19] helping us pushing the transition to [4:21] IPv6. [4:23] And just a quick survey, how many of you [4:24] guys are T-Mobile customers in here? [4:26] Raise a hand. And how many of you guys [4:28] have a iPhone? [4:30] So, I don't know if you guys are aware, [4:31] but but you're actually having a V6 only [4:34] experience today, right? So, we have [4:36] about, you know, give or take 10 million [4:38] iPhones in our network and there there's [4:41] no IPv4 interface. If you look at the [4:43] interface, you turn off Wi-Fi and you [4:45] look at your interfaces, [4:46] there is no IPv4 interface. It's there, [4:48] only IPv6. So, that's a pretty cool [4:51] accomplishment in my opinion and I mean, [4:53] this is stuff that's ready. This is [4:55] nothing we're going to turn on a year [4:57] from now, 2 years from now. This is now, [4:59] right now. We've had it on for a year [5:00] and a half. It works, right? And we've [5:03] really reached a utopia that I think we [5:05] all talked about. We have been able to [5:07] move clients to a a new [5:09] IP protocol without even them probably [5:12] knowing that they moved to an IPv6 only [5:14] experience and we turn off IPv4 [5:18] on them. [5:20] So, it's been a long journey starting [5:21] with Android 4.3 somewhere around 2013. [5:25] We've had talks similar to this talk [5:27] several times uh starting um in 20 [5:30] beginning of 2014, Apricot 37, where we [5:33] could announce based on the World IPv6 [5:36] Launch that we had 16% [5:39] of uh our customer base were preferring [5:42] IPv6 over IPv4 outbound from our AS. [5:47] And then from from there, we've we've [5:48] had uh several talks as you guys can see [5:51] and we've made steady progress. [5:53] It's tapering off a little bit here uh [5:55] at the end. [5:57] Uh as we're getting close to uh 100%, [6:00] obviously, it's a little harder and [6:01] there's more corner cases that we need [6:03] to take into account, but as you can [6:05] see, it's a good steady progress. And [6:07] today, based on the World IP [6:09] World IPv6 Launch at org website, uh [6:13] today uh some 94% of our traffic is uh [6:17] or 94% of our handsets, I should say, or [6:21] you know, they're preferring IPv6 over [6:23] IPv4 going out from our AS. So, that's a [6:26] pretty remarkable accomplishment. And [6:29] can you guys turn on the clock here so I [6:30] know how much uh time I have left? That [6:32] would be great. [6:34] Uh so, that's a pretty remarkable [6:35] accomplishment. At this point, you you [6:37] might want to look into like what's that [6:40] remaining 6%? You know, what kind of [6:42] corner cases are we like how's the [6:44] bottom of the barrel looking here in [6:45] terms of getting to 100% and, you know, [6:48] I think we're kind of the first [6:50] uh service provider really pushing, you [6:52] know, closer, you know, over 90% here [6:54] that, you know, we can we can look at [6:56] that and and and see what's going on. [6:59] If you look at the World IPv6 Launch, [7:01] there's, you know, a big bunch of [7:02] companies in there. They're doing a lot [7:04] of good stuff on IPv6. I think everybody [7:06] is transitioning one way or another. I [7:08] think on the wireless side of things is [7:11] us. Um Verizon Wireless is doing a great [7:13] job. They have a good adoption rate as [7:15] well. And then Reliance in India, who's [7:18] a new entrance to market and they could [7:19] kind of um go directly into IPv6. [7:24] So, looking at the remaining 6% and [7:26] zooming in, um I think it's five [7:29] categories [7:30] that um you know, is is mostly [7:32] contributing to to the remaining 6%. We [7:36] do have our enterprise clients. We have [7:38] not migrated our enterprise clients to [7:40] IPv6 or dual stack yes yet. [7:44] Uh the the guests are dual stack. So, if [7:47] you ever visit if you're ever going to [7:49] visit a T-Mobile campus, then if you [7:52] connect to the guest, you're actually [7:53] using NAT64 and DNS64 and those [7:56] technologies that we're using for our [7:58] handsets and you're using those for um [8:01] for your your experience there. But we [8:03] do we do have the guests are dual [8:04] stacked. So, if there's an IPv4 literal [8:07] or something like that coming up and you [8:09] have that emergency IPv4 connectivity as [8:12] well. [8:12] Uh more on that later on how DNS64 and [8:15] NAT64 works. [8:17] There's also some really old handsets. [8:19] So, Android handsets, for example, [8:21] before 4.3, they are not capable or [8:24] they're capable dual stack, but they're [8:25] not turned on in our network. [8:27] Um so, if you look at this link from [8:30] developer.android.com, [8:32] you can see it's about the world wide [8:35] population of these types of handsets [8:38] and tablets that are this old is about 4 [8:40] 4%. So, that's measured who is actually [8:44] going to the [8:45] uh to app app store to to download any [8:48] new applications. So, so about 4%. I [8:51] think that's since that's a worldwide [8:52] number, I think the number in our [8:53] network might be a little lower. [8:55] Typically, your primary device is your [8:57] phone and you might want to upgrade that [8:59] a little bit more often. You might have [9:01] an older tablet or something in your [9:02] home that might not be upgraded as often [9:05] and you're using a little longer. So, I [9:07] think in our network, it might be less [9:09] than 4%. Um and also, if you're a power [9:12] user, you probably upgraded your phone [9:15] at some point. So, most of the [9:16] connections we're seeing outbound, you [9:18] know, are coming from power users that [9:20] are using browsing the internet lot. So, [9:23] um they probably have a newer uh [9:25] handset. [9:27] So, maybe a a couple of percent or [9:28] something could still constitute, you [9:30] know, old phones that are not being [9:32] upgraded. And this is hard for us to do [9:34] anything. We just have to wait them out [9:35] and at some point they'll, you know, buy [9:37] a new phone and that old grandma, she'll [9:39] pass away and, you know, [9:42] at some point we'll we'll get to to no [9:44] old handsets. Uh but but they're still, [9:46] you know, I'm surprised because we're [9:47] talking about like Samsung Galaxy S2 um [9:50] as a good example of a phone that's, you [9:52] know, not supporting Android 4.3. So, [9:55] it's pretty old stuff. [9:58] Another thing that could contribute a [9:59] little bit to this is retries over IPv4 [10:02] for one or another reason. So, think [10:04] about we're having LTE network, you're [10:06] an Uber driver or something, you're [10:08] using Google Maps or [10:10] some other application, you're driving [10:11] to a tunnel, you might get some [10:13] connectivity issues. As you drive out of [10:16] the tunnel, Happy Eyeballs kicks in and [10:18] you get a V4 [10:19] connectivity. And then the question [10:21] becomes, do you stay now on V4 or do you [10:23] move back to IPv6? [10:25] There could be a little bit of this, you [10:26] know, I'm I'm just starting some [10:28] research on this and I'll cover that in [10:30] in the next slide, but it's not [10:32] significant. [10:33] Tethering is another case. So, if you [10:37] have a phone, you turn it on as a mobile [10:39] hotspot, some of our phones is going to [10:42] expose just V4 to your laptop, like an [10:45] iPhone, it'll be V4 only experience. [10:49] Whereas others, like I have an LG V30, a [10:51] fairly new phone, and that's actually [10:52] dual stack. So, if I do tethering with [10:55] that, I get a dual stack experience. So, [10:57] there's some corner cases there. [11:01] Maybe 1 to 2% I would estimate in that [11:03] category as well. [11:05] And then the last one is roaming. So, [11:07] our APN roaming protocol is set to IPv4 [11:10] only. And this is a global setting that [11:12] we set. So, that means that whenever [11:14] you're roaming to another network, you [11:16] will be IPv4 back to us and then out to [11:19] the internet from there. [11:21] So, since it's a global setting, we need [11:23] to make sure that all the service [11:25] providers in the world can handle before [11:27] we turn it on. So, it's a little [11:28] unfortunate that this is a global [11:29] setting for us. [11:31] And this is not only for international [11:33] roaming, you know, Canada, Mexico, [11:35] Europe and so forth, but we also have [11:37] roaming partners in areas where we don't [11:39] have [11:40] a lot of connectivity. So, like if you [11:42] drive up north there a couple of hours [11:45] into Wyoming, you're probably going to [11:46] find some spots where we don't have [11:48] coverage, but we're partnering with some [11:50] local [11:51] provider and they have coverage. And in [11:53] that case, you'll also be tunneled back [11:55] to us via IPv4. So, this is probably a 1 [11:58] to 2% as well. And as we build out our [12:01] coverage and we become less [12:03] dependent on those national roaming [12:06] partners, [12:07] this will this will actually burn down [12:09] this category. But the international [12:11] roaming at some point we'll have to make [12:12] a decision if we want to turn that on [12:14] and go V6. [12:16] So, so don't go to Mexico or Canada or [12:19] Europe for vacation this summer. If you [12:20] can stay in the US where you have [12:22] T-Mobile coverage, or maybe Puerto Rico [12:25] is a good alternative for you then, you [12:26] know, that would be good because then we [12:28] keep the statistics high. [12:31] Sweden is, you know, unfortunately, [12:33] that'll be international roaming, so [12:34] you're screwed, but you know, you can [12:36] watch the game on on Wednesday. We're [12:38] playing Mexico 5:00 a.m. in the morning [12:40] or 6:00 a.m. I guess here. [12:43] Anyway, [12:44] back to IPv6. So, there's um [12:48] Happy Eyeballs has been out for a while. [12:49] The whole idea is that if IPv6 [12:52] connectivity is bad for one or another [12:53] reason, [12:54] you should move over to IPv4. [12:57] So, there's a great site [12:59] that you can check your Happy Eyeballs, [13:01] the he.test-ipv6.com, [13:04] and it's like a JavaScript that every 5 [13:06] seconds [13:07] retries and you can you can see whether [13:09] you're V4 or V6. [13:11] And I've been driving around a lot [13:13] seeing, you know, how can I induce [13:15] failures here? So, so [13:18] so and see can I do I stay on IPv4 in [13:21] that case or am I going to go back to [13:23] IPv6 after after a while? [13:26] It's been tricky to in in real life [13:28] really induce these types of failures. [13:31] So, I don't know how common they are. I [13:32] need to look into this a little bit [13:33] further. [13:34] But I also set up a test bed that I just [13:36] had a kind of a JavaScript just [13:38] reloading an image every 5 seconds. And [13:41] then with IP tables just shut down IPv6 [13:43] and see how different browsers reacted. [13:46] And Safari as well as Google Chrome were [13:49] doing a good job, you know, quickly jump [13:51] over to IPv4 doing the Happy Eyeball [13:54] thing. And then as I remove that failure [13:56] and remove that IP tables rule with [13:59] IPv6, they quickly determine that V6 is [14:01] good enough again and they move back to [14:03] IPv6. So, that's what I call the returns [14:06] that probably not covered in too great [14:08] length how you should react to IPv6 [14:11] becoming available again after you're on [14:12] V4 in the in the RFCs and the Happy [14:15] Eyeball RFCs and so forth, but [14:17] interesting that you know, some of the [14:20] major browsers are doing a good job [14:21] there. Others though, like if you if [14:24] you're working on an Edge or an Internet [14:27] Explorer, if you induce that failure, [14:29] it's going to fail over and do [14:31] IPv4, but then even if IPv6 becomes [14:35] available again, it's actually not going [14:36] to fall back into IPv6. So, that's an [14:39] interesting corner case that maybe [14:40] there's some work here to be done by the [14:42] browser vendors. [14:44] Mozilla Firefox on my Android, I [14:47] couldn't get to work even doing Happy [14:49] Eyeball and fall over. So, once it [14:50] started to reload this image, it was [14:52] over V6. I shut it down and it never [14:54] actually fall back to IPv4. And I think [14:57] that is just because it's doing the [14:58] Happy Eyeball in the beginning for that [15:00] URL or that [15:02] um [15:03] domain name and it's not doing Happy [15:05] Eyeballs again. It's just assuming that [15:06] that's going to be good for the entire [15:07] session. [15:09] Right? Once I had [15:11] and similarly, if I induced the failure, [15:14] but I I started the [15:17] the browser [15:19] on IPv4, Firefox never failed back to [15:22] IPv6 even if IPv6 was available at the [15:24] later stage. So, I think there's some [15:26] corner cases here that needs to be [15:27] looked into as we're trying to trend [15:29] down to 100%. There there's some corner [15:31] cases here in the browsers that probably [15:33] need some some work to to make this work [15:35] a little bit better. [15:38] So, a little bit of a background on how [15:40] DNS64 NAT64 works. So, you have the [15:44] handset and the handset is a V6 only [15:46] handset. So, it'll send out a quad A [15:48] query because there's no V4 interface, [15:50] so there's no point in sending out any A [15:52] queries. [15:53] And that will reach our DNS64 box. [15:56] If there's a quad A record available, [15:58] we'll send that back to the client and [16:00] the client will take the black path out [16:02] to the V6 internet. So, that's the ideal [16:05] way of going because there's no middle [16:07] machines here that needs to do anything [16:10] and it's it's very convenient for us. [16:12] Worst case scenario, there's no V6, but [16:15] there is a V4 address. And in that case, [16:18] we do a synthetic we append a prefix, [16:21] the DNS64 prefix is 64FF9B [16:24] in this case and we convert this from [16:26] hex to from octal to hex and we send [16:30] that back to the client to trick him [16:31] into that there is actually an IPv6 [16:33] address for this [16:35] website that you're trying to to reach. [16:37] And that entire prefix is routed to our [16:41] NAT64 gateway. So, that packet will now [16:44] end up in our NAT64 gateway. We can [16:46] strip off the the prefix and we can send [16:49] that out on the on the V4 internet. [16:53] So, we have a ton of these NAT64 [16:55] gateways in our network, obviously. [16:57] They're rather expensive. You want to [16:58] deploy them in pairs so there's a [17:00] failure. So, ideally, you know, if you [17:02] guys can move all your content to IPv6, [17:05] then we don't we can decommission them [17:07] eventually, right? So, so we're getting [17:08] closer and closer and you know, we'll [17:11] see we'll see how this goes, but please [17:13] move over to IPv6. And I got to say, you [17:15] know, the the top content providers like [17:18] the Google, Facebook of the world, [17:21] they've been doing a tremendous good job [17:23] of enabling IPv6. So, so it's some of [17:26] the smaller CDNs and and some other [17:30] cloud providers that we're seeing some [17:31] traffic going into being V4 only still. [17:36] So, we're also using 464XLAT. [17:38] And 464XLAT is also an RFC 6877. [17:43] Cameron actually Cameron Byrne, he's one [17:45] of the authors of that. And what it is [17:47] it's a kind of a small little [17:49] piece of code on your Android handset [17:52] that is exposing an an IPv4 address to [17:54] your Android handset. And worst case [17:57] scenario, if you encounter like an IPv4 [17:59] literal, like something embedded into a [18:01] website, then if you only had V6, you [18:04] couldn't you were you were dead in the [18:06] water. But this gives you this [18:08] opportunity to still communicate over V4 [18:11] by sending it there, the the clat [18:15] demon that's running on your 464XLAT [18:17] device is now going to do kind of like [18:19] we're doing with the NAT64 DNS64. It [18:22] knows the DNS64 prefix, it's going to [18:24] append it, it's going to create an IPv6 [18:27] packet so it can go IPv6 only over our [18:29] network to the NAT64 gateway. And from [18:32] there, we strip that header and send it [18:34] out. So, that's how it works on Android [18:36] phones. [18:39] There's a a similar RFC or a kind of [18:41] another RFC that we're also using, RFC [18:44] 7050, [18:45] discovery of IPv6 prefixes. And this is [18:48] instead of hardcoding these DNS64 NAT64 [18:52] prefixes in our network, instead we send [18:55] a query for ipv4only.arpa, [18:57] a quad A query. And from there, [19:00] we can determine you can strip off the [19:02] the last 64 bits and from there the [19:04] handset can determine [19:06] what the prefix is and you start using [19:08] that. So, we don't have to hardcode and [19:10] provision anything. [19:11] So, this is great and that's how it's [19:13] working in in Android. And Apple devices [19:16] actually doing something similar when it [19:18] comes to what I call application level [19:20] fallback. So, if you go to this URL at [19:23] at the bottom there to the developer [19:25] manual for for Apple, [19:27] they're actually not recommending you if [19:29] you create some application to use the [19:31] low-level sockets. [19:33] Instead, they're recommending the stuff [19:35] in the middle that's kind of in bluish [19:37] here. So, they want you to use what they [19:39] call NSURLSession, which is a [19:41] higher-level API, and they will take [19:44] care of all the translation on your [19:46] behalf and falling back to IPv4 if [19:48] possible. [19:49] So, the blue stuff here is actually [19:53] using RFC 7050 sending out these IPv6 [19:58] IPv4 only ARPA [20:00] queries to figure out the prefix. And if [20:02] you encounter an IPv4 literal, you can [20:05] actually fall back to IPv4. So, here you [20:07] have a V6 only device with a browser [20:10] that understands that it's a V6 only [20:12] device and it has this backdoor kind of [20:15] into V4 as needed. So, it's kind of [20:18] cool, you know, I talked about this [20:19] example where you induce a failure just [20:21] by adding an IP um [20:24] IP tables. [20:25] And it's pretty cool, you sit there with [20:27] a V6 only Apple device and then you [20:29] induce a failure on V6 and then you see [20:32] all of a sudden V4 traffic coming in [20:33] from that V6 only device because it's [20:35] going via that NAT64 device. [20:38] So, so pretty cool stuff. [20:42] The one thing I want to mention here [20:43] though is that uh [20:45] you know, we're moving the IPv4 fallback [20:47] mechanism further up in the stack here [20:50] as a you know, as an industry. It used [20:52] to be a decision that we made from a [20:53] network perspective, do we want to dual [20:55] stack this uh server or client or do we [20:58] want to keep in V4 or want to do V6 only [21:00] on him? And that's something that then [21:02] kind of moved into the OS layer with the [21:04] 464 xlat. And now the application is [21:07] taking making that decision on our [21:08] behalf like when when is the SLA on V6 [21:11] not good enough and when do I need to [21:13] fall back into V4? So, it's kind of a [21:16] you know, an interesting but a little [21:18] bit scary new path that we're seeing [21:20] here and where's this going to end? You [21:21] know, there's a lot of discussion on DNS [21:24] over HTTP. You know, who says that the [21:26] applications the browser can't just [21:28] forward all the DNS records up to their [21:30] resolver and make decisions up there. [21:32] And who says they can't just forward [21:34] everything up to some central cloud and [21:37] from there just route it out on the [21:38] internet, right? So, something to kind [21:40] of think about and keep track of. [21:45] So, [21:47] some of the learnings that I feel we [21:51] we did well and and some things that I [21:53] want to share is that you know, [21:56] obviously invest in IPv6 a little bit [21:58] it's like your 401K retirement plan, [22:01] right? You you need to kind of [22:03] constantly invest and and and move, you [22:06] know, we've done a big investment which [22:08] was a very successful investment early [22:10] on with with our handsets and that [22:13] panned out really well for us. We're in [22:15] pretty good shape now and you know, but [22:16] we still have ways to go with our [22:18] internal clients with data centers. We [22:20] we have what we call NM net which is a [22:22] network management network which where [22:24] we're running out of RFC 1918 space. [22:26] So, we have still ways to go and but now [22:29] we can kind of we're working on the [22:31] traditional upgrade cycles when [22:32] something goes end of life, we can work [22:35] into and and enable IPv6 at that point. [22:38] Determine what sites and applications [22:40] are important. You know, we did a lot of [22:41] scanning in advance before we turn on [22:44] and turned off IPv4 on our [22:47] uh Apple devices. And some sites are [22:49] just broken, right? So, so we did a lot [22:52] of outreach. There was a card a credit [22:55] card issuer here in the US who had a DNS [22:57] problem with their domain and we [23:00] couldn't reach it. The DNS64 kind of [23:02] derailed and we couldn't reach that [23:04] site. And we managed to reach out to [23:05] them and and they actually fixed it you [23:07] know, before we launched. So, that was [23:09] great. Uh same thing as a university I [23:11] reached out to that fixed their sites [23:13] and some other some other sites as well. [23:16] Um so, you know, this has been live for [23:18] a year and a half now. So, if if you're [23:19] a you know, wireless service provider [23:21] from US or elsewhere, you know, if it [23:23] works for us chances are it's pretty [23:25] well it's going to work for you too, [23:26] right? So, I don't think we have to be [23:27] too concerned about turning off IPv4 [23:30] anymore. It works very good at least in [23:32] a wireless environment with Apple and [23:34] Android handsets. [23:36] Um [23:37] don't try to fix things with local [23:39] overrides. You know, one thing that you [23:41] know, when we had that card issuer [23:44] having problems, I was thinking maybe we [23:46] pop something in our own DNS to just [23:48] circumvent this problem from from [23:50] happening. [23:51] I think in general that's probably a bad [23:52] idea because you know, especially when [23:54] we talk about you know, credit cards and [23:56] those kinds of things, what happens if [23:57] they move and somebody else takes their [23:59] IP address and you know, you don't want [24:01] to be the man in the middle and be [24:02] liable for that. It's better if the [24:04] actually where the problem is, if they [24:06] can fix the problem there. And it's you [24:08] know, if you have the RFCs you know, [24:10] DNS64 NAT64 RFCs and you can point to [24:13] the exact you know, your violation on [24:16] section 5.3 in this RFC, it's easy for [24:19] easy that's good ammunition for them to [24:21] you know, to fix on their side. And also [24:23] pointing to the Apple policy that all [24:26] the applications today needs to be [24:29] capable of an IPv6 only experience in [24:31] Apple store. So, that's a requirement [24:33] and you know, they obviously they didn't [24:36] you know, some of the applications [24:37] didn't live up to that that requirement. [24:39] So, they have to go fix it on their [24:41] side. [24:42] Uh scan social media. You know, some of [24:44] this stuff when you're a large company [24:45] like a T-Mobile, it might be hard to get [24:47] through to the people that actually you [24:49] know, are working on this. You know, [24:50] we're small team that working deep in [24:52] into this with DNS and so forth. Uh but [24:55] social media is a great tool to just see [24:57] if people are having problems, you know, [24:58] let's try them on our phones and see if [25:00] we can reproduce it. And we found some [25:02] issues that way that we could fix and [25:04] work together with application uh [25:05] providers to fix. [25:08] And then you know, don't try to change [25:09] the world. Maybe keep things in [25:11] perspective. There's a lot of broken [25:12] stuff out there already. People are [25:14] under attack and there's stuff that [25:15] doesn't work. They're not RFC compliant [25:18] and you know, don't let like you know, a [25:20] few sites that doesn't work hinder your [25:23] you know, deployment or and and getting [25:26] off of IPv4. So, kind of dive into it [25:29] would be my suggestion and you know, [25:30] work it out. And it's been useful for [25:32] us, you know, just diving into this and [25:34] and and at the end of the day, you know, [25:37] everything worked out fine. So, [25:40] that's my last slide. We have about 4 [25:41] minutes for questions here. So, is [25:43] anybody who wants to [25:51] Go ahead. [25:57] 1 2 3 [25:58] Why don't we wait with you? We go ahead [26:00] to the right first and then we'll come [26:01] back to you. [26:13] IPv4 IPv6 nothing is working. IPv5 [26:18] Yeah, speak loud. [26:36] So, the the traffic volume ratio is [26:37] actually um [26:39] is an interesting question. You know, we [26:41] have a lot of V it's it's actually [26:44] you know, more more I would say more [26:46] than 94% of our traffic is V6 at this [26:49] point, right? Because [26:50] the most of the main content if you [26:52] think about you know, Oath or you know, [26:55] Yahoo and Facebook and Google and the [26:57] big content provider, they're all [26:59] already there and 96% of our handsets [27:02] are already there. So, you get this kind [27:04] of matrix where the traffic is is is [27:07] really really large in terms of IPv6 [27:10] most of the cases. [27:12] Hello. Okay. Um great talk. Thank you. [27:15] question. Uh [27:17] was there any kind of gradual rollout of [27:20] this change to IPv6 or was it a big [27:22] switch in a day that you say, yep, let's [27:24] do it. Yeah. And you are just Edward [27:27] from Spotify. Okay. Um [27:30] you know, it's it's it's started with [27:31] Android really, right? So, 4.3 it was [27:34] kind of slow start there and and then [27:37] Apple [27:38] provided support at the later stage. [27:41] So, it was kind of a a [27:43] a big it was like a flag day, right? [27:45] We're turning this on and then any [27:46] anybody with Android 4.3 is going to [27:48] have it, right? So, so so it wasn't [27:50] gradually but then we've grown gradually [27:52] because people are changing handsets, [27:54] they're getting new handsets and and [27:56] those kinds of things. [27:57] So, [27:59] all righty, cool. Any other questions? [28:02] If not, I want to thanks everybody for [28:04] listening. It's been great and let me [28:06] know if you have any more questions. I'm [28:07] here uh throughout the day and tomorrow. [28:10] So, thank you very much.