Welcome!

FinTech Journal Authors: Liz McMillan, Elizabeth White, Yeshim Deniz, Pat Romanski, Mehdi Daoudi

Related Topics: SDN Journal, Java IoT, Containers Expo Blog, @CloudExpo, Cloud Security, @BigDataExpo

SDN Journal: Blog Post

Stateless Transport Tunneling (STT) Meets the Network

At a high level the concepts of larger packets, hardware offload, reduced CPU load and interrupts all make sense

Last week I walked through the packet formats for VXLAN and NVGRE specifically focused on ways by which the overlay packets provide information to the physical network that help the physical network. Some of the initial extreme thoughts that the overlay and physical network can and should be completely ignorant of each other have softened more recently and more pragmatic thoughts of collaborating layers are being articulated. At Plexxi we have often mentioned that we believe the physical network and the overlay need to be closely orchestrated to get the most benefit out of the total network solution. And orchestration != ECMP.

In addition to VXLAN and NVGRE, Stateless Transport Tunneling (STT) is an encapsulation mechanism used by VMware, mostly for communication between server based vSwitches. It is a bit more involved and complicated than VXLAN and NVGRE, mostly because it was designed to carry large data packets, up to 64 Kbytes. Physical networks have limitations on the size of a packet that can be transferred. Ethernet standard maximum transmission unit (MTU) used to be 1500 bytes, but most ethernet devices these days can support jumbo packets allowing packets of 4, 9 or even 16 Kbytes in size. Even at those sizes, large data transfers are somewhat hampered by the work involved in taking a large chunk of data and then chopping them up into smaller portions to be transmitted. In a response to this, hardware vendors have taken some of this functionality and added it to the Network Interface Cards (NICs) on servers and have them do most of this segmentation and re-assembly work based on how TCP takes large portions of data and chops them into smaller segments. Doing his in hardware means it can be done faster, but more importantly, it removes this burden from the server CPUs, allowing them to do other (more useful) work.

STT was designed to make use of these TCP capabilities in NICs. STT can take ethernet packets up to 64 Kbytes from a VM on a server, and tunnel it to its destination as a 64 Kbyte entity. This STT frame has to be chopped into smaller pieces to match the MTU of the physical network, but an STT packet looks just like a TCP segment to the receiving NICs, allowing them to reconstruct the original 64 Kbyte packet without needing the CPU.

When the sending tunnel endpoint receives a large chunk of data to be transmitted at another VM at the other side of a tunnel, the vSwitch takes several steps to encapsulate this packet. First, it adds an STT Frame Header to the packet.

STT Frame Format 1

The STT Header is 18 bytes in length and has a variety of administrative fields, but the key field is the Context ID. This is a 64 bit field and its intended use is similar to the VXLAN Network Identifier (VNI) or the NVGRE Virtual Subnet ID (VSID). While the semantics of this field are somewhat defined, its value and how to use it is left open in the latest specifications. Its main purpose is to provide the receiving tunnel endpoint the information it needs to determine where this packet needs to be sent after decapsulation.

After the STT Frame Header has been added, this new packet (original packet  + new STT header) is chopped into smaller pieces so that each piece is at least 62 bytes smaller than the MTU of the physical network. Each of these new segments receives 24 byte TCP like header, a normal 20 byte IP header, and of course the final 18 byte Ethernet header before transmission. The magic (or ugliness for those less enamored by STT) is in the TCP like header. These 24 bytes are formatted just like a normal TCP header to ensure the hardware in the NICs can re-assemble segments that belong together. The traditional Acknowledgement field in TCP is used as a fragment ID, essentially telling the NIC that all packets/segments that come in with the same fragment ID belong together and should be reassembled into the larger original ethernet frame. The traditional Sequence number is used as an offset indicator, to tell the NIC in what order the fragments need to be put together.

STT Frame Format 2

Similar to VXLAN and NVGRE described last week, STT has a mechanism to create entropy for the physical network to distinguish flows from each other and allow them to be balanced using ECMP (or link aggregation – LAG) based deployments. In STT, the TCP source port is used to create entropy. The originating tunnel end point will use some hash calculation on the original packets header information and use the result to populate the TCP source port. Switches in the physical network can now use the TCP port information from the tunneled packet in their hash calculation for ECMP or LAG packet distribution.

While STT is likely to be more efficient than either VXLAN or NVGRE for the transfer of large amount of information because it offloads the segmentation and re-assembly, it carries significantly more overhead than either VXLAN or NVGRE in additional header information for smaller packets. STT adds 80 bytes of new header to a VM originated ethernet packet for the first segment of this packet, 62 for each following segment. Compare that to a consistent 46 bytes for each NVGRE encapsulated packet, and 54 bytes for VXLAN. For traffic between VMs on the same server this may not matter, but it certainly does for traffic carried across the physical network. For the plentiful mice flows, we have likely doubled the size and bandwidth required for each.

A probably more significant drawback of STT comes from its strength. Designed for large packet transfers, once an original packet is encapsulated with STT header, chopped into parts, then encapsulated into individual ethernet, IP and TCP (like) headers, only the first packet provides any clue or context of the original source, destination, protocol, application and other content. The relevant pieces of that will only be found in the first segment, any follow up segments only provide enough information about the tunnel endpoints and no other original context without the first segment. And that makes debugging really hard. It also makes it hard to differentiate traffic on the physical network, even at a very high level Virtual Network identifier. And every existing network based service (realizing that one of the goals of overlay networks is to push this to the vSwitches themselves) will also have a hard time deciding what to do with these packets.

At a high level the concepts of larger packets, hardware offload, reduced CPU load and interrupts all make sense. But most data center ethernet networks can easily support 9k or even 16k packets, so perhaps the gap between 16k packet based transfer and 64k semi-stream based communication is really not that much considering that the bulk of packets are small to begin with (remember those mice and elephants?). Perhaps aligning the MTU of the virtual port with that of the network may be worthwhile to have the STT and original header in each and every packet on the wire. Regardless of whether that is a real wire, or a virtual one.

[Today's fun fact: One of the primary reasons the Mayflower pilgrims ended their voyage at Plymouth Rock was pretty much the same reason people today suspend their journeys: they ran out of beer. No need for a funny punch line on that one]

The post Stateless Transport Tunneling (STT) meets the Network appeared first on Plexxi.

Read the original blog entry...

More Stories By Marten Terpstra

Marten Terpstra is a Product Management Director at Plexxi Inc. Marten has extensive knowledge of the architecture, design, deployment and management of enterprise and carrier networks.

@ThingsExpo Stories
SYS-CON Events announced today that Massive Networks will exhibit at SYS-CON's 21st International Cloud Expo®, which will take place on Oct 31 – Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA. Massive Networks mission is simple. To help your business operate seamlessly with fast, reliable, and secure internet and network solutions. Improve your customer's experience with outstanding connections to your cloud.
SYS-CON Events announced today that Grape Up will exhibit at SYS-CON's 21st International Cloud Expo®, which will take place on Oct. 31 – Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA. Grape Up is a software company specializing in cloud native application development and professional services related to Cloud Foundry PaaS. With five expert teams that operate in various sectors of the market across the U.S. and Europe, Grape Up works with a variety of customers from emergi...
Detecting internal user threats in the Big Data eco-system is challenging and cumbersome. Many organizations monitor internal usage of the Big Data eco-system using a set of alerts. This is not a scalable process given the increase in the number of alerts with the accelerating growth in data volume and user base. Organizations are increasingly leveraging machine learning to monitor only those data elements that are sensitive and critical, autonomously establish monitoring policies, and to detect...
Because IoT devices are deployed in mission-critical environments more than ever before, it’s increasingly imperative they be truly smart. IoT sensors simply stockpiling data isn’t useful. IoT must be artificially and naturally intelligent in order to provide more value In his session at @ThingsExpo, John Crupi, Vice President and Engineering System Architect at Greenwave Systems, will discuss how IoT artificial intelligence (AI) can be carried out via edge analytics and machine learning techn...
Everything run by electricity will eventually be connected to the Internet. Get ahead of the Internet of Things revolution and join Akvelon expert and IoT industry leader, Sergey Grebnov, in his session at @ThingsExpo, for an educational dive into the world of managing your home, workplace and all the devices they contain with the power of machine-based AI and intelligent Bot services for a completely streamlined experience.
When shopping for a new data processing platform for IoT solutions, many development teams want to be able to test-drive options before making a choice. Yet when evaluating an IoT solution, it’s simply not feasible to do so at scale with physical devices. Building a sensor simulator is the next best choice; however, generating a realistic simulation at very high TPS with ease of configurability is a formidable challenge. When dealing with multiple application or transport protocols, you would be...
With tough new regulations coming to Europe on data privacy in May 2018, Calligo will explain why in reality the effect is global and transforms how you consider critical data. EU GDPR fundamentally rewrites the rules for cloud, Big Data and IoT. In his session at 21st Cloud Expo, Adam Ryan, Vice President and General Manager EMEA at Calligo, will examine the regulations and provide insight on how it affects technology, challenges the established rules and will usher in new levels of diligence a...
An increasing number of companies are creating products that combine data with analytical capabilities. Running interactive queries on Big Data requires complex architectures to store and query data effectively, typically involving data streams, an choosing efficient file format/database and multiple independent systems that are tied together through custom-engineered pipelines. In his session at @BigDataExpo at @ThingsExpo, Tomer Levi, a senior software engineer at Intel’s Advanced Analytics ...
In the enterprise today, connected IoT devices are everywhere – both inside and outside corporate environments. The need to identify, manage, control and secure a quickly growing web of connections and outside devices is making the already challenging task of security even more important, and onerous. In his session at @ThingsExpo, Rich Boyer, CISO and Chief Architect for Security at NTT i3, discussed new ways of thinking and the approaches needed to address the emerging challenges of security i...
SYS-CON Events announced today that Dasher Technologies will exhibit at SYS-CON's 21st International Cloud Expo®, which will take place on Oct 31 - Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA. Dasher Technologies, Inc. ® is a premier IT solution provider that delivers expert technical resources along with trusted account executives to architect and deliver complete IT solutions and services to help our clients execute their goals, plans and objectives. Since 1999, we'v...
There is only one world-class Cloud event on earth, and that is Cloud Expo – which returns to Silicon Valley for the 21st Cloud Expo at the Santa Clara Convention Center, October 31 - November 2, 2017. Every Global 2000 enterprise in the world is now integrating cloud computing in some form into its IT development and operations. Midsize and small businesses are also migrating to the cloud in increasing numbers. Companies are each developing their unique mix of cloud technologies and service...
SYS-CON Events announced today that IBM has been named “Diamond Sponsor” of SYS-CON's 21st Cloud Expo, which will take place on October 31 through November 2nd 2017 at the Santa Clara Convention Center in Santa Clara, California.
SYS-CON Events announced today that Datera, that offers a radically new data management architecture, has been named "Exhibitor" of SYS-CON's 21st International Cloud Expo ®, which will take place on Oct 31 - Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA. Datera is transforming the traditional datacenter model through modern cloud simplicity. The technology industry is at another major inflection point. The rise of mobile, the Internet of Things, data storage and Big...
SYS-CON Events announced today that Akvelon will exhibit at SYS-CON's 21st International Cloud Expo®, which will take place on Oct 31 – Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA. Akvelon is a business and technology consulting firm that specializes in applying cutting-edge technology to problems in fields as diverse as mobile technology, sports technology, finance, and healthcare.
WebRTC is great technology to build your own communication tools. It will be even more exciting experience it with advanced devices, such as a 360 Camera, 360 microphone, and a depth sensor camera. In his session at @ThingsExpo, Masashi Ganeko, a manager at INFOCOM Corporation, will introduce two experimental projects from his team and what they learned from them. "Shotoku Tamago" uses the robot audition software HARK to track speakers in 360 video of a remote party. "Virtual Teleport" uses a...
SYS-CON Events announced today that Datera will exhibit at SYS-CON's 21st International Cloud Expo®, which will take place on Oct 31 – Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA. Datera offers a radically new approach to data management, where innovative software makes data infrastructure invisible, elastic and able to perform at the highest level. It eliminates hardware lock-in and gives IT organizations the choice to source x86 server nodes, with business model option...
With 10 simultaneous tracks, keynotes, general sessions and targeted breakout classes, Cloud Expo and @ThingsExpo are two of the most important technology events of the year. Since its launch over eight years ago, Cloud Expo and @ThingsExpo have presented a rock star faculty as well as showcased hundreds of sponsors and exhibitors! In this blog post, I provide 7 tips on how, as part of our world-class faculty, you can deliver one of the most popular sessions at our events. But before reading the...
In his session at 21st Cloud Expo, Carl J. Levine, Senior Technical Evangelist for NS1, will objectively discuss how DNS is used to solve Digital Transformation challenges in large SaaS applications, CDNs, AdTech platforms, and other demanding use cases. Carl J. Levine is the Senior Technical Evangelist for NS1. A veteran of the Internet Infrastructure space, he has over a decade of experience with startups, networking protocols and Internet infrastructure, combined with the unique ability to it...
Internet of @ThingsExpo, taking place October 31 - November 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA, is co-located with 21st Cloud Expo and will feature technical sessions from a rock star conference faculty and the leading industry players in the world. The Internet of Things (IoT) is the most profound change in personal and enterprise IT since the creation of the Worldwide Web more than 20 years ago. All major researchers estimate there will be tens of billions devic...
SYS-CON Events announced today that CA Technologies has been named "Platinum Sponsor" of SYS-CON's 21st International Cloud Expo®, which will take place October 31-November 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA. CA Technologies helps customers succeed in a future where every business - from apparel to energy - is being rewritten by software. From planning to development to management to security, CA creates software that fuels transformation for companies in the applic...