OpenOnload-201606-u1.3 ====================== This is a minor update release to OpenOnload201606-u1.2 containing a number of important bug fixes, an updated net driver, and support for recent Linux kernels and enterprise distributions. Please see the ChangeLog for a list of bugs fixed. Testing for this release has included the current beta version of RHEL7.4, and any issues found have been addressed. We therefore anticipate that it should work with this distribution, but we won't be able to confirm this until the final version is available. This release includes support for Linux kernel 4.11. 4.10 kernel support was added in the previous release. During testing of this release we have discovered an issue with signal handling of 32 bit applications running on a 64 bit kernel for kernel versions 4.10 or later. This is not a regression from the previous release. It will be addressed in the next update release. Until that is available we recommend avoiding onloading 32 bit applications that use signals on Linux kernel versions 4.10 or later. The Release Notes for OpenOnload-201606-u1.2 are included below. For more details see the TCPDirect User Guide and Onload User Guide. Linux distribution support -------------------------- This package is supported on: - Red Hat Enterprise Linux 6.7 - 6.8 - Red Hat Enterprise Linux 7.1 - 7.3 (and potentially RHEL7.4) - SuSE Linux Enterprise Server 11 sp4 - SuSE Linux Enterprise Server 12 sp1 and sp2 - Canonical Ubuntu Server LTS 14.04, 16.04 - Canonical Ubuntu Server 16.10, 17.04 - Debian 7 "Wheezy" - Debian 8 "Jessie" - Debian 9 "Stretch" - Linux kernels 2.6.32 - 4.11 OpenOnload-201606-u1.2 ====================== This is a minor update release to OpenOnload-201606-u1.1 containing a number of important bug fixes, an updated net driver, and support for Linux 4.9 and 4.10 kernels. Please see the ChangeLog for a list of bugs fixed. The Release Notes for OpenOnload-201606-u1.1, which includes a brief summary of new features and known limitations, are included below. For more details see the TCPDirect User Guide and Onload User Guide. Linux distribution support -------------------------- This package is supported on: - Red Hat Enterprise Linux 6.7 - 6.8 - Red Hat Enterprise Linux 7.1 - 7.3 - SuSE Linux Enterprise Server 11 sp4 - SuSE Linux Enterprise Server 12 sp1 and sp2 - Canonical Ubuntu Server LTS 14.04, 16.04 - Canonical Ubuntu Server 16.10 - Debian 7 "Wheezy" - Debian 8 "Jessie" - Linux kernels 2.6.32 - 4.10 OpenOnload-201606-u1.1 ====================== This is a minor update release to OpenOnload-201606-u1 containing a number of important bug fixes. Please see the ChangeLog for a list of bugs fixed. The Release Notes for OpenOnload-201606-u1, which includes a brief summary of new features and known limitations, are included below. For more details see the TCPDirect User Guide and Onload User Guide. OpenOnload-201606-u1 ==================== This release builds on top of the stable OpenOnload-201606 release and adds support for the TCPDirect library and API. See the ChangeLog for a list of bugs fixed. A brief summary of new features and known limitations are included below - for more details see the TCPDirect User Guide and Onload User Guide. TCPDirect --------- Supported hardware platforms: TCPDirect is currently only available for the Solarflare Flareon Ultra 8500 series of network adapters with a suitable licence installed. Documentation: Installation of TCPDirect follows the same process as OpenOnload. Please see the OpenOnload User Guide or Quick Start Guide for details. Please see the TCPDirect User Guide for details of configuring and using TCPDirect. TCP Alternatives Firmware: Usage of the TCPDirect Alternatives API needs a more recent firmware version than is currently supplied with the Solarflare Flareon Ultra 8500 series network adapters. If you plan to use this feature please contact support@solarflare.com to obtain suitable firmware. Routing: TCPDirect makes routing decisions when a connection is established, and caches that information. During the connection lifetime the routing information is not updated, so: - it does not detect a change of MAC address of the next hop - it does not do PMTU discovery, it uses the MTU specified in route table entry - it does not detect route changes TCPDirect also can't use network interfaces other than the interface specified in the attribute. An attempt to use a route via another network interface will result in error. Changes compared to preview: There have been a number of updates to the TCPDirect API since the initial preview was made, including - All TCPDirect headers are now accessed by including zf.h - zft_send_single() added for optimised send of a single iovec - zft_send() expanded to support multiple iovecs - zft_send()/zft_send_single() can send partial iovecs, and return ssize_t - Add support for MSG_MORE flag on TCP send calls - API in general uses sockaddr rather than sockaddr_in - zfur_zc_recv() now only returns a single datagram as documented - Add zf_stack_has_pending_work() - Add zft_get_mss() - Removed unimplemented API calls from the headers. It is still intended to add support for things like hardware timestamps in a future release. - Change zf_muxer_wait() to take a timeout in nanoseconds rather than milliseconds - Added zf_debug wrapper script to make use of debug libraries easier - zf{tcp,udp}pingpong sample apps rewritten, and zfaltpingpong added to demonstrate the alternatives API. Linux distribution support -------------------------- This package is supported on: - Red Hat Enterprise Linux 6.6 - 6.8 - Red Hat Enterprise Linux 7.0 - 7.2 - SuSE Linux Enterprise Server 11 sp2, sp3, sp4 - SuSE Linux Enterprise Server 12 base and sp1 - Canonical Ubuntu Server LTS 14.04, 16.04 - Canonical Ubuntu Server 16.10 - Debian 7 "Wheezy" - Debian 8 "Jessie" - Linux kernels 2.6.32 - 4.8 New configuration options ------------------------- EF_WODA_SINGLE_INTERFACE: This mode removes the constraint on ordering being correct across interfaces when using onload_ordered_epoll_wait(). If this mode is enabled by setting EF_WODA_SINGLE_INTERFACE=1 then traffic is only ordered with respect to other traffic on the same interface. This can give a significant latency benefit with certain traffic patterns. EF_TCP_SHARED_LOCAL_PORTS: This feature improves the performance of TCP active-opens. It reduces the cost of both blocking and non-blocking connect() calls, reduces the latency to establish new connections, and enables scaling to large numbers of active-open connections. It also reduces the cost of closing these connections These improvements are achieved by sharing a set of local port numbers amongst active-open sockets, which saves the cost and scaling limits associated with installing packet steering filters for each active-open socket. Shared local ports are only used when the local port is not explicitly assigned by the application." Set this option to >=1 to enable local port sharing. The value set gives the initial number of local ports to allocate when the Onload stack is created. More shared local ports are allocated on demand as needed up to the maximum given by EF_TCP_SHARED_LOCAL_PORTS_MAX. Note that typically only one local shared port is needed, as different local ports are only needed when multiple connections are made to the same remote IP:port. Onload Extensions API --------------------- This release adds onload_socket_nonaccel() to the Onload extensions API. This function provides a convenient way to allocate a socket that is not accelerated by Onload. Its arguments and return codes are identical to socket(). This release also makes the return codes and errno use from the Onload extensions API consistent between the main library and the two stub implementations. Onload Remote Monitor --------------------- The OnloadRemoteMonitor (ORM) system for gathering Onload statistics from a monitoring system has been improved and extended in this release. These statistics can now be accessed via http - the example webserver is started using 'orm_webserver '. Alternatively, 'orm_json' can be invoked directly by a customer script. An example of using collectd to gather statistics is provided in src/tests/onload/onload_remote_monitor/using_collectd/. When running the example webserver, statistics can now be accessed for an individual named stack using a URL like http://:/onload/stackname//lots/ OpenOnload-201606 Release Notes =============================== This is a major update release that adds new features to OpenOnload. A brief summary of new features and known limitations are included below - for more details see the Onload user guide. See the ChangeLog for a list of bugs fixed. Linux distribution support -------------------------- This Onload package is supported on: - Red Hat Enterprise Linux 5.10 - 5.11 - Red Hat Enterprise Linux 6.6 - 6.8 - Red Hat Enterprise Linux 7.0 - 7.2 - Red Hat Messaging Realtime and Grid 2.4, 2.5 - SuSE Linux Enterprise Server 11 sp2, sp3, sp4 - SuSE Linux Enterprise Realtime Extension 11 - SuSE Linux Enterprise Server 12 base and sp1 - Canonical Ubuntu Server LTS 14.04 - Canonical Ubuntu Server 15.10, 16.04 - Debian 7 "Wheezy" - Debian 8 "Jessie" - Linux kernels 2.6.32 - 4.6 Solarflare Flareon Ultra 8500 NICs ---------------------------------- This release adds full support for the new range of Solarflare Flareon Ultra 8500-series NICs. If you are interested in learning more about the improved performance and features available please contact sales@solarflare.com New configuration options ------------------------- EF_TCP_LISTEN_REPLIES_BACK: When a TCP listening socket replies to an incoming SYN, this option forces Onload to ignore the route table and to reply on the same network interface the SYN was received from. This mode could be considered as a way to achieve some basic source-based routing EF_HIGH_THROUGHPUT_MODE: This option causes onload to optimise the stack and VI configuration for throughput at the cost of latency In addition, some of the control-plane module options have been moved or renamed - please see the control plane section below for details. ef_vi Transmit Alternatives API ------------------------------- This release adds a new method for sending data to the ef_vi API. Using the ef_vi_transmit_alt* functions a set of alternative packets can be queued in the NIC ready to be sent. Once the decision about which alternative set of packets to send is made, it can be released onto the network with extremely low latency. Please refer to the ef_vi documentation for further details. This API is supported on 8000-series NICs and later. ef_vi Capabilities API ---------------------- This release adds support for an ef_vi API that allows users to query at runtime what features and capabilities are available on the hardware as it is currently configured. This allows applications to be built once, then run on a variety of Solarflare NICs and tailor the feature set they use to the current hardware capabilities. Please see src/include/etherfabric/capabilites.h for the API documentation and details of the features that can currently be queried. IPv6 ef_vi filtering -------------------- This release adds support for IPv6 filters in ef_vi. Please see the documentation for ef_filter_spec_set_ip6_local() and ef_filter_spec_set_ip6_full() for further details. This is supported on 7000-series NICs and later. onload_thread_get_spin extension API ------------------------------------ This release adds a new extensions API call to query per-thread spin settings, to complement the existing onload_thread_set_spin(). onload_thread_get_spin() retrieves the spin settings previously configured via onload_thread_set_spin(). Please refer to src/include/onload/extensions.h for the API documentation. Teamd support for mixed bonds ----------------------------- OpenOnload-201509 added support for bonds configured using teamd in addition to the long-standing bonding module. It has recently become apparent that Onload is not correctly dealing with teamd-configured bonds that have a mix of Solarflare and non-Solarflare interfaces. Onload should recognise these as unacceleratable but currently will attempt to accelerate traffic on the Solarflare interfaces in the bond, which may lead to unexpected behaviour should the bond failover to a non-Solarflare interface. We will address this in a future update release. To work around the problem we recommend using the bonding module to configure bonds that have a mix of Solarflare and non-Solarflare interfaces in them. sfc_resource /proc file name change ----------------------------------- To avoid a naming collision when interfaces are renamed the files that used to be at /proc/driver/sfc_resource/ethX/pdY have moved to /proc/driver/sfc_resource/devices/0000:0Z.00.0/pdY New onload_cplane module ------------------------ This release of Onload adds a new binary module named onload_cplane. This new module provides some of the module parameters that used to be handled by other solarflare modules: - oo_bond_poll_base - oo_igmp_on_failover - max_layer2_interfaces - max_routes - max_neighs - oo_teaming_dump_period - oo_teaming_listen_updates In previous versions max_layer2_interfaces was used to limit the size of both the llap table (table of IP interfaces) and ipif table (table of IP addresses). This has now been separated and max_local_addrs has been added to control the size of the ipif table. The onload_cplane module also takes a new parameter cplane_debug_bits, similar to the existing oo_debug_bits parameter, to control the level of debug logging produced. A number of /proc files have been changed: - All /proc/driver/onload/mib-* have moved to /proc/driver/onload_cplane/mib-*; the files themselves are exactly the same as previously, apart from /proc/driver/onload/mib-hwport which has been removed. - /proc/driver/onload/cplane has been renamed to /proc/driver/onload_cplane/stats. - /proc/driver/onload_cplane/version is a new file containing version information of the onload_cplane module. Some build-time configuration options have been effectively removed as they are set at build time of the onload_cplane module: - CI_CFG_MAX_REGISTER_INTERFACES - CI_CFG_TEAMING Finally, some Linux kernel versions break the API symbols used by Onload and users should upgrade to a later release where these kernel bugs have been fixed: - 4.2.0 and 4.2.1: please use 4.2.2 or later - 3.7.x and 3.8.x: please use 3.7.10 or >=3.9 - 3.0.9-rt* from MRG2.1: please use later MRG kernel sfc_aoe module -------------- The sfc_aoe module is no longer included in the Onload distribution. This will in future be distributed separately to allow independent releases of these two products. If this causes issues please contact support@solarflare.com Interrupt-driven event burst ---------------------------- During testing for this release we have encountered a rare issue when running Onload in interrupt-driven mode with multiple threads or processes sharing a stack and using both ports of a NIC at high traffic rates. In this situation Onload can cause the NIC to generate a large burst of events which in turn will overflow the net driver's event queue. Due to the timings involved this mainly affects the new 8500 series NICs, but can in theory apply to earlier generations of hardware. It is not a new issue, so if you have not experienced this on previous Onload releases there is no additional risk in upgrading to this release. The symptoms include warnings from the kernel driver about driverlink handle_event() operations taking a long time and MCDI requests completing without an event. This may cause the driver to reset the NIC in an attempt to recover. Solarflare have a fix to resolve this that is currently being tested and will be included in an update release.