Copyright Myricom 2003-2005, Markus Fischer fischer@myri.com REQUIREMENTS The Sockets over Myrinet Module (AF_MYRI) implemenation require that you have the kernel sources installed in order to compile the software package. There is *no* requirement to rebuild your operating system or patch your distribution. Sockets over Myrinet Module will nicely fit into your environment without any system specific changes. The Sockets-GM Module requires GM-2.0.14 or GM-2.1.4 or higher. For Myrinet Express (MX), the Sockets-MX Module requires MX 1.0. It requires the configure --enable-kernel-lib option when compiling MX. MX needs to be loaded before loading Sockets-MX into the kernel. INSTALLATION Enabling Sockets over Myrinet: Establishing Sockets over Myrinet functionality is achieved by inserting the af_myri.o module into the OS. This is typically done using insmod under Linux. The module requires an IP address as an argument for identification. (e.g: insmod /opt/sdp-myri/af_myri.o sdp_ip_addr=192.168.1.112) The AF_INET IP address should be taken (eth0). Mapping Socket Communication of applications to use Sockets over Myrinet: The goal is to allow the module af_myri.o to map TCP/IP Traffic to Myrinet. This should be achieved for already precompiled applications. The following options exist for achieving this mapping: 1) Modification to the syscall_table which will remap AF_INET traffic to the AF_MYRI (27) protocol family. As a result any application will be mapped to Sockets over Myrinet If only specific applications should be mapped to AF_MYRI, then the second option is more feasible 2) Provide a small shared library which will remap AF_INET traffic when calling socket() to the AF_MYRI protocol family: e.g.: export LD_PRELOAD=/opt/sdp-myri/libgmsocks.so.1.0 If you want any application to be run under the new sockets family, then you should add the LD_PRELOAD configuration at a system/distribution specific location. e.g: /etc/bash.bashrc This will enable remote shells to be run under the new protocol family as well (e.g: ssh remote-host ls will be run under the new protocol) FUNCTIONALITY The AF_MYRI desgin is implementing a new socket protocol family. Any application which has been mapped to AF_MYRI will trap into the Kernel, but experience no additional protocol overhead. The Myrinet network which guarantees error free data delivery is a key for achieving a reliable protocol for socket applications. Depending on the message size RDMA or buffer semantics will be used to transfer data. One might wonder what happens if application are run in a heterogeneous network environment. As such you could consider to have a distributed application within and outside Myrinet. In this case the new socket protocol will use the traditional AF_INET protocol to achieve full transparency. An architectural overview is depicted in the following: __________________________________ | User Mode Socket Application | ===============|================== Libc | User ========================================= Socket Layer (AF Families) Kernel | | AF_INET | | AF_MYRI | <--------------| TCP/IP | GM / MX | | | | |--------| | | Ethernet Network Myrinet By enabling the fallback mechanism to use AF_INET over TCP/IP it is also possible to let system services work with user mode applications. For example if a remote system has an sshd running and the local hosts contacts via ssh under the new socket protocol, then ssh will still work since it will fall back to AF_INET. However, the newly created process on the remote host will be able to talk directly using the new socket protocol, without overhead. CONFIGURATION GM: A default GM installation is required. An easy way to have the GM hostnames be suitable for Sockets-GM, an /etc/gm/hostname. (e.g: /etc/gm/hostname.0) file containing the AF_INET IP address will set the name automatically. For Channelbonding you need the IP addresses, followed by a colon and board id. Example: user@host:~$ more /etc/gm/hostname.0 192.131.130.83:0 user@host:~$ more /etc/gm/hostname.1 192.131.130.83:1 Note: if you create a new hostname entry GM needs to be restarted. MX: You can set the MX hostname using mx_hostname under tools e.g.: sudo /opt/mx/sbin/mx_start_stop restart sudo /opt/mx/bin/mx_hostname -n $1 sudo /opt/mx/bin/mx_hostname -c If the MX hostname and the specified sdp_ip_addr do not match, the Sockets-MX module will fail to load to avoid unnecessary configuration issues. Please consult dmesg, or tail /var/log/messages for further information For your convenience, the Sockets-MX_MODULE directory contains a script which takes a hostfile as input: ./verify_hostnames. This hostfile should contain the names of the nodes which the script will configure for AF_MYRI use. It assumes that MX has been loaded. MODULE PARAMETERS AF_MYRI offers the following parameters to be set during module load time: sdp_num_board={1,2} - specify the number of Myrinet Boards to use (Board bonding), [1] sdp_board_one={1,2,...} - specify which board number to use as board 1, [0] sdp_board_two={1,2,...} - specify which board number to use as board 2, [1] sdp_zero_copy={0,1} - turn on zero copy, [0] sdp_zero_copy_zize={1000 - n} - specify the message size in bytes when to switch to the zero copy protocol [1000000] COMPILING A common makefile is provided for Linux 2.4.x and Linux 2.6.x Configuration is performed as module params. Run autoheader; autoconf; configure; make LOADING AF_MYRI An example for a fully features Sockets-GM installation sudo ./af_myri.ko sdp_num_board=2 sdp_board_one=2 sdp_board_two=4 sdp_ip_addr=192.168.1.111 sdp_zero_copy=1 sdp_zero_copy_len=500000 This command allows to use two boards for sending, receiving. The board numbers are 2 and 4 [gm_board_info can be called to get information], zero cop is enabled and will start for data transfers with 500000 Bytes. EXAMPLES The examples directory contains small examples to get familiar with Sockets. over Myrinet. There are additional Readme files which help. Otherwise please do not hesitate to contact help@myri.com