2.6 Client/Server Communications
To tune client/server or distributed applications, you need to identify all communications that occur during execution. The most important factors to look for are the number of transfers of incoming and outgoing data and the amount of data transferred. These elements affect performance the most. Generally, if the amount of data per transfer is less than about one kilobyte, the number of transfers is the factor that limits performance. If the amount of data being transferred is more than about a third of the network's capacity, the amount of data is the factor limiting performance. Between these two endpoints, either the amount of data or the number of transfers can limit performance, although in general, the number of transfers is more likely to be the problem.
As an example, web surfing with a browser typically hits both problems at different times. A complex page with elements from multiple sites can take longer to display completely than one simple page with 10 times more data. Many different sites are involved in displaying the complex page; each site must have its server name converted to an IP address, which can take many network transfers.[12] Each site then needs to be connected to and downloaded from. The simple page needs only one name lookup and one connection, and this can make a huge difference. On the other hand, if the amount of data is large compared to the connection bandwidth (the speed of the Internet connection at the slowest link between your client and the server machine), the limiting factor is bandwidth, so the complex page may display more quickly than the simple page.
[12] The DNS name lookup is often hierarchical, requiring multiple DNS servers to chain a lookup request to resolve successive parts of the name. Although there is only one request as far as the browser is concerned, the actual request may require several server-to-server data transfers before the lookup is resolved.
Several generic tools are available for monitoring communication traffic, all aimed at system and network administrators (and quite expensive). I know of no general-purpose profiling tool targeted at application-level communications monitoring; normally, developers put their own monitoring capabilities into the application or use the trace mode in their third-party communications package, if they use one. (snoop, netstat, and ndd on Unix are useful communication-monitoring tools. tcpdump and ethereal are freeware communication-monitoring tools.)
If you are using a third-party communications package, your first step in profiling is to make sure you understand how to use the full capabilities of its tracing mode. Most communications packages provide a trace mode to log various levels of communication details. Some let you install your own socket layer underlying the communications; this feature, though not usually present for logging purposes, can be quite handy for customizing communications tracing.
For example, RMI (remote method invocation) offers very basic call tracing enabled by setting the java. rmi.server.logCalls property to true, e.g., by starting the server class with:
% java -Djava.rmi.server.logCalls=true <ServerClass> ...
The RMI framework also lets you install a custom RMI socket factory. This socket customization support is provided so that the RMI protocol is abstracted away from actual communication details, and it allows sockets to be replaced by alternatives such as nonsocket communications or encrypted or compressed data transfers.
For example, here is the tracing from a small client/server RMI application. The client simply connects to the server and sets three attributes of a server object using RMI. The three attributes are a boolean, an Object, and an int, and the server object defines three remotely callable set( ) methods for setting the attributes:
Sun Jan 16 15:09:12 GMT+00:00 2000:RMI:RMI TCP Connection(3)-localhost/127.0.0.1: [127.0.0.1: tuning.cs.ServerObjectImpl[0]: void setBoolean(boolean)] Sun Jan 16 15:09:12 GMT+00:00 2000:RMI:RMI TCP Connection(3)-localhost/127.0.0.1: [127.0.0.1: tuning.cs.ServerObjectImpl[0]: void setObject(java.lang.Object)] Sun Jan 16 15:09:12 GMT+00:00 2000:RMI:RMI TCP Connection(3)-localhost/127.0.0.1: [127.0.0.1: tuning.cs.ServerObjectImpl[0]: void setNumber(int)]
If you can install your own socket layer, you may also want to install a customized logging layer to provide details of the communication. An alternative way to trace communications is to replace the sockets (or other underlying communication classes) directly, providing your own logging. In the next section, I provide details for replacing socket-level communication for basic Java sockets.
In addition to Java-level logging, you should be familiar with system- and network-level logging facilities. The most ubiquitous of these is netstat, a command-line utility that is normally executed from a Unix shell or Windows command prompt. For example, using netstat with the -s option provides a full dump of most network-related structures (cumulative readings since the machine was started). By filtering this, taking differences, and plotting various data, you get a good idea of the network traffic background and the extra load imposed by your application.
Using netstat with this application shows that the connection, the resolution of the server object, and the three remote method invocations require four TCP sockets and 39 packets of data (frames) to be transferred. These include a socket pair opened from the client to the registry to determine the server location, then a second socket pair between the client and the server. The frames include several handshake packets required as part of the RMI protocol, and other overhead that RMI imposes. The socket pair between the registry and server are not recorded because the pair lives longer than the interval that measures differences recorded by netstat. However, some of the frames are probably communication packets between the registry and the server.
Another useful piece of equipment is a network sniffer. This is a hardware device you plug into the network line that views (and can save) all network traffic that is passed along that wire. If you absolutely must know every detail of what is happening on the wire, you may need one of these.
More detailed information on network utilities and tools can be found in system-specific performance tuning books (see Chapter 14 for more about system-specific tools and tuning tips).
2.6.1 Replacing Sockets
Occasionally, you need to be able to see what is happening to your sockets and to know what information is passing through them and the size of the packets being transferred. It is usually best to install your own trace points into the application for all communication external to the application; the extra overhead is generally small compared to network (or any I/O) overhead and can usually be ignored. The application can be deployed with these tracers in place but configured so as not to trace (until required).
However, the sockets are often used by third-party classes, and you cannot directly wrap the reads and writes. You could use a packet sniffer plugged into the network, but this can prove troublesome when used for application-specific purposes (and can be expensive). A more useful possibility I have employed is to wrap the socket I/O with my own classes. You can almost do this generically using the SocketImplFactory, but if you install your own SocketImplFactory, there is no protocol to allow you to access the default socket implementation, so another way must be used. (You could add a SocketImplFactory class into java.net, which then gives you access to the default PlainSocketImpl class, but this is no more generic than the previous possibility, as it too cannot normally be delivered with an application.) My preferred solution, which is also not deliverable, is to wrap the sockets by replacing the java.net.Socket class with my own implementation. This is simpler than the previous alternatives and can be quite powerful. Only two methods from the core classes need changing, namely those that provide access to the input stream and output stream. You need to create your own input stream and output stream wrapper classes to provide logging. The two methods in Socket are getInputStream( ) and getOutputStream( ), and the new versions of these look as follows:
public InputStream getInputStream( ) throws IOException {
return new tuning.socket.SockInStreamLogger(this, impl.getInputStream( ));
}
public OutputStream getOutputStream( ) throws IOException {
return new tuning.socket.SockOutStreamLogger(this, impl.getOutputStream( ));
}
The required stream classes are listed shortly. Rather than using generic classes, I tend to customize the logging on a per-application basis. I even tend to vary the logging implementation for different tests, slowly cutting out more superfluous communications data and headers so that I can focus on a small amount of detail. Usually I focus on the number of transfers, the amount of data transferred, and the application-specific type of data being transferred. For a distributed RMI type communication, I want to know the method calls and argument types and occasionally some of the arguments: the data is serialized and so can be accessed using the Serializable framework.
As with the customized Object class in Section 2.4, you need to ensure that your customized Socket class comes first in your (boot) classpath, before the JDK Socket version. The RMI example from the previous section results in the following trace when run with customized socket tracing. The trace is from the client only. I have replaced lines of data with my own interpretation (in bold) of the data sent or read:
Message of size 7 written by Socket Socket[addr=jack/127.0.0.1,port=1099,localport=1092] client-registry handshake Message of size 16 read by Socket Socket[addr=jack/127.0.0.1,port=1099,localport=1092] client-registry handshake Message of size 15 written by Socket Socket[addr=jack/127.0.0.1,port=1099,localport=1092] client-registry handshake: client identification Message of size 53 written by Socket Socket[addr=jack/127.0.0.1,port=1099,localport=1092] client-registry query: asking for the location of the Server Object Message of size 210 read by Socket Socket[addr=jack/127.0.0.1,port=1099,localport=1092] client-registry query: reply giving details of the Server Object Message of size 7 written by Socket Socket[addr=localhost/127.0.0.1,port=1087,localport=1093] client-server handshake Message of size 16 read by Socket Socket[addr=localhost/127.0.0.1,port=1087,localport=1093] client-server handshake Message of size 15 written by Socket Socket[addr=localhost/127.0.0.1,port=1087,localport=1093] client-server handshake: client identification Message of size 342 written by Socket Socket[addr=localhost/127.0.0.1,port=1087,localport=1093] client-server handshake: security handshake Message of size 283 read by Socket Socket[addr=localhost/127.0.0.1,port=1087,localport=1093] client-server handshake: security handshake Message of size 1 written by Socket Socket[addr=jack/127.0.0.1,port=1099,localport=1092] Message of size 1 read by Socket Socket[addr=jack/127.0.0.1,port=1099,localport=1092] Message of size 15 written by Socket Socket[addr=jack/127.0.0.1,port=1099,localport=1092] client-registry handoff Message of size 1 written by Socket Socket[addr=localhost/127.0.0.1,port=1087,localport=1093] Message of size 1 read by Socket Socket[addr=localhost/127.0.0.1,port=1087,localport=1093] Message of size 42 written by Socket Socket[addr=localhost/127.0.0.1,port=1087,localport=1093] client-server rmi: set boolean request Message of size 22 read by Socket Socket[addr=localhost/127.0.0.1,port=1087,localport=1093] client-server rmi: set boolean reply Message of size 1 written by Socket Socket[addr=localhost/127.0.0.1,port=1087,localport=1093] Message of size 1 read by Socket Socket[addr=localhost/127.0.0.1,port=1087,localport=1093] Message of size 120 written by Socket Socket[addr=localhost/127.0.0.1,port=1087,localport=1093] client-server rmi: set Object request Message of size 22 read by Socket Socket[addr=localhost/127.0.0.1,port=1087,localport=1093] client-server rmi: set Object reply Message of size 45 written by Socket Socket[addr=localhost/127.0.0.1,port=1087,localport=1093] client-server rmi: set int request Message of size 22 read by Socket Socket[addr=localhost/127.0.0.1,port=1087,localport=1093] client-server rmi: set int reply
Here is one possible implementation for the stream classes required by the altered Socket class:
package tuning.socket;
import java.io.InputStream;
import java.io.OutputStream;
import java.io.IOException;
import java.net.Socket;
public class SockStreamLogger
{
public static boolean LOG_SIZE = false;
public static boolean LOG_MESSAGE = false;
public static void read(Socket so, int sz, byte[ ] buf, int off) {
log(false, so, sz, buf, off); }
public static void written(Socket so, int sz, byte[ ] buf, int off) {
log(true, so, sz, buf, off); }
public static void log(boolean isWritten, Socket so,
int sz, byte[ ] buf, int off)
{
if (LOG_SIZE)
{
System.err.print("Message of size ");
System.err.print(sz);
System.err.print(isWritten ? " written" : " read");
System.err.print(" by Socket ");
System.err.println(so);
}
if (LOG_MESSAGE)
System.err.println(new String(buf, off, sz));
}
}
public class SockInStreamLogger extends InputStream
{
Socket s;
InputStream in;
byte[ ] one_byte = new byte[1];
public SockInStreamLogger(Socket so, InputStream i){in = i; s = so;}
public int available( ) throws IOException {return in.available( );}
public void close( ) throws IOException {in.close( );}
public void mark(int readlimit) {in.mark(readlimit);}
public boolean markSupported( ) {return in.markSupported( );}
public int read( ) throws IOException {
int ret = in.read( );
one_byte[0] = (byte) ret;
//SockStreamLogger.read(s, 1, one_byte, 0);
return ret;
}
public int read(byte b[ ]) throws IOException {
int sz = in.read(b);
SockStreamLogger.read(s, sz, b, 0);
return sz;
}
public int read(byte b[ ], int off, int len) throws IOException {
int sz = in.read(b, off, len);
SockStreamLogger.read(s, sz, b, off);
return sz;
}
public void reset( ) throws IOException {in.reset( );}
public long skip(long n) throws IOException {return in.skip(n);}
}
public class SockOutStreamLogger extends OutputStream
{
Socket s;
OutputStream out;
byte[ ] one_byte = new byte[1];
public SockOutStreamLogger(Socket so, OutputStream o){out = o; s = so;}
public void write(int b) throws IOException {
out.write(b);
one_byte[0] = (byte) b;
SockStreamLogger.written(s, 1, one_byte, 0);
}
public void write(byte b[ ]) throws IOException {
out.write(b);
SockStreamLogger.written(s, b.length, b, 0);
}
public void write(byte b[ ], int off, int len) throws IOException {
out.write(b, off, len);
SockStreamLogger.written(s, len, b, off);
}
public void flush( ) throws IOException {out.flush( );}
public void close( ) throws IOException {out.close( );}
}
