When it comes to server consoles, users have two major
options, graphical or text. I personally prefer a serial
console to a graphical one, and this preference isn't all
rooted in snobbery. Serial console switches, unlike KVM
switches, are cross-platform and relatively inexpensive. The
problem with serial console switches, though, is they aren't
expandable. As we buy additional servers, we often find
ourselves short of available ports. Typically, I've punted in
these situations by using an available serial port on a nearby
machine and a null modem cable to create a temporary console
server.
This situation got us thinking. It would be really cool to
have a dedicated, remotely accessible console server with a
lot of serial ports connected to all of our servers. One way
to do this would be to use multiport serial cards, but they
are comparable in price to serial console switches and have a
similar scalability problem. Instead, we figured out how to
use the readily available USB bus with USB-to-serial adapters.
This solution works well even in our mixed environment, and it
is also scalable and inexpensive.
Before embracing this idea as the console access cure-all,
however, we should discuss the limitations. The first is the
USB bus, which is limited to 127 devices per controller. This
seems like a lot, but it is important to remember that hubs
count as one device, and under Linux the root hub also counts
as a device. So, consider a tree of USB hubs and USB-serial
adapters. Let n be the number of ports per hub, k be the
number of hubs in the tree and r be the number of USB-serial
adapters. This gives us a total of n * k ports. To connect the
k hubs in a tree we require a minimum of k – 1 interconnects,
so the maximum number of USB-serial adapters is r = nk – (k –
1). Because there is a maximum of 126 devices (127 minus one
for the root hub), in the best case we have k + r = 126.
Solving for nk yields two equations, nk = 125 and r = 126 – k.
When nk = 125 does not have an integer solution, it is best to
round k up; otherwise, less than the maximum number of ports
will be available. Consequently, with four-port hubs (n = 4)
we have k = 32 and r = 94, for a maximum of 94 USB-serial
adapters on a single controller. With seven-port hubs, k = 18
and r = 108.
We can do a little better than 127 devices by using
additional USB controllers. Although most PCs are equipped
with multiple USB ports, these ports often are on the same
controller, and the 127 device limit constrains both ports.
Even if we add more controllers, there is an upper limit of
256 USB-serial devices allowed, or r </= 256. This is a
limitation of the number of allowable minor numbers for the
assigned USB-serial major number, which is 188. More
industrious users probably could modify the driver to allow
additional majors. Adding additional controllers also may be
required in cases where the serial ports have a high baud
rate; keep in mind the USB bus is limited to 12Mbps. Timing
issues also may keep the usable number of USB-serial adapters
below the theoretical limit, but adding controllers is
relatively easy and inexpensive.
The other limitation of using USB devices is the
interconnect length; cables are limited to a maximum length of
five meters. Effective cable length can be extended by using
an active device, such as a powered hub or powered extension
cable, though even an active extension cable counts as a
device. The total depth is limited to seven tiers, counting
the root and the bottom device. This means there is a maximum
of six 5m interconnects for a total length of 30m. Although
30m is sufficient to reach each corner of our server room, the
reach can be increased another 15m by using shielded RS-232
cable to connect the USB-serial adapter to the server console
port. Other ways of extending the RS-232 signal are available;
for example, use a pair of RS-422 adapters with an effective
range of about 1.3km.
The choice of hub is irrelevant; however, all hubs must be
powered to limit signal degradation. However, the selection of
USB-serial adapters is important. The first adapter we tried
was a Xircom/Entrega, which was listed as experimental in the
kernel driver list. As it turns out, the vendor never provided
source code for its drivers, so the Linux driver was developed
through reverse engineering. Though this driver works with
some models, it didn't work with the one we purchased. To
avoid a similar fate, spend some time looking through the
USB-users mailing list archive or the USB device database (see
Resources).
The Keyspan line of adapters often is recommended, but they
are fairly expensive, around $50 each. We managed to find some
Maxxtro adapters based on the pl2303 chip for about $15 each,
which have worked well. These adapters are essentially a cable
with a USB connector on one end, some circuitry in the middle
and a male DB9 on the other. The only other thing needed was a
DB9f-DB9f serial cable to connect the adapter to the console
port; the serial cable needs to be a null modem cable.
We spent about $16 for each hub, $5 for each 5m USB A/B
cable, $15 for each USB-serial adapter and about $2 for each
1m null modem cable. If we assume that each four-port hub
connects three adapters, leaving the fourth port for
descendant hubs, and that each hub requires one 5m
interconnect, then each console port costs (16 + 5 + 3 * 15 +
3 * 2) / 3, or $24/console on average. Consequently, 16
USB-serial console ports will cost about $384 US.
Of course, there is the expense of the console server
itself, but I'm discounting this because we simply used a
machine that otherwise would have been given away. Secondly,
most serial console switches do not come with the necessary
cabling hardware I've included in the per-port cost of the USB
solution.
Now that you are ready to buy some hardware and try it out,
how do you make it work? First, configure your kernel with USB
support. At a minimum, you need CONFIG_USB, CONFIG_USB_SERIAL
and CONFIG_USB_SERIAL_PL2303 (or the driver for the adapter
you choose) set to y or m. You also need one of the USB
controller drivers, EHCI, UHCI or OHCI. I recommend building
all these drivers as modules to simplify any troubleshooting
you may need to perform. Once the modules are built, plug in a
USB-serial adapter, load the modules and check the output of
dmesg. You should see several lines of output, ending
with: usbserial.c: PL-2303 converter detected
usbserial.c: PL-2303 converter
↪now attached to ttyUSB0
Here I connected one of the USB-serial adapters to a hub.
The adapter has been assigned the device name ttyUSB0, as it
was the first USB-serial adapter detected. If you were to
connect this adapter to a console port and point minicom at
/dev/ttyUSB0, you should be able to establish a connection.
Although the process is that simple, this is where we
encounter a problem. Once I connect a server to this adapter,
there is no assurance this server is always available at
/dev/ttyUSB0. The ttyUSB device numbers are assigned in the
order that the devices are detected on the bus. Although the
order in which devices are detected is a well defined
algorithm, the problem is the USB-serial driver always assigns
the first available ttyUSB device number. As a simple example,
consider two devices, ttyUSB0 and ttyUSB1. If we disconnect
the adapter assigned to ttyUSB1, disconnect ttyUSB0 and then
reconnect the adapter that was ttyUSB1, it is now ttyUSB0
because that is the first available ttyUSB device number.
Of course, we could avoid this problem by not plugging and
unplugging devices, though arguably this ability is a strength
of USB. However, there is still a problem: devices may be
detected at times other than at module load. Consider adding a
new USB-serial adapter to an existing tree of devices, perhaps
to connect a new server. Because the new device is now the
last detected device, it receives the next available ttyUSB
device number. This probably will be the highest ttyUSB device
number, assuming no devices have been disconnected previously.
However, if this device is added close to the root hub of the
tree, then the next time the console server is rebooted or the
USB-serial module is reloaded, this device may be assigned a
low ttyUSB device number, as it probably will be one of the
earliest detected devices.
A possible solution: what if we could locate the desired
USB-serial adapter by its position in the device tree? This
option would be more reliable, because it is less likely that
the existing structure will be modified. That is, we can
choose to preserve the position of existing devices in the
tree, even when adding new devices. Inspection of
/proc/bus/usb/devices reveals that each device detected on the
bus has a topology field of the form: T: Bus=# Lev=# Prnt=# Port=# Cnt=# Dev#=#
The fields of interest are Port, which indicates the
position of this device on its parent device (usually a hub),
and Prnt, which indicates the USB device ID of the device,
again usually a hub, to which this device is connected.
Working backward from a specific device to the root, the path
of the device can be determined recursively. Although this
does connect the USB path to a specific device that represents
its position in the tree, the information about which ttyUSB
device number was assigned is not available from
/proc/bus/usb/devices. The only connection we have to the USB
path is the assigned USB device number from the Dev field.
Unfortunately, USB device numbers are assigned in the order
the devices are detected. As such, it has the same problem as
before; if a device is added to an existing tree it will be
assigned the next available USB device number, which may not
be the same assignment when the USB-serial module is reloaded.
What is needed is a way to associate the USB path directly
to the assigned ttyUSB device number. As it turns out, the USB
developers already have solved this problem. Starting with
kernel 2.4.20-pre7, a new proc entry for the USB-serial driver
exists: /proc/tty/driver/usb-serial. This driver contains
entries that, at least as of pre7, look like: 0: module:pl2303 name:"PL-2303" vendor:067b
↪product:2303 num_ports:1 port:1
↪path:usb-00:07.2-2.3.4
1: module:pl2303 name:"PL-2303" vendor:067b
↪product:2303 num_ports:1 port:1
↪path:usb-00:07.2-2.4.4
The first colon-separated field is the assigned ttyUSB
device number, and the USB path is the bit after the path:
part. In this example, the first line indicates that ttyUSB0
is the USB-serial device connected to path usb-00:07.2-2.3.4,
which working backward, translates to port 4 of a hub plugged
in to port 3 of a hub, plugged in to port 2 of the root hub
00:07.2.
The 00:07 part is a bit of a bonus. This field uniquely
describes the USB controller, so we also have a way of
determining the controller to which the device is assigned.
To make this actually work, it is necessary to store a
mapping of server names to USB paths. That is, after building
a tree of USB-serial devices and connecting them to the server
consoles, create a text file that maps the server names to the
USB paths from /proc/tty/driver/usb-serial. Then it is a
simple matter to write a script that accepts a server name,
parses this file to determine the USB path and then parses
/proc/tty/driver/usb-serial to determine the ttyUSB device
number. Once the ttyUSB device number is known, the script can
establish a connection to the port using minicom as an
example. I went a step further and wrote a script that probes
each ttyUSB device, tries to determine the host from the login
banner and then records the USB path and server name. This
isn't foolproof though, as some OSes don't provide the
hostname in the banner. Even worse, if the console is left
logged on, the banner won't be available. Still, this option
makes creating the mapping file a little easier.
Once a console port is connected to a USB-serial adapter,
the console can be accessed by using minicom, screen or your
favorite terminal program to connect to the assigned ttyUSB
port number. This will work immediately if the server supports
a serial console natively, as do most Sun, HP-UX and IBM
machines. Less commonly, some PCs include a console
redirection feature in the BIOS that allows full redirection
of video and keyboard to a serial port. If hardware
redirection is not available, software redirection can be used
by setting up a tty entry in /etc/inittab, similar to one of
the following:
IRIX: t1:23:respawn:/sbin/suattr -C
↪CAP_FOWNER,CAP_DEVICE_MGT,CAP_DAC_WRITE+ip
↪-c "exec /sbin/getty ttyd1 console"
A good starting point for working with remote serial
consoles is the Remote-Serial-Console-HOWTO (see Resources).
The disadvantage of using a software-redirected port is you
are unable to access the console if the machine fails to boot.
You also are unable to access any portions of the system that
occur before the OS starts. As an example, with a PC you are
unable to access the PC BIOS or any controller BIOS. If this
level of access is critical, you may be interested in a
product called PC Weasel (http://www.realweasel.com/), which creates a
fully accessible, hardware-redirected console and also
provides the ability to reset the PC remotely.
Because of the inability to hard-reset or access the
console before the OS boots without a hardware console port,
this console-access solution does not replace being there.
However, this access solution is easy to extend and costs
almost the same as a traditional serial console switch. In
addition, ports can be added as needed and can be added hot.
What really separates a USB console server from a manual
switch is the ability to access the consoles remotely and in
parallel. That is, as many users as desired can connect
remotely to the USB console server and then connect each shell
to a separate, albeit unique, console. Moreover, the only
limit to accessing the connected consoles is the limit of
accessing the USB console server; you could use a local
keyboard and monitor, SSH, dial-up modem, a custom Web
interface, e-mail and so on. Of course, because the USB-serial
adapters provide a standard serial port, other applications
may be possible as well. Currently, we are considering using
this system to monitor our UPS devices and manage shutdown
events.
