When we are going to build a dedicated cluster, it is not a good idea to use a common Ethernet adaptor. Neverthless it is cheap and reliable, big bandwidth applications will find a bottleneck here when we have 16 nodes or more. There are some good network interfaces, more expensive than Ethernet, but they increase dramatically the overall performance of our cluster, and that's why they are strongly recommended.
Today, best solution is Myricom net, but tomorrow may vary. Anyway, we must to renember that the PC bus clock may be a bottleneck, limiting the speed to 133Mbps -33MHz busspeed, today nonwritten standard-. IBM NetInfinity 7000 server series can run four times faster, arise to a 64 bits, 66MHz PCI bus, but not all cards are able to afford this.