How to detect bugs caused by network failure?

Hi everybody

We develop a system consists of a server and several clients. Now, we need to test the cases of failures occurred in the network between the clients and server. How should I generate and control all kinds of network failure between the clients and server?

I imagine to use a computer with 2 NIC (one NIC connect to the server and the other connect to client) and on that computer, there will be a software to control the packages between 2 NIC in order to simulate all kinds of network failure. Anybody has experience about this "testing model"?

Thank you for your reply

H.

▶ Reply to This

Replies to This Discussion

Permalink Reply by Ido Schacham on September 29, 2008 at 9:04pm: No need for a 2 NIC system, have a proper server and client running. Here are some ideas, hope it will inspire you for a whole lot more:
- Disconnect the network cable in the middle of a transaction on the client/server.
- Shutdown the client/server in the middle of a transaction. Another test would be to bring them up again and see what happens.
- Use NIST Net or some other network emulator to cause big network delays, packet loss, corrupt data, fragmentation, etc.
- Use a proxy/man-in-the-middle and control which packets pass, which don't, or modify data in the packets (mostly relevant for web, may be possible for other protocols).
- Try to access a server that's non-existent.
- Have the client/server use DHCP and renew IPs in the middle of a connection.
- Use a Firewall to block connections at will.

Happy testing :); ▶ Reply to This

Permalink Reply by Adam White on October 1, 2008 at 3:19pm: H,

This is a problem we ran into as well. One tool we use to help with this kind of testing is VMware Workstation. It allows you to set up two or more machines and intentionally drop packets. You drop packets based on %, you can limit the bandwidth and do other types of network interactions. There are all kinds of interesting things you can do with it.

You might also look into holodeck - we tried it when it was in it's earlier versions but haven't revisited it lately.

http://www.securityinnovation.com/holodeck/fragility-testing.shtml

Adam; ▶ Reply to This

Permalink Reply by Srihari Palangala on October 16, 2008 at 4:29am: Hi H, All,
It is sometimes difficult to recreate complex network configurations in a physical test lab that reflect the real world setup and configuration. Especially when you get into scenarios with network routers, NATing, public/private networks etc. In these cases you will benefit from using virtual lab automation solutions that help you create n-tier configurations with their networking setup reflecting the real world.

You can then test a lot of the failures on this virtual machine network. You can clone/copy the configuration easily and even run multiple copies of these configs in parallel without worrying about IP conflicts.

This detailed blog post might make things a little more clear:

Hope this helps,

Srihari Palangala; ▶ Reply to This