I’m confused when I see people from other industries taking banking industry as example of high quality software caused by high stake/risk. Banking software is far from perfect. Perfect software cost a lot and is illusion anyway. If internet banking sites are more or less user friendly, then back-office and teller software is not much better than green screens form 80ties. There are a lot of back office people doing workarounds, manual fixes, etc. Why so? I think the reason is high complexity of the software and low return on investment, because humans are still so good at doing their job. And by complexity I don’t just mean code complexity or pages of specifications. The true complexity is behind communication (both one-way, i.e. manuals and two-way such a emails) between people who wrote some (legacy code, 3rd party libraries, development tools, etc.) software in past, people who try to use that software to create and integrate new software and people who are supposed to use that software in future, but are using (and biased by) a different software today. And by the way - add the regulators/auditors to the picture in this industry.
The disliked part of the job
In last 5 or more years I’ve been frequently doing and managing testing of the software the banks pay for. Yes, there is a lot of „ceremonial testing” as James Bach have been calling it or „testing theatre” as I’ve red Ben Kelly calling it, but I don’t want to talk about it. I admit part of our job is what I don’t call a testing, but an evidence generation. Here in this industry software testing is not just an investigation conducted to provide stakeholders with information about the quality. Because our stakeholders are just as interested in the investigation protocol details and the ability to overtake that investigation and make their own conclusions.
Nevertheless there are a lot of interesting testing challenges that you will be missing in most of the other industries. Those unique challenges is want I want to discuss today.
But first of all – the context
Why banking industry is so different? Is it because the stake of a money to be lost? Probably. However as a technical person I will tell you that the biggest difference is the complexity of the IT solution. It changes depending on sub-industry and bank, but a worst case picture is like this: you have more than 20 different systems developed independently and integrated later. They are created by at least 10 different vendors over last 20 or more years (but you will not find a single developer in any of those companies who would be coding or maintaining this software more than 5 years). The business supported by all those 20 systems is so wide that there is not a single person in a bank to understand it all. Actually, most of the bank people rely on existing software as the software knows and help them to understand and manage the business. They don’t know the business logic in details themselves – computer made this knowledge obsolete. When you ask a subject matter expert – how this SWIFT should be processed, they will just run the SWIFT through the system to see how the system process it – and that’s how it should they will say.
High stake, high quality?
Yes, this software help banks to organize huge money transfers – millions or even milliards. But how about the quality? As I’ve wrote – the software in use is 10 or even 20 years old. It’s not uncommon for bank people to work with mainframe green screens and having to retype IDs, codes, etc from one screen to another rather than just click on a hyperlink or do a drag and drop with a mouse. You can’t compare the their experience with what we experience using free software such as google maps. Depending on how you define quality bank’s people are probably working with most terrible software in this world. That’s because software is used by their internal stuff – not by their customers. Internet banking applications are typically the most user friendly piece of software there.
Anyway, let’s get back to the main topic – how do we test this software that are designed to be not user friendly, but still reliable.
Testing step 1: get it working on my machine, or system testing
The first step is obvious and just like in any industry: software works on developer’s machine, executed from their IDE, with stubs, etc. The only difference is the complexity and as a result the number of stubs, fake data used. In other words – the difference between the developer’s machine configured environment and the final system environment is enormous. The ultimate goal is to get it working deployed on web server, connected to real database, filled with customer data. But otherwise it’s in no way interesting topic for me – so I would like to skip this step and move on to first real tester’s task
Testing step 2: smoke test automation
We want to make sure it still works when new builds are deployed, or any changes to environment done. Anything could break there – it’s just too complicated and complex to predict. And it does break more often that don’t. Even of you deploy every day or several times a day. So defining very limited (no more than 30-120 person minutes big) “smoke tests” is almost a must and automating them is a very good idea because
Testing step 3: testing different cases
So, developers have developed and unit tested code. But they are too much “unit” and it don’t help you. Let’s take an example unit tests process different values of a single SWIFT field. The only issue is – they are changing value of one field without adjust the other fields, so they are testing with swifts that are: 1. Invalid 2. Will be correctly processed by other modules/components processing swift before this component. It’s as simple as this – tester should design a SWIFT payment that 1. Is valid 2. Will pass all components, taking into account know issues, limitation and yet unfinished functionality.
So you challenge is not to do the equivalence partitioning of each value, but to discover a valid business data that would fulfill the condition from the requirements document or user story. If you haven’t tried doing it you will never believe how difficult it is.
Testing step 4: understanding the regression
The next challenge is during your regression testing. If your team is doing incremental development then your test data that we valid on last time could be invalid today, or could trigger completely different business case. So you can’t just register regression bug. You need revisit and understand once again the details of whole flow of data processing.
Testing step 5: Explaining what you have done
Your specifications, your user stories typically identify a single feature of the whole system. How does this feature translate and map into business cases that your customers operate with? That’s not uncommon that you will talk in completely different languages. They would talk about different markets (i.e have you tested the Japan market specifics), while you would talk about special business-rules, configurations, default values, etc. So how do you make sure:
Add a Comment