Initially I was not involved in this issue since it did not concern my specific area. I was not aware of the issue when it started. By the time it was brought to my attention, it had become a crisis.
I was a member of a team implementing ERP application for a Jewellery manufacturing company. It was a very complex implementation and we had totally customized the manufacturing process to meet the complex process and stringent security requirements of the customer.
From the time raw materials were consumed to the time the finished product was sent to the depot for Order fulfilment, we customized the entire process flow.
One element of the customization was a customized window to receive the finished product from the production process. The problem was that when the user was receiving the finished product, in certain cases, instead of the original receipt quantity, a different and significantly higher quantity was being received. The only pattern we observed was that the received quantity always ended up with 'zero',implying that the receipt quantity was a multiple of 10. For example, if the original quantity was 23, in some cases the receipt quantity was 230 and in some other cases it was 230000 !!
It was baffling...
Remember, this was jewellery industry. We were dealing with items of very high value. The explosion of finished goods quantity had serious consequences right from profitability and taxation to import restrictions...
Our team analyzed the issue in test instance. We could not find anything abnormal. The production output was being received in the right quantity. Other than suggesting the reversal of the differential quantity we could not suggest any solution.
However, the problem continued to exist in the production instance.
As soon as I got involved, the first thing did was to review my team on the analysis they had done. It was very clear that the team had analyzed the issue very thoroughly and had documented the test results. The had tried replicating the issue with different user logins, different items, both single and multiple line production orders and during different times of the day, including some testing done at night. They had also documented the test results very professionally. It was obvious that the team had done a great job of testing the issue.
Then I got down to testing various scenarios in the test instance. The idea was to replicate all the tests done by the team and see if I can get same results. First I asked and got a clone of the most recent application and database installed in the test instance. Then I, along with the team did a battery of tests. All of them were inconclusive and did not show any abnormality...
Yet problem was being reported from all parts of the country and multiple users.
Now it was time for visual inspection.
Since many users had raised the issue, I choose a user who was available in the Factory itself and sat next to him and asked him to do a production receipt.
It was the initial stages of production cut-over and we were facing significant performance issues. Every user had complained that the production instance was very, very slow. In fact the cursor to move from one field to another took anywhere between 30 seconds to three minutes.
I observed the user as he entered the production quantity. I saw that after entering the quantity, the user had pressed the 'Tab' key to move to the next field which was the 'Submit' button. Due to high transaction volumes by many users who had simultaneously logged in, the system was taking a long time to move to the the 'Submit' button. All this while the impatient user was keeping the palm of his hand on the mouse and impatiently clicking the mouse, apparently for some magic to happen.
Finally when the cursor moved to the next step, we checked the quantity. As expected the quantity had exploded.
Immediately I had a hunch. I asked the user to repeat the transaction again, but this time after entering the production quantity and tabbing out to the 'Submit' button, I asked him to click the mouse button only after the focus had shifted to the 'Submit' button. You could see the veins popping out of his forehead as he waited impatiently for the focus to shift to the 'Submit' button. The mouse was right there and you are not allowed to touch it. His right index finger was twitching as we waited.
When the focus shifted to 'Submit' button, I asked him to click the mouse button. When the process completed, we checked the quantity and the quantity had not changed.
Now, I asked him to repeat the process but this time click the mouse 2 times, one time after tabbing out of the quantity field and before the cursor focused on the 'Submit' button and once after the focus was shifted.. On checking we found that the receipt quantity had been multiplied by 10. When we repeated this controlled experiment again asking the user to click three times, two before focus and once after, the quantity had been multiplied by 100 (10X10). When we tried again, clicking four times (as expected, three before the focus and one after), the quantity had been multiplied by 1000 (10X10X10).
Every time the user was clicking the mouse after tabbing out into the 'Submit' button, the system was multiplying the receipt quantity by multiples of 10 for every mouse click after the first one.
This was happening only in production instance because that is where the performance issue was. Many impatient users were trying to receive finished products into the inventory so that they can process the sales !. We could not replicate this issue in Test Instance since there was no performance issue there.
Once we identified the Root Cause, the problem was quickly fixed.
Note: This also explains the power of visual observation. When you test on a test instance, you are not exactly replicating the issue. You are missing out a key element of the transaction, the User. And many issues are created by Users as the 'Curious Case of Flickering Fingers' demonstrate
No comments:
Post a Comment