英文介紹,The Cost of Bugs,NOTE,Customer Requirements,Specifications,Software Design Documents,Test Documents,Software Development Lifecycle Models,Waterfall Model,Spiral Model,Summary,軟體測試,軟體測試的狹義論和廣義論,軟體測試的辨證論,軟體測試的風險論,軟體測試的經濟學觀點,軟體測試的標準論,
英文介紹
The Cost of Bugs
As you will learn in Chapter 2, software doesn’t just magically appear there’s usually a planned, methodical development process used to create it.
Figure 1.2. The cost to fix bugs can increase dramatically over time.
Time When Bug Is Found
The costs are logarithmic that is, they increase tenfold as time increases. A bug found and fixed during the early stages when the specification is being written might cost next finds it, the cost could easily be thousands or even millions of dollars.
As an example of how this works, consider the Disney Lion King case discussed earlier. The root cause of the problem was that the software wouldn’t work on a very popular PC platform. If, in the early specification stage, someone had researched what PCs were popular and specified that the software needed to be designed and tested to work on those configurations, the cost of that effort would have been minimal. If that didn’t occur, a backup would have been for the software testers to collect samples of the popular PCs and verify the software on them. They would have found the bug, but it would have been more expensive to fix because the software would have to be debugged, fixed, and retested. The development team could have also sent out a preliminary version of the software to a small group of customers in what’s called a beta test. Those customers, chosen to represent the larger market, would have likely discovered the problem. As it turned out, however, the bug was completely missed until many thousands of CD-ROMs were created and purchased. Disney ended up paying for telephone customer support, product returns, replacement CD-ROMs, as well as another debug, fix, and test cycle. It’s very easy to burn up your entire product’s profit if serious bugs make it to customer.
What Exactly Does a Software Tester Do?
You’re now seen examples of really nasty bugs. You know what the definition of a bug is, and you know how costly they can be. By now it should be pretty evident what a tester’s goal is:
The goal of a software tester is to find to bugs:
You may run across product teams who want their testers to simply confirm that the software works, not to find bugs. Reread the case study about the Mars Polar Lander, and you’ll see why this is the wrong approach. If you’re only testing things that should work and setting up your tests so they’ll pass, you will miss the things that don’t work. You will miss the bugs.
If you’re missing bugs, you’re costing your project and your company money. As a software tester you shouldn’t be content at just finding bugs you should think about how to find them sooner in the development process, thus making them cheaper to fix.
The goal of a software tester is to find bugs and find them as early as possible.
But, finding bugs, even finding them early, isn’t enough. Remember the definition of a bug. You, the software tester, are the customer’s eyes, the first one to see the software. You speak for the customer and must seek perfection.
This goal of a software tester is to find bugs, find them as early as possible, and make sure they get fixed.
This final definition is very important. Commit it to money and refer back to it as you learn the testing techniques discussed throughout the rest of this book.
NOTE
It’s important to note that “fixing” a bug does not necessarily imply correcting the software. It could mean adding a comment in the user manual or providing special training to the customer. It could require changing the statistics that the marketing group advertise or even postponing the release of the buggy feature. You’ll learn throughout this book that although you’re seeking perfection and making sure that the bugs get fixed, that there are practical realities to software testing. Don’t get caught in the dangerous spiral of unattainable perfection.
What Makes a Good Software Tester?
In the movie star Trek II: the Wrath of Khan, Spock says, “As a matter of cosmic history, it has always been easier to destroy than to create.” At first glance, it may appear that a software tester’s job would be easier than a programmer’s. breaking code and finding bugs must surely be easier than writing the code in the first place. Surprisingly, it’s not. The methodical and disciplined approach to software testing that you’ll learn in this book requires the same hard work and dedication that programming does. It involves very similar skills, and although a software tester doesn’t necessarily need to be a full-fledged programmer, having that knowledge is a great benefit.
Today, most mature companies treat software testing as a technical engineering profession. They recognize that having trained software testers on their project teams and allowing them to apply their trade early in the development process allows them to build better quality software. Unfortunately, there are still a few companies that don’t appreciate the challenge of software testing and the value of well-done testing effort. In a free market society, these companies usually aren’t around for long because the customers speak with their wallets and choose not to buy their buggy products. A good test organization (or the lack of one) can make or break a company.
Here’s a list of traits that most software testers should have:
They are explorers. Software testers aren’t afraid to venture into unknown situations. They love to get a new piece of software, install it on their PC, and see what happens.
They are troubleshooters. Software testers are good at figuring out why something doesn’t work. They love puzzles.
They are relentless. Software testers keep trying. They may see a bug that quickly vanishes or is difficult to re-create. Rather than dismiss it as a fluke, they will try every way possible to find it.
They are creative. Testing the obvious isn’t sufficient for software testers. Their job is to think up creative and even off-the-wall approaches to find bugs.
They are (mellowed) perfectionists. They strive for perfection, but they know when it becomes unattainable and they’re okay with getting as close as they can.
They exercise good judgment. Software testers need to make decisions about what they will test, how long it will take, and if the problem they’re looking at is really a bug.
They are tactful and diplomatic. Software testers are always the bearers of news. They have to tell the programmers that their baby is ugly. Good software testers know how to do so tactfully and professionally and know how to work with programmers who aren’t always tactful and diplomatic.
They are persuasive. Bugs that testers find won’t always be viewed as severe enough to be fixed. Testers need to be good at making their points clear, demonstrating why the bug does indeed need to be fixed, and following through on making it happen.
In addition to these traits, having some education in software programming is a big bugs. As you’ll see in Chapter 6. “Examining the Code,” knowing how software is written can give you a different view of where bugs are found, thus making you a more efficient and effective tester. It can also help you develop the testing tools discussed in Chapter 15, “Automated Testing and Test Tools.”
Lastly, if you’re an expert in some non-computer field, your knowledge can be invaluable to a software team creating a new product. Software is being written to do just about everything today. Your knowledge of teaching, cooking, airplanes, carpentry, medicine, or whatever would be a tremendous help finding bugs in software for those areas.
Summary
Software testing is a critical job. With the size and complexity of today’s software, it’s imperative software testing be performed professionally and effectively. Too much is at risk. We don’t need more defective computer chips, crashed system, or stolen credit card numbers.
In the following chapters of Part I “The Big Picture,” you’ll learn more about the big picture of software development and how software testing fits in. This knowledge is critical to helping you apply the specific test techniques covered in the remainder of this book.
Quiz
These quiz questions are provided for your further understanding. See Appendix A, “Answers to Quiz Questions,” for the answers but don’t peek!
1: In the Year 2000 bug example, did Dave do anything wrong?
2: True or False: It’s important what term your company or team calls problem in its software.
3: What’s wrong with just testing that a program works as expected?
4: How much more does it cost to fix a bug found after the product is released than it does from very start of the project?
5: what’s the goal of a software tester?
6: True or False: A good tester relentlessly strives for perfection.
7: Give several reasons why the product specification is usually the largest source of bugs in a software product.
Chapter 2. The Software Development Process
IN THIS CHAPTER
Product Components
Software Project Staff
Software Development Lifecycle Models
To be an effective software tester, it’s important to have least a high-level understanding of the overall process used to develop software. If you write small programs as a student hobbyist, you’ll find methods you use are much different from what big companies use to develop software. The creation of a new software product may involve dozens, hundreds, even thousands of team members all playing different roles an working together under tight schedules. The specific of what these people do, how they interact, and how they make decisions are all part of the software development process.
The goal of this chapter isn’t to teach you everything about the software development process that would take an entire book! The goal is to give you can overview of the all the pieces that go into a software product and a look at a few of the common approaches in use today. With this knowledge you’ll have a better understand of how best to apply the software testing skills you learn in the later chapters of this book.
The highlights of this chapter include
What major components go into a software product
What different people and skills contribute to a software product
How software progress from an idea to a final product
Product Components
What exactly is a software product? Many of us think of it as simply a program that we download from the Internet or install from DVD that runs on our computer. That’s a pretty good description, but in reality, many hidden pieces go into making that software. There are also many pieces that “come in the box” that are often taken for granted or might even be ignored. Although it may be easy to forget about all those parts, as a software tester, you need to be aware of them, because they’re all testable pieces and all have bugs.
What Effort Goes Into a Software Product?
First, look at what effort goes into a software product. Figure 2.1 Identifies a few of the abstract pieces that you may not have considered.
Figure 2.1 A lot of hidden effort goes into a software product.
So what are all these things, besides the actual code, that get funneled into the software? At first glance they probably seem much less tangible than the program listing a programmer creates. And they definitely aren’t something that can be viewed directly from the product’s CD-ROM. But, to paraphrase a line from an old spaghetti sauce commercial, “they’re in there.” At least, they should be.
The term used in the software industry to describe a software product component that’s created and passed on to someone else is deliverable. The easiest way to explain what all deliverables are organize them into major categories.
Customer Requirements
Software is written to fulfill some need that a person or group of people has. Let’s call them the customer. To properly fill that need, the product development team must find out what the customer wants. Some teams simply guess, but most collect detailed information in the form of surveys, feedback from previous versions of the software, competitive product information, magazine reviews, focus groups, and numerous other methods, some formal, some not. All this information is then studied, condensed, and interpreted to decide exactly what features the software product should have.
Specifications
The result of the customer requirements studies is really just raw data. It doesn’t describe the proposed product, it just confirms whether it should (or shouldn’t) be created and what features the customers want. The specifications take all this information plus any unstated but mandatory requirements and truly define what the product will be, what it will do, and how it will look.
The format of specifications varies greatly. Some companies specially those developing products for the government, aerospace, financial, and medical industries use a very rigorous process with many checks and balance. The result is an extremely detailed and thorough specification that’s locked down, meaning that it can’t change except under very extreme conditions. Everyone on the development team knows exactly what they are creating.
The most of specifications varies greatly. Some companies especially those developing products for the government, aero space, financial, and medical industries use a very rigorous process with many checks and balances. The result is an extremely detailed and thorough specification that’s locked down, meaning that it can’t change except under very extreme conditions. Every on the development team knows exactly what they are creating.
There are development teams, usually ones creating software for less-critical applications, who produce specifications on cocktail napkins, if they create them at all. This has the distinct advantage of being very flexible, but there’s lot of risk that not everyone is “on the same page.” And, what the product finally becomes isn’t know until released.
Schedules
A key part of a software product is its schedule. As a project grows in size and complexity, with many pieces and many people contributing to the product, it becomes necessary to have some mechanism to track its progress. This could range from simple task lists to Gantt charts (see Figure 2.2) to detailed tracking of every minute task with project management software.
Figure 2.2. A Gantt chart is a bar that shows a project’s tasks against a horizontal timeline.
The goals of scheduling are to know which work has been completed, how much work is still left to do, and when it will be finished.
Software Design Documents
One common misconception is that when a programmer creates a program, he simply sits down and starts writing code. That may happen in some small, informal software shops, but for anything other than the smallest programs, there must be a design process to plan out how the software will be written. Think about this book, which required an outline before the first words were typed, or a building, which has blueprints drawn the first concrete is poured. The same planning should happen with software.
The documents that programmers create vary greatly depending on the company, the project, and the team, but their purpose is to plan and organize the code that is to be written.
Here is a list of a few common software design documents:
Architecture. A document that describes the overall design of the software, including descriptions of all the major pieces and how they interact with each other.
Data Flow Diagram. A formalized diagram that shows how data moves through a program. It’s sometimes referred to as a bubble chart because it’s drawn by circles and lines.
State Transition Diagram. Another formalized diagram that breaks the software into basic states, or conditions, and shows the means for moving from one state to the next.
Flowchart. The traditional means for pictorially describing a program’s logic. Flowcharting isn’t very popular today, but when it’s used, writing the program code from a detailed flowchart is a very simple process.
Commented Code. There’s an old saying that you may write code once, but it will be read by someone at least 10 times. Properly embedding useful comments in the software code itself is extremely important, so that programmers assigned to maintain the code can more easily figure out what it does and how.
Test Documents
Test documentation is discussed in detail in Chapters 1720 but is mentioned here because it’s integral to what makes up a software product. For the same reasons that programmers must plan and document their work, software testers must as well. It’s not unheard of for a software test team to create more deliverables than the programmers.
Here’s a list of the more important test deliverables:
The test plan describes the overall method to be used to verify that the software meets the product specification and the customer’s needs. It includes the quality objectives, resource needs, schedules, assignments, methods, and so forth.
Test cases list the specific items that will be tested and describe the detailed steps that will be followed to verify the software.
Bug reports describe the problems found as the test cases are followed. These could be done on paper but are often tracked in a database.
Test tools and automation are described in detail in Chapter 15, “Automated Testing and Test Tools.” If you team is using automated methods to test your software, the tools you use, either purchased or written in-house, must be documented.
Metrics, statistics, and summaries convey the progress being made as the test work progresses. They take the form of graphs, chart, and written reports.
What Parts Make Up a Software Product?
So far in this chapter you ‘re learned about the effort that goes into creating a software product. It’s also important to realize that when the product is ready to be boxed up and shipped out the door, it’s not just the code that gets delivered. Numerous supporting parts go along with it (see Figure 2.3). Since all these parts are seen or used by the customer, they need to be tested too.
Figure 2.3. The software CD-ROM is just one of the many pieces that make up a software product.
It’s unfortunate, but these components are often overlooked in the testing process. You’ve surely attempted to use a product’s built-in help file and found it to be not so helpful it or worse just plain wrong. Or, maybe you’ve checked the system requirements on a sticker on the side of a software box only find out after you bought it that the software didn’t work on your PC. These seem like simple things to test, but no one probably even gave them a second look before the product was okayed for release. You will.
Later in this book you’ll learn about these non-software pieces and how to properly test them. Until then, keep this list in mind as just a sampling of what more there is to a software product than just the code:
Help files User’s manual
Samples and examples Labels and stickers
Product support info Icons and art
Error messages Ads and marketing material
Setup and installation Readme file
Software Project Staff
Now that you know what goes into a software product and what ships with one, it’s time to learn about all the people who create software. Of course, this varies a great deal based on the company and the project, but for the most part the roles are the same, it’s just the titles that are different.
The following lists, in no particular order, the major players and what they do. The most common names are given, but expect variations and additions:
Project managers, program managers, or produces drive the project from beginning to end. They’re usually responsible for writing the product spec, managing the schedule, and making the critical decisions and trade-offs.
Architects or system engineers are the technical experts on the product team. They’re usually very experienced and therefore are qualified to design the overall systems architecture or design for the software. They work very closely with the programmers.
Programmers, developers, or coders design and write software and fix the bugs that are found. They work closely with the architects and project managers to create the software. Then, they work closely with the project managers and testers to get the bugs fixed.
Testers or QA (Quality Assurance) Staff are responsible for finding and reporting problems in the software product. They work very closely with all members of the team as they develop and run their tests, and report the problems they find. Chapter 21, “Software Quality Assurance,” thoroughly cover the differences between software testing and software quality assurance tasks.
Technical writers, user assistance, user education, manual writers, or illustrators create the paper and online documentation that comes with a software product.
Configuration management or builder handles the process of pulling together all the software written by the programmers and all the documentation created by the writers and putting it together into a single package.
As you can see, several groups of people contribute to a software product. On large teams there may be dozens or hundreds working together. To successfully communicate and organize their approach, they need a plan, a method for getting from point A to point B. that’s what the next section is about.
Software Development Lifecycle Models
A running joke in the computer industry is that three things should never be seen in the process of being created: laws, sausage, and software. Their creation process is so messy and disgusting that it’s best to just wait and see the final result. That may or may not be totally true, but with most old sayings, there is a grain of truth behind the words. Some software is developed with the rigor and discipline of a fine craftsman, some software with tightly controlled chaos, and other software is stuck together with duct tape and chewing gum. Usually, in the end, it’s apparent to the customer what process was used. The process used to create a software product from its initial conception to its public release is known as the software development lifecycle model.
As discussed previously, there are many different methods that can be used for developing software, and no model is necessarily the best for a particular project. There are four frequently used models, with most others just variations of these:
Big-Bang
Code-and-Fix
Waterfall
Spiral
Each model has its advantages and disadvantages. As a tester, you’ll likely encounter them all and will need to tailor your test approach to fit the model being used for your current project. Refer to these model description as you read the rest of this book and think about how you would apply the various testing techniques you learn under each of them.
Big-Bang Model
One theory of the creation of the universe is the big-bang theory. It states that billions of years ago, the universe was created in a single huge explosion of nearly infinite energy. Everything that exists is the result of energy and matter lining up to produce this book, DVDs, and Bill Gates. If the atoms didn’t line up just right, these things might all be just quivering masses of goop.
The big-bang model for software development shown in Figure 2.4 follows much the same principle. A huge amount of matter (people and money) is put together, a lot of energy is expended often violently and out comes the perfect software product…or it doesn’t.
Figure 2.4. The big-bang model is by far the simplest method of software development.
The beauty of the big-bang method is that it’s simple. There is little if any planning, scheduling, or formal development process. All the effort is spent developing the software and writing the code. It’s process that is used if the product requirements aren’t well understood and the final release date is completely flexible. It’s also important to have very flexible customers, too, because they won’t know what they’re getting until the very end.
Notice that testing isn’t shown in Figure 2.4. In most cases, there is little to no formal testing done under the big-bang model. If testing does occur, it’s squeezed in just before the product is released. It’s a mystery why testing is sometimes inserted into this model, but it’s probably to make everyone feel good that some testing was performed.
If you are called in to test a product under the big-bang model, you have both an easy and a difficult task. Because the software is already complete, you have the perfect specification the product itself. And, because it’s impossible to go back and fix things that are broken, your job is really just to report what you find so the customers can be told about the problems.
The downside is that, in the eyes of project management, the product is ready to go, so your work is holding up delivery to the customer. The longer you take to do your job and the more bugs you find, the more contentious the situation will become. Try to stay away from testing in this model.
Code-and-Fix Model
The code-and-fix model shown in Figure 2.5 is usually the one that project teams fall into by default if they don’t consciously attempt to use something else. It’s a step up, procedurally, from the big-bang model, in that it at least requires some idea of what the product requirements are.
Figure 2.5. The code-and-fix model repeats until someone gives up.
A wise man once said, “there’s never time to do it right, but there’s always time to do it over.” That pretty much sums up this model. A team using this approach usually starts with a rough idea of what they want, does some simple design, and then processed into a repeating cycle of coding, testing, and fixing bugs. At some point they decide that enough is enough and release the product.
As there’s very little overhead for planning and documenting a project team can show results immediately. For this reason, the code-and-fix model works very well for small projects intended to be created quickly and then thrown out shortly after they’re done, such as prototypes and demos. Even so, code-and-fix has been used on many large and well-known software products. If your word processor or spreadsheet software has lots of little bugs or it just doesn’t seem quite finished, it was likely created with the code-and-fix model.
Like the big-bang model, testing isn’t specifically called out in the code-and-fix model but does play a significant role between the coding and the fixing.
As a tester on a code-and-fix project, you need to be aware that you, along with the programmers, will be in a constant state of cycling. As often as every day you’ll be given new or updated releases of the software and will set off to test it. You’ll run your tests, report the bugs, and then get a new software release. You may not have finished testing the previous release when the new one arrives, and the new one may have new or changed features. Eventually, you’ll get a chance to test most of the features, find fewer and fewer bugs, and then someone (or the schedule) will decide that it’s time to release the product.
You will most likely encounter the code-and-fix model during your work as a software tester. It’s good introduction to software development and will help you appreciate the more formal methods.
Waterfall Model
The waterfall method is usually the first one taught in programming school. It’s been around forever. It’s simple, elegant, and makes sense. And, it can work well on the right project. Figure 2.6 shows the steps involved in this model.
Figure 2.6. The software development process flows from one step the next in the waterfall model.
A project using the waterfall model moves down a series of steps starting from an initial idea to a final product. At the end of each step, the project team holds a review to determine if they’re ready to move to the next step. If the project isn’t ready to progress, it stays at that level until it’s ready.
Notice three important things about the waterfall method:
There’s large emphasis on specifying what the product will be. Note that the development or coding phase is only a single block!
The steps are discrete; there’s no overlap.
There’s no way to back up. As soon as you’re on a step, you need to complete the tasks for that step and then move on you can’t go back. [1]
[1] Variations of the waterfall model loosen the rules a bit, allowing some overlap of the steps and the ability to back up one step if necessary.
This may sound very limiting, and it is, but it works well for projects with a well-understood product definition and a disciplined development staff. The goal is to work out all the unknowns and nail down all the details before the first line of code is written. The drawback is that in today’s fast moving culture, with products being developed on Internet time, by the time a software product it so carefully thought out and defined, the original reason for its being may have changed.
Form a testing perspective, the waterfall model offers one huge advantage over the other models presented so far. Everything is carefully and thoroughly specified. By the time the software is delivered to test group, every detail has been decided on, written down, and turned into software. From that, the test group can create an accurate plan and schedule. They know exactly what they’re testing, and there’s no question about whether something is a feature or a bug.
But, with this advantage, comes a large disadvantage. Because testing occurs only at the end, a fundamental problem could creep in early on and not be detected until days before the scheduled product release. Remember from Chapter 1, “Software Testing Background,” how the cost of bugs increases over time? What’s needed is model that folds the testing tasks in earlier to find problems before they become too costly.
Spiral Model
It’s not quite utopia, but the spiral model (see Figure 2.7) goes a long way in addressing many of the problems inherent with the other models while adding a few of its own nice touches.
Figure 2.7. The spiral model starts small and gradually expands as the project becomes better defined and gains stability.
The spiral model was introduced by Barry Boehm in 1986 in his Association for Computing Machinery (ACM) paper, “A Spiral of Software Development and Enhancement.” It’s used fairly often and has proven to be an effective approach to developing software.
The general idea behind the spiral model is that you don’t define everything in detail at the very beginning. You start small, define your important feature, try them out, get feedback from your customers, and then move on to the next level. You repeat this until you have your final product.
Each time around the spiral involves six steps:
1. Determine objectives, alternatives, and constraints.
2. Identify and resolve risks.
3. Evaluate alternatives.
4. Develop and test the current level.
5. Plan the next level.
6. Decide on the approach for the next level.
Built into the spiral mode is a bit of waterfall (the steps of analysis, design, develop, test), a bit of code-and-fix (each time around the spiral), and a bit of big-bang (look at it from the outside). Couple this with the lower costs of finding problems early, and you have a pretty good development model.
If you’re a tester, you’ll like this model. You’ll get a chance to influence the product early by being involved in the preliminary design phases. You’ll see where the project has come from and where it’s going. And, at the very end of the project, you won’t feel as rushed to perform all your testing at the last minute. You’re been testing all along, so the last push should only be a validation that everything is okay.
Summary
You now have an understanding of how software products are create both goes into them and the processes used to put them together. As you can see, there’s no definitive approach. The four models presented here are just examples. There are many others and lot of variations of these. Each company, each project, and each team will choose what works for them. Sometimes they will choose right, sometimes they will choose wrong. Your job as a software tester is to work the best you can in the development model you’re in, applying the testing skills you learn in the rest of this book to create the best possible.
Quiz
These quiz questions are provided for your further understanding. See Appendix A, “Answers to Quiz Questions,” for the answers but don’t peek!
軟體測試
在G J Myers的經典著作《軟體測試之藝術》（The Art of Software Testing）中，給出了軟體測試（Software Testing）的定義：“程式測試是為了發現錯誤而執行程式的過程”。這個定義，被業界所認可，經常被引用。除此之外，G J Myers還給出了與測試相關的三個重要觀點，那就是：
