PDF version of this report
You must have Adobe Acrobat reader to view, save, or print PDF files. The reader
is available for free
download.
IT Quality Assurance
Copyright 2021, Faulkner Information Services. All Rights
Reserved.
Docid: 00021021
Publication Date: 2104
Publication Type: TUTORIAL
Preview
IT Quality Assurance is the process of making sure that a new or
updated information technology (IT) component – software, hardware,
firmware, or service – meets or exceeds its stated requirements. The most
practiced form of IT Quality Assurance – and the primary subject of this report
– is Software Quality Assurance (SQA). SQA is an ongoing part of the software
development lifecycle process
that monitors software needs from development through deployment and even post-deployment.
Ensuring the quality of enterprise software is vital
since factors such as changes to a company’s information
infrastructure can adversely affect the operation of
previously-certified software.
Report Contents:
- Executive Summary
- Related Reports
- The Evolution of SQA and SQA Best
Practices - Mobile Application Security Assurance – SQA at Work
- Recommendations
- Web Links
Executive Summary
[return to top of this report]
While quality is often portrayed as a complex subject, the concept is
relatively simple. In their landmark text, Quality Data Processing,
authors Claude W. Burrill and Leon W. Ellsworth define quality as "meeting
requirements."1
Related Faulkner Reports |
Total Quality Management Tutorial |
The ISO 9001 Quality Management Standard Standard |
Whether applied to a hardware product like a server, a software
product like an operating system or application, or a information
technology service like change management, quality is achieved when the
product or service meets or exceeds the requirements it was
designed to satisfy.
The Four Domains
The four domains of IT Quality Assurance are software, hardware, firmware,
and service. From an enterprise user, rather than IT provider, perspective, Software
Quality Assurance is the most practiced form of IT Quality
Assurance since software is the most volatile, i.e., most frequently modified,
component in a typical IT system or solution.
The enterprise IT and Quality Assurance (QA) departments are usually
responsible for installing, configuring, testing, and updating vendor-supplied
software, while enterprise Application Development and QA are normally tasked with
designing, developing, testing, deploying, and updating enterprise application
software.
Software Quality Assurance
In the case of a software program, quality requirements are normally grouped into
seven categories:
- Functionality – Does the software do
what it’s supposed to do? - Performance – Does the software operate
with the required speed and capacity? - Security – Does the software prevent
the exposure of sensitive information? - Interoperability – Does the software
interact, as necessary, with other programs in the same information system
environment? - Compliance – Does the software operate
in a manner consistent with enterprise policies and government regulations? - Resilience – Does the software rebound
in the presence of unexpected operating conditions, including intentional
misuse? - Maintainability – Can the software be readily
revised (including automatically)?
Consideration of these categories is essential during software design to
ensure that all relevant requirements are identified and prioritized. This is the responsibility of the
software development team and, from an IT governance perspective, the enterprise
Software Quality Assurance team.
Software Quality Assurance (SQA) is the process of guaranteeing that new or
updated software meets its stated requirements. SQA is an ongoing part of the
software development lifecycle (SDLC) process that monitors software needs from
development through deployment and even post-deployment. Ensuring the quality of
enterprise software is vital since environmental factors like changes to a
company’s information infrastructure can adversely affect the operation of
previously-certified software.
The Software Quality Assurance process is normally overseen by a Software
Quality Assurance team, which may be part of the Information Technology
department but which, ideally, is attached to the enterprise Quality Assurance
department to guarantee independence.
While the SQA team has overall accountability for the SQA process, the
process is conducted by multiple parties, including:
- Software development personnel – Usually
peer personnel since the people who developed the software in question may
exhibit testing-related biases. - SQA team personnel – Individuals trained in
software testing. - Enterprise employees – Especially if the
employees are prospective users. - Third-party specialists – For example,
where security is a primary concern, hiring penetration testers may be
indicated. - Enterprise clients – Trusted clients who agree to try out pre-production
software versions (a process referred to as beta testing). - "Crowdsourced" testers – Crowdsourced testing is conducted by an
adhoc community of software testers, assembled for the purpose of performing
specific QA tests. Crowdsourced testing is particularly useful for complex
development projects, where in-house resources are insufficient to exercise
all possible use cases. Crowdsourced testers are often valued for their
objective feedback since they are not enterprise employees or other
insiders.
In addition to its regular responsibilities, the SQA team should stress
good programming practices, such as encouraging planners and programmers to:
- Reduce code complexity, through the reuse of already vetted program
modules, subroutines, and microservices. - Avoid errors – especially security errors – by monitoring reports of
successful security exploits. - Creating a code base that may be readily revised as required.2
Hardware Quality Assurance
While often missing from their mission statements, enterprise IT and QA are
responsible for the quality
operation of enterprise mainframes, servers, storage systems, PCs, workstations,
network equipment, and other physical infrastructure.
As analyst Jesse St. Laurent observes, enterprises "must implement hardware
QA testing … to avoid outages." This obligation even extends to
software-defined data centers (SDDCs). "[Hardware] within a SDCC does not
always operate as intended; hardware components sometimes have bugs that can
only be detected through use. [However,] by implementing a comprehensive
hardware QA program, larger problems can be avoided, and software can be taught
to handle problems smarter as they arise."3
Firmware Quality Assurance
Mostly hidden from view, firmware is a type of "built-in" software "that is etched
directly into a piece of hardware."4 Firmware, or "microcode",
provides hardware with its most elemental intelligence, enabling, for example, a
PC motherboard to detect the PC hard drive.5
From an IT Quality Assurance standpoint, enterprise IT and QA should
proactively track vendor firmware updates and, if appropriate, apply such
updates precisely as directed.
Service Quality Assurance
In addition to assuring the quality of IT software, hardware, and firmware,
the IT Quality Assurance function should assure the quality of foundational IT systems management services, including, but not limited to:
- Change/Release Management
- Incident/Problem Management
- Configuration Management
- Capacity Planning
- Patch Management
- Identity/Access Management
- Security Management
One preferred approach is to embrace Total Quality Management (TQM).
Proponents of TQM assert that any process, service, or methodology can be
enhanced via a cycle of continuous improvement, iteratively performing
activities in five major areas:
- Focusing on customer needs;
- Developing and tapping employees’ potential;
- Involving everyone in efforts to find better protocols and procedures;
- Managing business processes; and
- Managing by using reliable data.6
The Evolution of SQA and SQA Best Practices
[return to top of this report]
Although the Software Quality Assurance process has existed formally or
informally since the first software was developed, SQA first received serious
attention as a systems management discipline in the 1970s when quality
evangelists like Philip Crosby and J.M. Juran demonstrated the economic
importance of quality in manufacturing and other industrial processes.
As an example of quality’s influence, shortly after General Electric acquired RCA in the mid-1980s, the company
established an independent and autonomous Quality Assurance department within
the combined RCA/GE corporate data center. GE’s goal was to leverage
quality assurance to eliminate or, at least, mitigate any
integration-related data processing problems.
Although the software development community adopted quality as a principal
priority, attention to quality was escalated in the early 1990s with the advent
of the Internet and e-commerce. Until then, most enterprise-developed
software was executed internally by enterprise employees. With e-commerce,
enterprise software was transformed. A new breed of customer-facing
applications were developed in which clients invoked enterprise
software directly and in real time.
As a result, whatever consideration enterprise developers may have received over issues related to poor design or poor coding (after all, their
customers were enterprise employees) was suddenly gone. Applications
needed to work right the first time. They also needed to work in an
Internet environment not designed with security in mind.
Over the past quarter century, the science of quality assurance has advanced
with the embrace of quality management philosophies and standards such as Six
Sigma, Total Quality Management (TQM), and ISO 9000. In addition to these
elements, the process of Software Quality Assurance has been enhanced
through the acceptance of best practices.
Best SQA Practices
-
Limitations of Testing
– All SQA analysts should be reminded of the inherent limitations of
testing. Although integral to SQA, no amount of testing can guarantee that
software is defect-free. There is no way to test all combinations
of data inputs, scenarios, and preconditions within a piece of software.
Remember, too, that running the same set of tests over and over will not reveal more issues.7 -
Secure Testing – All software tests
should comply with enterprise information security and privacy policies and standards; in
particular, policies and standards pertaining to the protection of personally identifiable information (PII). -
Prior Approval
– Except for emergency fixes, no software should be promoted to production
level without prior SQA approval. -
Requirement-to-Test Correspondence –
For each software requirement, there should be a corresponding test (or
tests) to
ensure the requirement has been met. -
Test Database –
All software tests should be accumulated and indexed in a test database. Tests devised for one program may be applicable to other programs. -
End-to-End Software Testing and Beyond – Software should
be tested
against requirements during – and after – development. With few exceptions, software exists within a larger
information infrastructure. Any planned or unplanned change to that
infrastructure could adversely affect software operation. For example, an
application designed for one model of iPhone might not work on a subsequent
model. Worse still, the application might work but
with subtle, hard-to-detect differences. -
Penetration Testing
– To help alleviate security concerns, SQA should consider enlisting the aid
of a security firm expert in penetration testing. Penetration testing, or pen testing, is an invasive process in which Software Quality
Assurance
engages an independent security consultant or firm to probe an enterprise’s
software defenses. The purpose is to identify real world
vulnerabilities using techniques that would likely be employed by known bad
actors. -
Test Automation – To improve the
quantity and quality of software testing, adopt advanced automation. "[Automation] minimizes the human effort required to efficiently run tests,
reduces time-to-market and the cost of errors because the tests are
performed up to 10 times faster when compared to manual testing [processes]."8 - Software Reversion – One often overlooked aspect of software
deployment is software reversion. Once a software program is installed,
can it be de-installed? Are there any negative consequences, such as lost
or compromised data? While reversion may not be possible – or practical –
for a major software change, such as an operating system upgrade, SQA should
insist that developers create and test reversion protocols for most new
software. -
Change Management and Communication
– New or updated software should be deployed in accordance with enterprise
change management policies and procedures. Except for emergency fixes,
all affected parties should be alerted at least several days prior to
software installation. - SQA Documentation – Keep complete – readily auditable – records
of all SQA activity.
COTS & SaaS Quality Assurance
Today, many, if not most, of the software programs executed by enterprise
users are developed by third-party firms. These extra-enterprise
applications fall into two broad categories:
- Locally-installed commercial off-the-shelf (COTS) software, and
- Cloud-accessed software as a service (SaaS) software.
Importantly, the same best quality assurance practices that apply to
home-grown software apply to COTS and SaaS software.
Although difficult to enforce, the SQA team should insist that COTS and
SaaS software undergo quality testing prior to production use. This
can be accomplished by establishing a "sandbox" or test environment where SQA members and prospective enterprise users can exercise the software in
relative safety.
Ideally, the final acquisition of COTS and SaaS software should be
conditioned on successful SQA team and user testing.
DevOps & SQA
Gaining increased traction among enterprise software developers is a concept
called "DevOps", which emphasizes the tight integration of software DEVelopment
and IT OPerationS. As depicted in Figure 1, the DevOps cycle relies, in large measure, on
SQA to enable a smooth, error-resistant circuit from Development to Operations and
back to Development.
Figure 1. SQA Enables Every Element of DevOps
Source: Pixabay
Mobile Application Security Assurance – SQA at Work
[return to top of this report]
In a connected world where enterprise software operates under constant threat
of viruses, worms, Trojans, and other malware, one of the principal
responsibilities of Software Quality
Assurance is software security. Owing to their growth and popularity,
mobile applications present a security danger.
Reacting to this risk, the US National Institute of Standards and Technology
(NIST) has developed a list of recommendations for vetting the security of
mobile apps.
To provide security assurance for mobile apps, enterprises should develop
security requirements that specify, for example, how data used by an app should
be secured, the environment in which an app will be deployed, and the acceptable
level of risk for an app. To help ensure that an app conforms to such
requirements, an app vetting process should be performed.9
Sample App Vetting Process10
Planning
- Perform a risk analysis
to understand and document the potential security impact of mobile apps on the
enterprise’s computing, networking, and data resources. - Review and document the mobile device
hardware and operating system security controls, for example, an encrypted file
system, and identify which security and privacy requirements can be addressed by
the mobile device itself. - Review and document mobile enterprise
security technologies, such as Mobile Device Management (MDM) solutions, and
identify which security and privacy requirements can be addressed by these
technologies. - Review the
enterprise’s mobile
security architecture and understand what threats are mitigated through the
technical and operational controls. Identify potential security and privacy
risks that are not mitigated through these technical and operational controls. - Develop enterprise app security
requirements by identifying general and context-sensitive requirements. - Educate enterprise staff on the
limitations of app vetting and the value of human involvement in an app vetting
process. - Procure an adequate budget for
performing an app vetting process. - Hire personnel, particularly auditors,
with appropriate expertise.
App Testing
- Review licensing agreements associated
with analyzers and understand the security implications surrounding the
integrity, intellectual property, and licensing issues when submitting an app to
an analyzer. (Note: An analyzer is a service, tool, or human that tests
an app for specific software vulnerabilities and may be internal or external
to the enterprise.) - Ensure that apps transmitted over the
network use an encrypted channel (e.g., SSL) and that apps are stored on a
secure machine at the analyzer’s location. In addition, ensure that only
authorized users have access to that machine. - Identify general app security
requirements needed by the enterprise. - Select appropriate testing tools and
methodologies for determining the satisfaction or violation of general app
security requirements. - Ensure that app updates are tested.
- Leverage existing testing results where
possible.
App Approval/Rejection
- Use analyzers that leverage a
standardized reporting format or risk assessment methodology, or that provide
intuitive and easy-to-interpret reports and risk assessments. - Ensure sufficient training of auditors
on both the enterprise’s security requirements and interpretation of analyzer
reports and risk assessments. Use multiple auditors to increase likelihood of
appropriate recommendations. - Identify
enterprise-specific vetting
criteria necessary for vetting context-sensitive app security requirements. - Monitor public databases, mailing lists,
and other publicly available security vulnerability reporting repositories to
keep abreast of new developments that may impact the security of mobile devices
and mobile apps.
Recommendations
[return to top of this report]
Establish a Software Quality Assurance Organization
While many large enterprises can boast a Software Quality Assurance
organization, many small-to-medium-sized enterprises (SMEs) cannot claim a fully
functioning SQA group.
To help remedy this deficiency, enterprise quality advocates should remind
senior management of the financial and reputational risks associated with poor
quality, including poor software quality. These risks include:
- Lost business
- Lost customers
- Lost assets
- Fraud and theft
- Legal liability11
As the enterprise Risk department would likely concede, the cost of
establishing – and funding – a Software Quality Assurance organization would be
less than the expenses incurred from software that malfunctions or exposes the
enterprise to crippling malware.
In forming an SQA organization, observe the following:
-
Make SQA part of the
enterprise Quality Assurance or Risk departments. Avoid
Information Technology to prevent any conflicts of interest. -
Appoint a seasoned
IT manager as the SQA lead. This person should possess
extensive software development experience. Obviously, any type of
quality assurance experience would be a plus. -
Assemble a
multi-disciplinary SQA team. Include full-time members with
software development experience, and part-time members from each of the
enterprise’s major business functions, such as sales, finance, production,
transportation, customer service, and legal. These part-time
contributors will be able to evaluate emerging software from a business and
customer perspective. -
Provide quality
assurance skills training for all team members. This training can
be delivered in house or through online or offsite classes. - Partner with the
enterprise Project Management Office (PMO). Ask the PMO to develop policies, protocols,
and procedures that will allow SQA to align its operations with the
enterprise software development process. Rather than being viewed as
software development cops, this integration will encourage software
developers to accept SQA members as valued members of the software
development team.
Augment SQA with AI and Big Data
While the trend toward automated testing has allowed SQA to conduct more
tests more frequently, the coming integration with artificial intelligence,
specifically, machine learning, will allow the development of smarter tests, thereby improving both
the quantity and quality of exercises aimed at detecting and remediating
software flaws.
According to Altexsoft, a software engineering firm, "Although test automation solutions in the intelligence area are not
well-established yet, the shift towards more intelligence in testing is
inevitable. Cognitive automation, machine learning, self-remediation,
and predictive analysis are promising emerging techniques for the future
of test automation.
"Adopting smarter automation solutions will be essential for testing
the emerging intelligent applications and products in their rapidly
changing business environments."12
In addition to AI, the engineers at Altexsoft believe in the ability of Big
Data to facilitate future SQA testing. "[Managing] huge volumes of data that are
constantly uploaded on various platforms demands a unique approach for testing
as traditional techniques can no longer cope.
"Big Data testing is aimed at checking the quality of data and
verifying data processing. Data quality [checking] involves various
characteristics like conformity, accuracy, duplication, consistency,
validity, data completeness, etc. Data processing verification comprises
performance and functional testing. [Critically], Big Data testing demands a high
level of testing skills as the processing is very fast."13
Begin Implementing IT Quality Assurance
Once the principles and practices of Software Quality Assurance are firmed
ingrained, enterprise IT and QA should start layering in the other essential
elements of IT Quality Assurance: Hardware, Firmware, and Service.
To expedite
the process – and reduce any organizational resistance – utilize the existing
Software Quality Assurance organization and technical structure – with
appropriate modifications, of course – to deliver Hardware, Firmware, and
Service Quality Assurance.
Develop enterprise confidence in IT Quality
Assurance by tackling several high-value targets. For example:
- Hardware: Test the responsiveness of enterprise PCs, which can
diminish in the presence of new, memory-intensive applications. Add memory
to revitalize any otherwise well-functioning units. - Firmware: Survey hardware vendors to identity any recommended
firmware updates, and arrange for their installation. - Service: Test employees’ knowledge of security protocols,
especially e-mail management and phishing avoidance. Conduct "security
awareness" training sessions as necessary.
Web Links
[return to top of this report]
- International Organization for Standardization (ISO): http://www.iso.org/
- US National Institute of Standards and Technology (NIST):
http://www.nist.gov/
References
1 Claude W. Burrill and Leon W. Ellsworth.
Quality Data
Processing: The Profit Potential for the 80s.
Tenafly, NJ: Burrill-Ellsworth
Associates, Inc. 1982:12.
2
Bennett Garner. “What Is Software Quality Assurance?”
Intertech, Inc. November
7, 2017.
3 Jesse St. Laurent. "The SDDC: Hardware Quality Assurance Matters."
Informa
PLC. October 30, 2015.
4 Kuntal Chakraborty (reviewer). "Firmware."
Techopedia Inc. February 25,
2021.
5 Codrut Neagu. "What Is Firmware? What Does Firmware
Do?"
DigitalCitizen.life. January 29, 2021.
6 For more information, see the Faulkner tutorial
entitled "Total Quality Management."
7 "Quality Assurance, Quality Control and Testing — The Basics of Software
Quality Management." AltexSoft. 2021.
8 Ibid.
9
Steve Quirolgico, Jeffrey Voas, Tom Karygiannis, Christoph Michael, and Karen Scarfone. NIST Special Publication 800-163:
"Vetting the Security of Mobile Applications."
US National Institute of Standards and Technology.
January 2015:24-5.
10 Ibid. p.24-5.
11 Claude W. Burrill and Leon W.
Ellsworth. Quality Data Processing: The Profit Potential for the 80s.
Tenafly, NJ: Burrill-Ellsworth Associates, Inc. 1982:39-44.
12 "Quality Assurance, Quality Control and Testing — The Basics of Software
Quality Management." AltexSoft. 2021.
13 Ibid.
About the Author
[return to top of this report]
James G. Barr is a leading business continuity analyst and
business writer with more than 30 years’ IT experience. A member of
"Who’s Who in Finance and Industry," Mr. Barr has designed,
developed, and deployed business continuity plans for a number of Fortune
500 firms. He is the author of several books, including How to
Succeed in Business BY Really Trying, a member of Faulkner’s Advisory
Panel, and a senior editor for Faulkner’s Security Management
Practices. Mr. Barr can be reached via e-mail at jgbarr@faulkner.com.
[return to top of this report]