IT Quality Assurance


IT Quality Assurance










PDF version of this report
You must have Adobe Acrobat reader to view, save, or print PDF files. The reader
is available for free
download.

IT Quality Assurance

by James G. Barr

Copyright 2021, Faulkner Information Services. All Rights
Reserved.

Docid: 00021021

Publication Date: 2104

Publication Type: TUTORIAL

Preview

IT Quality Assurance is the process of making sure that a new or
updated information technology (IT) component – software, hardware,
firmware, or service – meets or exceeds its stated requirements. The most
practiced form of IT Quality Assurance – and the primary subject of this report
– is Software Quality Assurance (SQA). SQA is an ongoing part of the software
development lifecycle process
that monitors software needs from development through deployment and even post-deployment.
Ensuring the quality of enterprise software is vital
since factors such as changes to a company’s information
infrastructure can adversely affect the operation of
previously-certified software.

Report Contents:

Executive Summary

[return to top of this report]

While quality is often portrayed as a complex subject, the concept is
relatively simple. In their landmark text, Quality Data Processing,
authors Claude W. Burrill and Leon W. Ellsworth define quality as "meeting
requirements."1

Related
Faulkner Reports
Total Quality Management
Tutorial
The ISO 9001 Quality Management Standard
Standard

Whether applied to a hardware product like a server, a software
product like an operating system or application, or a information
technology service like change management, quality is achieved when the
product or service meets or exceeds the requirements it was
designed to satisfy.

The Four Domains

The four domains of IT Quality Assurance are software, hardware, firmware,
and service. From an enterprise user, rather than IT provider, perspective, Software
Quality Assurance is the most practiced form of IT Quality
Assurance since software is the most volatile, i.e., most frequently modified,
component in a typical IT system or solution.

The enterprise IT and Quality Assurance (QA) departments are usually
responsible for installing, configuring, testing, and updating vendor-supplied
software, while enterprise Application Development and QA are normally tasked with
designing, developing, testing, deploying, and updating enterprise application
software.

Software Quality Assurance

In the case of a software program, quality requirements are normally grouped into
seven categories:

  1. Functionality – Does the software do
    what it’s supposed to do?
  2. Performance – Does the software operate
    with the required speed and capacity?
  3. Security – Does the software prevent
    the exposure of sensitive information?
  4. Interoperability – Does the software
    interact, as necessary, with other programs in the same information system
    environment?
  5. Compliance – Does the software operate
    in a manner consistent with enterprise policies and government regulations?
  6. Resilience – Does the software rebound
    in the presence of unexpected operating conditions, including intentional
    misuse?
  7. Maintainability – Can the software be readily
    revised (including automatically)?

Consideration of these categories is essential during software design to
ensure that all relevant requirements are identified and prioritized. This is the responsibility of the
software development team and, from an IT governance perspective, the enterprise
Software Quality Assurance team.

Software Quality Assurance (SQA) is the process of guaranteeing that new or
updated software meets its stated requirements. SQA is an ongoing part of the
software development lifecycle (SDLC) process that monitors software needs from
development through deployment and even post-deployment. Ensuring the quality of
enterprise software is vital since environmental factors like changes to a
company’s information infrastructure can adversely affect the operation of
previously-certified software.

The Software Quality Assurance process is normally overseen by a Software
Quality Assurance team, which may be part of the Information Technology
department but which, ideally, is attached to the enterprise Quality Assurance
department to guarantee independence.

While the SQA team has overall accountability for the SQA process, the
process is conducted by multiple parties, including:

  1. Software development personnel – Usually
    peer personnel since the people who developed the software in question may
    exhibit testing-related biases.
  2. SQA team personnel – Individuals trained in
    software testing.
  3. Enterprise employees – Especially if the
    employees are prospective users.
  4. Third-party specialists – For example,
    where security is a primary concern, hiring penetration testers may be
    indicated.
  5. Enterprise clients – Trusted clients who agree to try out pre-production
    software versions (a process referred to as beta testing).
  6. "Crowdsourced" testers – Crowdsourced testing is conducted by an
    adhoc community of software testers, assembled for the purpose of performing
    specific QA tests. Crowdsourced testing is particularly useful for complex
    development projects, where in-house resources are insufficient to exercise
    all possible use cases. Crowdsourced testers are often valued for their
    objective feedback since they are not enterprise employees or other
    insiders.

In addition to its regular responsibilities, the SQA team should stress
good programming practices, such as encouraging planners and programmers to:

  • Reduce code complexity, through the reuse of already vetted program
    modules, subroutines, and microservices.
  • Avoid errors – especially security errors – by monitoring reports of
    successful security exploits.
  • Creating a code base that may be readily revised as required.2

Hardware Quality Assurance

While often missing from their mission statements, enterprise IT and QA are
responsible for the quality
operation of enterprise mainframes, servers, storage systems, PCs, workstations,
network equipment, and other physical infrastructure.

As analyst Jesse St. Laurent observes, enterprises "must implement hardware
QA testing … to avoid outages." This obligation even extends to
software-defined data centers (SDDCs). "[Hardware] within a SDCC does not
always operate as intended; hardware components sometimes have bugs that can
only be detected through use. [However,] by implementing a comprehensive
hardware QA program, larger problems can be avoided, and software can be taught
to handle problems smarter as they arise."3

Firmware Quality Assurance

Mostly hidden from view, firmware is a type of "built-in" software "that is etched
directly into a piece of hardware."4 Firmware, or "microcode",
provides hardware with its most elemental intelligence, enabling, for example, a
PC motherboard to detect the PC hard drive.5

From an IT Quality Assurance standpoint, enterprise IT and QA should
proactively track vendor firmware updates and, if appropriate, apply such
updates precisely as directed.

Service Quality Assurance

In addition to assuring the quality of IT software, hardware, and firmware,
the IT Quality Assurance function should assure the quality of foundational IT systems management services, including, but not limited to:

  1. Change/Release Management
  2. Incident/Problem Management
  3. Configuration Management
  4. Capacity Planning
  5. Patch Management
  6. Identity/Access Management
  7. Security Management

One preferred approach is to embrace Total Quality Management (TQM).
Proponents of TQM assert that any process, service, or methodology can be
enhanced via a cycle of continuous improvement, iteratively performing
activities in five major areas:

  1. Focusing on customer needs;
  2. Developing and tapping employees’ potential;
  3. Involving everyone in efforts to find better protocols and procedures;
  4. Managing business processes; and
  5. Managing by using reliable data.6

The Evolution of SQA and SQA Best Practices

[return to top of this report]

Although the Software Quality Assurance process has existed formally or
informally since the first software was developed, SQA first received serious
attention as a systems management discipline in the 1970s when quality
evangelists like Philip Crosby and J.M. Juran demonstrated the economic
importance of quality in manufacturing and other industrial processes.

As an example of quality’s influence, shortly after General Electric acquired RCA in the mid-1980s, the company
established an independent and autonomous Quality Assurance department within
the combined RCA/GE corporate data center. GE’s goal was to leverage
quality assurance to eliminate or, at least, mitigate any
integration-related data processing problems.

Although the software development community adopted quality as a principal
priority, attention to quality was escalated in the early 1990s with the advent
of the Internet and e-commerce. Until then, most enterprise-developed
software was executed internally by enterprise employees. With e-commerce,
enterprise software was transformed. A new breed of customer-facing
applications were developed in which clients invoked enterprise
software directly and in real time.

As a result, whatever consideration enterprise developers may have received over issues related to poor design or poor coding (after all, their
customers were enterprise employees) was suddenly gone. Applications
needed to work right the first time. They also needed to work in an
Internet environment not designed with security in mind.

Over the past quarter century, the science of quality assurance has advanced
with the embrace of quality management philosophies and standards such as Six
Sigma, Total Quality Management (TQM), and ISO 9000. In addition to these
elements, the process of Software Quality Assurance has been enhanced
through the acceptance of best practices.

Best SQA Practices

  1. Limitations of Testing
    – All SQA analysts should be reminded of the inherent limitations of
    testing. Although integral to SQA, no amount of testing can guarantee that
    software is defect-free. There is no way to test all combinations
    of data inputs, scenarios, and preconditions within a piece of software.
    Remember, too, that running the same set of tests over and over will not reveal more issues.7
  2. Secure Testing – All software tests
    should comply with enterprise information security and privacy policies and standards; in
    particular, policies and standards pertaining to the protection of personally identifiable information (PII).
  3. Prior Approval
    – Except for emergency fixes, no software should be promoted to production
    level without prior SQA approval.
  4. Requirement-to-Test Correspondence –
    For each software requirement, there should be a corresponding test (or
    tests) to
    ensure the requirement has been met.
  5. Test Database –
    All software tests should be accumulated and indexed in a test database. Tests devised for one program may be applicable to other programs.
  6. End-to-End Software Testing and Beyond – Software should
    be tested
    against requirements during – and after – development. With few exceptions, software exists within a larger
    information infrastructure. Any planned or unplanned change to that
    infrastructure could adversely affect software operation. For example, an
    application designed for one model of iPhone might not work on a subsequent
    model. Worse still, the application might work but
    with subtle, hard-to-detect differences.
  7. Penetration Testing
    – To help alleviate security concerns, SQA should consider enlisting the aid
    of a security firm expert in penetration testing. Penetration testing, or pen testing, is an invasive process in which Software Quality
    Assurance
    engages an independent security consultant or firm to probe an enterprise’s
    software defenses. The purpose is to identify real world
    vulnerabilities using techniques that would likely be employed by known bad
    actors.
  8. Test Automation – To improve the
    quantity and quality of software testing, adopt advanced automation. "[Automation] minimizes the human effort required to efficiently run tests,
    reduces time-to-market and the cost of errors because the tests are
    performed up to 10 times faster when compared to manual testing [processes]."8
  9. Software Reversion – One often overlooked aspect of software
    deployment is software reversion. Once a software program is installed,
    can it be de-installed? Are there any negative consequences, such as lost
    or compromised data? While reversion may not be possible – or practical –
    for a major software change, such as an operating system upgrade, SQA should
    insist that developers create and test reversion protocols for most new
    software.
  10. Change Management and Communication
    – New or updated software should be deployed in accordance with enterprise
    change management policies and procedures. Except for emergency fixes,
    all affected parties should be alerted at least several days prior to
    software installation.
  11. SQA Documentation – Keep complete – readily auditable – records
    of all SQA activity.

COTS & SaaS Quality Assurance

Today, many, if not most, of the software programs executed by enterprise
users are developed by third-party firms. These extra-enterprise
applications fall into two broad categories:

  1. Locally-installed commercial off-the-shelf (COTS) software, and
  2. Cloud-accessed software as a service (SaaS) software.

Importantly, the same best quality assurance practices that apply to
home-grown software apply to COTS and SaaS software.

Although difficult to enforce, the SQA team should insist that COTS and
SaaS software undergo quality testing prior to production use. This
can be accomplished by establishing a "sandbox" or test environment where SQA members and prospective enterprise users can exercise the software in
relative safety.

Ideally, the final acquisition of COTS and SaaS software should be
conditioned on successful SQA team and user testing.

DevOps & SQA

Gaining increased traction among enterprise software developers is a concept
called "DevOps", which emphasizes the tight integration of software DEVelopment
and IT OPerationS. As depicted in Figure 1, the DevOps cycle relies, in large measure, on
SQA to enable a smooth, error-resistant circuit from Development to Operations and
back to Development.

Figure 1. SQA Enables Every Element of DevOps

Figure 1. SQA Enables Every Element of DevOps

Source: Pixabay

Mobile Application Security Assurance – SQA at Work

[return to top of this report]

In a connected world where enterprise software operates under constant threat
of viruses, worms, Trojans, and other malware, one of the principal
responsibilities of Software Quality
Assurance is software security. Owing to their growth and popularity,
mobile applications present a security danger.

Reacting to this risk, the US National Institute of Standards and Technology
(NIST) has developed a list of recommendations for vetting the security of
mobile apps.

To provide security assurance for mobile apps, enterprises should develop
security requirements that specify, for example, how data used by an app should
be secured, the environment in which an app will be deployed, and the acceptable
level of risk for an app. To help ensure that an app conforms to such
requirements, an app vetting process should be performed.9

Sample App Vetting Process10

Planning

  1. Perform a risk analysis
    to understand and document the potential security impact of mobile apps on the
    enterprise’s computing, networking, and data resources.
  2. Review and document the mobile device
    hardware and operating system security controls, for example, an encrypted file
    system, and identify which security and privacy requirements can be addressed by
    the mobile device itself.
  3. Review and document mobile enterprise
    security technologies, such as Mobile Device Management (MDM) solutions, and
    identify which security and privacy requirements can be addressed by these
    technologies.
  4. Review the
    enterprise’s mobile
    security architecture and understand what threats are mitigated through the
    technical and operational controls. Identify potential security and privacy
    risks that are not mitigated through these technical and operational controls.
  5. Develop enterprise app security
    requirements by identifying general and context-sensitive requirements.
  6. Educate enterprise staff on the
    limitations of app vetting and the value of human involvement in an app vetting
    process.
  7. Procure an adequate budget for
    performing an app vetting process.
  8. Hire personnel, particularly auditors,
    with appropriate expertise.

App Testing

  1. Review licensing agreements associated
    with analyzers and understand the security implications surrounding the
    integrity, intellectual property, and licensing issues when submitting an app to
    an analyzer. (Note: An analyzer is a service, tool, or human that tests
    an app for specific software vulnerabilities and may be internal or external
    to the enterprise.)
  2. Ensure that apps transmitted over the
    network use an encrypted channel (e.g., SSL) and that apps are stored on a
    secure machine at the analyzer’s location. In addition, ensure that only
    authorized users have access to that machine.
  3. Identify general app security
    requirements needed by the enterprise.
  4. Select appropriate testing tools and
    methodologies for determining the satisfaction or violation of general app
    security requirements.
  5. Ensure that app updates are tested.
  6. Leverage existing testing results where
    possible.

App Approval/Rejection

  1. Use analyzers that leverage a
    standardized reporting format or risk assessment methodology, or that provide
    intuitive and easy-to-interpret reports and risk assessments.
  2. Ensure sufficient training of auditors
    on both the enterprise’s security requirements and interpretation of analyzer
    reports and risk assessments. Use multiple auditors to increase likelihood of
    appropriate recommendations.
  3. Identify
    enterprise-specific vetting
    criteria necessary for vetting context-sensitive app security requirements.
  4. Monitor public databases, mailing lists,
    and other publicly available security vulnerability reporting repositories to
    keep abreast of new developments that may impact the security of mobile devices
    and mobile apps.

Recommendations

[return to top of this report]

Establish a Software Quality Assurance Organization

While many large enterprises can boast a Software Quality Assurance
organization, many small-to-medium-sized enterprises (SMEs) cannot claim a fully
functioning SQA group.

To help remedy this deficiency, enterprise quality advocates should remind
senior management of the financial and reputational risks associated with poor
quality, including poor software quality. These risks include:

  1. Lost business
  2. Lost customers
  3. Lost assets
  4. Fraud and theft
  5. Legal liability11

As the enterprise Risk department would likely concede, the cost of
establishing – and funding – a Software Quality Assurance organization would be
less than the expenses incurred from software that malfunctions or exposes the
enterprise to crippling malware.

In forming an SQA organization, observe the following:

  1. Make SQA part of the
    enterprise Quality Assurance or Risk departments.     Avoid
    Information Technology to prevent any conflicts of interest.
  2. Appoint a seasoned
    IT manager as the SQA lead.     This person should possess
    extensive software development experience. Obviously, any type of
    quality assurance experience would be a plus.
  3. Assemble a
    multi-disciplinary SQA team.     Include full-time members with
    software development experience, and part-time members from each of the
    enterprise’s major business functions, such as sales, finance, production,
    transportation, customer service, and legal. These part-time
    contributors will be able to evaluate emerging software from a business and
    customer perspective.
  4. Provide quality
    assurance skills training for all team members.     This training can
    be delivered in house or through online or offsite classes.
  5. Partner with the
    enterprise Project Management Office (PMO).     Ask the PMO to develop policies, protocols,
    and procedures that will allow SQA to align its operations with the
    enterprise software development process. Rather than being viewed as
    software development cops, this integration will encourage software
    developers to accept SQA members as valued members of the software
    development team.

Augment SQA with AI and Big Data

While the trend toward automated testing has allowed SQA to conduct more
tests more frequently, the coming integration with artificial intelligence,
specifically, machine learning, will allow the development of smarter tests, thereby improving both
the quantity and quality of exercises aimed at detecting and remediating
software flaws.

According to Altexsoft, a software engineering firm, "Although test automation solutions in the intelligence area are not
well-established yet, the shift towards more intelligence in testing is
inevitable. Cognitive automation, machine learning, self-remediation,
and predictive analysis are promising emerging techniques for the future
of test automation.

"Adopting smarter automation solutions will be essential for testing
the emerging intelligent applications and products in their rapidly
changing business environments."12

In addition to AI, the engineers at Altexsoft believe in the ability of Big
Data to facilitate future SQA testing. "[Managing] huge volumes of data that are
constantly uploaded on various platforms demands a unique approach for testing
as traditional techniques can no longer cope.

"Big Data testing is aimed at checking the quality of data and
verifying data processing. Data quality [checking] involves various
characteristics like conformity, accuracy, duplication, consistency,
validity, data completeness, etc. Data processing verification comprises
performance and functional testing. [Critically], Big Data testing demands a high
level of testing skills as the processing is very fast."13

Begin Implementing IT Quality Assurance

Once the principles and practices of Software Quality Assurance are firmed
ingrained, enterprise IT and QA should start layering in the other essential
elements of IT Quality Assurance: Hardware, Firmware, and Service.

To expedite
the process – and reduce any organizational resistance – utilize the existing
Software Quality Assurance organization and technical structure – with
appropriate modifications, of course – to deliver Hardware, Firmware, and
Service Quality Assurance.

Develop enterprise confidence in IT Quality
Assurance by tackling several high-value targets. For example:

  • Hardware: Test the responsiveness of enterprise PCs, which can
    diminish in the presence of new, memory-intensive applications. Add memory
    to revitalize any otherwise well-functioning units.
  • Firmware: Survey hardware vendors to identity any recommended
    firmware updates, and arrange for their installation.
  • Service: Test employees’ knowledge of security protocols,
    especially e-mail management and phishing avoidance. Conduct "security
    awareness" training sessions as necessary.

[return to top of this report]

References

1 Claude W. Burrill and Leon W. Ellsworth.
Quality Data
Processing: The Profit Potential for the 80s.
Tenafly, NJ: Burrill-Ellsworth
Associates, Inc. 1982:12.

2

Bennett Garner. “What Is Software Quality Assurance?”
Intertech, Inc. November
7, 2017.

3 Jesse St. Laurent. "The SDDC: Hardware Quality Assurance Matters."
Informa
PLC. October 30, 2015.

4 Kuntal Chakraborty (reviewer). "Firmware."
Techopedia Inc. February 25,
2021.

5 Codrut Neagu. "What Is Firmware? What Does Firmware
Do?"
DigitalCitizen.life. January 29, 2021.

6 For more information, see the Faulkner tutorial
entitled "Total Quality Management."

7 "Quality Assurance, Quality Control and Testing — The Basics of Software
Quality Management." AltexSoft. 2021.

8 Ibid.

9
Steve Quirolgico, Jeffrey Voas, Tom Karygiannis, Christoph Michael, and Karen Scarfone. NIST Special Publication 800-163:
"Vetting the Security of Mobile Applications."
US National Institute of Standards and Technology.
January 2015:24-5.

10 Ibid. p.24-5.

11 Claude W. Burrill and Leon W.
Ellsworth. Quality Data Processing: The Profit Potential for the 80s.
Tenafly, NJ: Burrill-Ellsworth Associates, Inc. 1982:39-44.

12 "Quality Assurance, Quality Control and Testing — The Basics of Software
Quality Management." AltexSoft. 2021.

13 Ibid.

About the Author

[return to top of this report]

James G. Barr is a leading business continuity analyst and

business writer with more than 30 years’ IT experience. A member of

"Who’s Who in Finance and Industry," Mr. Barr has designed,

developed, and deployed business continuity plans for a number of Fortune

500 firms. He is the author of several books, including How to

Succeed in Business BY Really Trying, a member of Faulkner’s Advisory

Panel, and a senior editor for Faulkner’s Security Management

Practices. Mr. Barr can be reached via e-mail at jgbarr@faulkner.com.

[return to top of this report]