By Douglas C. SchmidtPrincipal Researcher While agile methods have become popular in commercial software development organizations, the engineering disciplines needed to apply agility to mission-critical, software-reliant systems are not as well defined or practiced. To help bridge this gap, the SEI recently hosted the Agile Research Forum. The event brought together researchers and practitioners from around the world to discuss when and how to best apply agile methods in mission-critical environments found in government and many industries. This blog posting, the fourth installment in a multi-part series highlighting research presented during the forum, summarizes a talk by James Over, manager of the Team Software Process (TSP) initiative, who advocated the building of self-managed teams, planning and measuring project process, designing before building, and making quality the top priority, among other principles associated with applying agile methods at-scale. Over’s Talk on Balancing Agility and Discipline In his opening comments to the audience, Over shared his views on agility and discipline and stressed the importance of finding a balance between the two. Over said that his presentation and research on combining agility and discipline is based on his work with software teams and software projects, although it is applicable to other fields. One of the reasons that agile methods are so popular today is their ability to respond to change. As evidence of this popularity, Over pointed out that about 40 percent of software developers use one or more agile methods, compared with 13 percent that only use traditional methods, according to a 2010 Dr. Dobbs survey on trends among global developers. One reason that agile methods have become so popular is that the pace of change is accelerating. Organizations are seeking solutions that will allow them to become more responsive to change. Agile methods provide some key parts of that capability, Over said.Balance is key for organizations seeking to implement agile methods. While organizations work to improve agility, they must do so in a disciplined way.  Discipline is particularly important for organizations like DoD acquisition programs and other federal agencies developing large, mission-critical software-reliant systems at scale. It’s important for organizations to understand what is meant by agility. While several definitions exist, Over stated one he likes: Agility is responding rapidly and efficiently to change with consistency. An agile business, he told the audience, should be able to respond quickly to change, make decisions quickly, and respond quickly to customers’ needs every single time, not just occasionally. What Does Agile Look Like Many software organizations claim to be agile because they follow agile principles, but they often lack an understanding of what it means to be agile. To achieve a greater understanding of agile methods, Over recommended that organizations measure their agility and evaluate their success along the following factors: Response time. Organizations should assess how quickly they respond to a customer’s needs. What sort of experiences are users having with the software they are producing in terms of response time? Efficiency. Organizations should consider their ability to deliver software. Can they produce projects with the desired balance between cost and quality? Are processes performing efficiently? Can organizations respond to change quickly and efficiently or do costs balloon out of control when changes are made to the content or requirements of the system? Consistency. Does every customer have the same experience? When customers interact with an organization, are they always seeing the same response time, the same types of efficiency, and the same types of behavior on each application? Impediments to Agility in Software In 2010, the authors of the Agile Manifesto reunited to hold a retrospective on Agile. Examining the state of the practice in the last decade, the authors identified 10 impediments to achieving agility. From that list, Over identified the following five impediments that he deemed critical for organizations to overcome when they seek to making agile methods work in practice: Lack of a ready backlog, which is the list of features that developers prioritize to build the software, is a serious concern. The manifesto authors found that 80 percent of teams had a ready backlog, but within that backlog, only 10 percent of items were ready for implementation. As a consequence, delays would occur in projects because the backlog was not prepared. Lack of being "done" at the end of a sprint means that as the project is nearing the end of the sprint and declaring the sprint completed, some work items are being deferred or skipped, which often  causes delays. The most common cause is a poorly implemented practice or a deferred practice. For example, skipping unit testing can cause delays later in the project, as well as increase cost by at least a factor of 2. Lack of teamwork surfaces when the team fails to come together and operate as a team and instead focuses on its individual needs. In this type of setting, developers focus on their own user features or stories and don’t pay attention to what the team is doing. They attend their daily standups, but fail to report problems that might affect the rest of the project. When team gets to the end of a sprint and discovers that there is still a lot of work to do, therefore, sprint failure and project delay will result. Lack of good design stemming from organizational structures that affects the kind of designs that software teams produce.  For example, an inflexible hierarchical structure often results in inflexible hierarchical designs, which in turn can yield code that is brittle and hard to use, excessively expensive, and prone to high failure rates. Tolerating defects is an unfortunate reality for many teams as they near the end of a sprint. They are out of time and deferring defects to the end of a sprint. Teams in this situation often take a set of unit tests or other issues and defer them to later sprints. When issues are deferred (which is a form of technical debt, as discussed by Ipek Ozkaya in her Agile Research Forum presentation), the result is increased in cost and higher failure rates later in the lifecycle because the defects aren’t being addressed as they are discovered. Avoiding the Impediments by Balancing Agility and Discipline Over next identified work that he and other SEI researchers have conducted to remedy the impediments to agility described above. Over’s work with software teams has focused on identifying a set of principles that teams should adopt to improve agility and response time. Over highlighted the following five principles that help address impediments identified by the Agile community in their retrospective: Build high-performance teams. Software is knowledge work. Build self-managed teams that make their own plans, negotiate their commitments, and know the project status precisely. Plan every project. Never make a commitment without a plan. Use historical data. If the plan doesn’t fit the work, fix the plan. Change is not free. Use a measured process to track progress. To measure the work, define its measures. To measure the process, define its steps. If the process doesn’t fit the work, fix the process. Tracking progress via a measured process need not require a complicated solution given the appropriate set of tools and a codified method for applying the tools effectively. Design before you implement. Design is critical since it informs the software implementation. Good designs produce less code and simplify the downstream evolution of the code. Design what you know and explore the rest. When you know enough, finish the design and then implement it.   To be clear, this principle isn’t espousing "big upfront design" (BUFD). It is BUFD without the BUF, or just D. Produce enough design to build the implementation. If the design is still too challenging, explore the problem first via prototyping to reduce design risk. Applying this principle effectively depends on various factors, including the size and complexity of the application, as well as familiarity with the problem, domain, methods, and tools. Make quality software the top priority. Defects are inevitable. Poor quality wastes resources. The sooner a developer fixes a problem, the better.  As Jeff Sutherland stated in his retrospective, tolerating defects, ignoring pair programming or code review practices, and inadequate unit testing will substantially reduce team velocity. Continuous integration tools and automated testing are also important, but testing alone is insufficient because poor quality software will always cost more to produce because finding and fixing defects is still expensive, and the longer you wait to fix them the greater the cost of repair. In summary, Over told the audience that in working on 20 different projects in 13 organizations to implement these disciplined agile principles, SEI researchers found that organizations delivered more functionality than originally planned or finished ahead of schedule. Among the other benefits, Over stated, was that projects realized test costs of less than 7 percent of the total cost with an average cost of quality of only 17 percent. Also, the projects delivered software with an average of only six defects in 100,000 lines of new and modified code. Finally, discipline is the key to agility, Over explained, adding that agility can only be achieved when everyone in an organization acts professionally and uses a disciplined, measured approach to their work. What's Ahead The first posting in this series summarized discussions by Anita Carleton, director of the SEI’s Software Engineering Process Management Program, and Teri Takai, chief information officer for the DoD. Carleton provided an overview of the forum and discussed areas where SEI work is focused on applying Agile methods at-scale. Takai then discussed how Agile methods have been introduced into the DoD software acquisition and development environment. The second posting summarized discussions by Mary Ann Lapham, a senior researcher in the SEI’s Acquisition Support Program, who highlighted the importance of collaboration with end users, as well as among cross-functional teams, to facilitate the adoption of Agile approaches into DoD acquisition programs. The third posting highlighted the forum presentation by Ipek Ozkaya, a senior researcher in the SEI’s Research, Technology, and System Solutions Program, who discussed the use of strategic, intentional decisions to incur architectural technical debt. The technical debt metaphor describes the tradeoff between taking shortcuts in software development to speed up product delivery and slower—but less risky—software development. In the next—and final—posting in this series I will summarize my presentation at the forum on the importance of applying agile methods to common operating platform environments (COPEs) that have become increasingly important for the DoD. I will explain how these methods can encourage more effective collaboration between users, developers, testers, and certifiers to help the DoD successfully build more integrated, interoperable, and affordable software systems. We look forward to hearing your thoughts on applying agile at-scale in the comments section below. Additional Resources To learn more about the Team Software Process (TSP), please visit www.sei.cmu.edu/tsp. The slides and recordings from the SEI Agile Research Forum can be accessed at www.sei.cmu.edu/go/agile-research-forum.

By Douglas C. Schmidt, Principal Researcher While agile methods have become popular in commercial software development organizations, the engineering disciplines needed to apply agility to mission-critical, software-reliant systems are not as well defined or practiced. To help bridge this gap, the SEI recently hosted the Agile Research Forum. The event brought together researchers and practitioners from around the world to discuss when and how to best apply agile methods in mission-critical environments found in government and many industries. This blog posting, the fifth and final installment in a multi-part series highlighting research presented during the forum, summarizes a presentation I gave on the importance of applying agile methods to common operating platform environments (COPEs) that have become increasingly important for the Department of Defense (DoD). The first half of my presentation motivated the need for COPEs that help collapse today’s stove-piped, software-reliant DoD system solutions to decrease costs, spur innovation, and increase acquisition and operational performance. Since this material has appeared in my first SEI blog posting on COPEs, I’ll skip it in this posting and focus on the second half of my presentation, which discussed how applying agile methods can encourage more effective collaboration between users, developers, testers, and certifiers of COPEs to help the DoD build integrated, interoperable, and affordable software-reliant systems more successfully. What’s Taking So Long to Achieve the Promise of COPEs? Decades of public and private research and development investments—coupled with globalization and ubiquitous connectivity—has enabled information technology to become a commodity, where common off-the-shelf (COTS) hardware and software artifacts get faster and cheaper at a remarkably predictable pace.  During the past two decades, we've benefitted from the commoditization of hardware and networking elements. More recently, the maturation and widespread adoption of object-oriented programming languages, operating environments, and middleware is helping to commoditize many software components and architectural layers. Despite these technology advances, however, developing affordable and dependable COPE-based solutions remains elusive for the DoD. There are a number of reasons for this, including When DoD acquisition programs have tried to apply COPEs, they’ve tended to spend a lot of time building common software infrastructure, which consists largely of various layers of middleware. This process leads to something called the serialized phasing problem, where developers spend so much time building the infrastructure, that it may take years (often hundreds or thousands of person years) to develop this infrastructure. Meanwhile, application developers sit idle, not knowing precisely what the characteristics of the infrastructure will be. As a result, by the time the infrastructure has matured enough to support application development, the teams often discover the infrastructure provides inappropriate quality-of-service (QoS) or functionality. If this discovery occurs late in the lifecycle (e.g., during system integration, which is all too common in large DoD acquisition programs) it’s extremely costly to remedy. A related problem faced by acquisition programs is the length of time it takes to get projects under contract via traditional contracting models. While this glacial contracting pace is not unique to COPEs, it greatly exacerbates the serialized phasing problem outlined above. In particular, if contract delays become excessive, infrastructure developers won’t have sufficient time to build and test the common software infrastructure thoroughly. If the common software infrastructure is not built and tested properly, application developers will end up wrestling with many defects and bottlenecks they are ill-equipped to handle. Problems stemming from serialized phasing that are caused by contract delays will result in even further delays in application delivery. Classic DoD waterfall models assume that requirements can be largely defined early in the lifecycle. When some of these requirements change—as they certainly will as COPEs are reapplied in different contexts—it becomes quite expensive for the government to request the change orders needed to make the modifications. Without more streamlined and flexible lifecycle and contracting models, the inevitable changes to COPEs will be prohibitively expensive, thereby obviating the goal of cost savings achieved by sharing common software. There’s a long-term trend in the DoD towards adopting COTS technologies for hardware and software. Although many COTS products are well-suited for mainstream commercial applications, they’re often not as well-suited for certain types of DoD systems, especially mission-critical weapons systems. The challenge for COPE developers is to ensure that they don’t base their common software infrastructure on COTS products that work well when applied in contexts where an 85 percent solution is sufficient (where dropping an occasional call or manually refreshing a hanging web page is acceptable) in a DoD weapons system with more stringent QoS requirements.  In these mission-critical environments—where the right answer delivered too late becomes the wrong answer—many COTS products are too big, too slow, or too unreliable to serve as the basis of mission-critical COPEs. If DoD common software infrastructure is built naively atop a house of sand, applications will be hard-pressed to provide the end-to-end QoS that’s expected of them in hostile settings. Another challenge facing developers of COPEs is overly-adhering to ossified standards and reference architectures that made sense a decade or so ago, but which failed to keep pace with technology advances. Unlike mainstream commercial systems—where technologies are often refreshed every couple of years—a longer-term perspective is needed to develop COPEs for DoD weapons systems. Likewise, it’s essential to integrate new technologies into COPEs to keep pace with advances in hardware, software, platforms, and requirements. We also need software architectures that are more flexible and resilient than simply adhering to standards that become obsolete. The crucial issue here is not new standards, but the capability of coming up with new architectures and new standards that themselves can be refreshed and reapplied with minimal impact over long periods of time. How Agility Helps Achieve the Promise of COPEs At the heart of the problems described above is a lack of a holistic approach that balances key business, managerial and technical drivers at scale. The last part of my presentation discussed key success drivers for COPE initiatives that proactively and intentionally exploit commonality across multiple DoD acquisition programs and outlined ways in which agility helps to make those drivers more effective.  In my experience working at the SEI, DARPA, and Vanderbilt University’s Institute for Software-Integrated Systems for the past several decades, successful COPE efforts require a balance of the following drivers: Business drivers, which focus on achieving effective governance and broad acceptance of the economic aspects of COPEs. Examples include managed industry/government consortia, agile contracting models, and effective data rights and licensing models. Management drivers, which focus on ensuring effective leadership and guidance of COPE initiatives. Examples include mastery of agile lifecycle methods, strong science and technology connections to reduce technical risk, and of course a solid understanding of the critical role of software for the DoD. Technical drivers, which focus on the foundations of COPE development. Examples include systematic reuse expertise, agile architecture expertise, and automated conformance and regression test suites. As outlined above, agility can be applied to help each of these drivers. For example, agility can be applied to expedite contracting, which is an important business driver. Getting task, delivery, and change orders in place quickly via agile contracting models (such as Indefinite-Delivery, Indefinite-Quantity (IDIQ) contract vehicles) helps to mitigate serialized phasing problems. Likewise, agile contracting methods can also help ensure that the people who are building the systems and the people who are doing the acquisition triage and align their priorities so that the contract supports the needs and considerations of key stakeholders. Agility can also be applied to manage COPE lifecycles more effectively, which is a key management driver. Common software infrastructure development efforts work best when there are close, interactive feedback loops between people building the applications and the systems engineers and the software engineers and developers building the infrastructure. Without these feedback loops, it’s easy to develop many reusable artifacts that aren’t useful and won’t be applied systematically. An agile approach is thus essential to enable close cooperation between users, developers, testers, and certifiers throughout the lifecycle to ensure the necessary COPE capabilities are delivered rapidly and robustly  avoid integration "surprises" where things tend to break or underperform at unexpected times late in the lifecycle when its more expensive to fix problems We’ve observed in recent years that rolling out incremental deliveries of a COPE capability every 4 to 8 months helps application developers establish a battle rhythm of knowing when to upgrade and when to leverage what’s in the common software infrastructure. This tight spiral avoids long and fragile serialized phasing lifecycles, which is something that agile methods do a good job of from a management perspective. Finally, agility can be applied to ensure architectural flexibility, which is a crucial technical driver. Long-lived, software-reliant DoD systems need software architectures that are change tolerant. Inevitably standards will evolve; hardware will get better, faster, and cheaper; and software programming languages, operating systems, and middleware will all evolve over time. It is therefore essential to devise ways of "future proofing" COPE architectures and using technologies, techniques, and methods (such as the technical debt work that Ipek Ozkaya discussed at the Agile Research Forum) when making decisions about when to reengineer and when to refactor. These decisions should be based on empirical data and evidence, rather than relying on forecasts or legacy commitments that become obsolete and ossified over time. Equally important is the ability to select COTS technologies for use in COPEs that have appropriate QoS capabilities for the ways in which they are applied to particular missions. Some COTS- and standards-based products work well in mission-critical contexts, whereas others don’t. The choice of which ones to use should be driven as much as possible through trade studies, empirical analysis, and various types quantitative analysis, as opposed to the latest techno-fad. The following table summarizes how agility helps resolve the COPE challenges discussed above: COPE Challenges How Agility Helps Resolve COPE Challenges Serialized phasing of COPE infrastructure and application development postpones identifying design flaws that degrade system QoS until late in the lifecycle, i.e., during system integration Enables close cooperation of users, developers, testers, and certifiers throughout lifecycle to rapidly deliver COPE capabilities and avoid integration "surprises" without needing extensive upfront planning and serialized phasing Emphasizes incremental rollout of COPEs by delivering useful capability every 4 to 8 months to reduce risk via early validation by application developers and users Glacial contracting processes don’t support timely delivery of COPE capabilities to meet mission needs Engages users and testers in developing COPE contract scope, evaluation criteria, incentives, and terms/conditions to ensure contracting supports all needs/considerations Contracting models that assume COPE requirements can be defined fully up front are expensive when inevitable changes occur Expedites execution of COPE work packages via multiple award Indefinite-Delivery, Indefinite-Quantity (IDIQ) contract vehicles, and issue Task/Delivery Orders for each release QoS suffers when COPE initiatives attempt to use COTS products that are not suited for mission-critical DoD combat systems Leverages common development, test and production platforms, and QoS-enabled standards-based COTS to deliver COPE capabilities faster, cheaper, and more interoperably, without redundant ad hoc infrastructure Rigid adherence to ossified standards and reference architectures impedes COPE technology refresh and limits application capabilities Establishes a change-tolerant architecture enabled by discovery learning that promotes decisions based on empirical data/evidence, rather than forecasts or legacy commitments   The Road Ahead for COPE Agility One reason for the spotty track record of success with COPE-based systems is that the DoD hasn’t taken a holistic view of the way these types of systems are built. Existing brittle and proprietary stovepiped approaches to acquisition systems do not address the cost efficiency and cyber security challenges the DoD are wrestling with, nor do they help to deploy software and new technologies to the field rapidly. Moreover, managing the production of these systems via waterfall processes is simply not an effective way forward, especially in the shadow of sequestration. Developing COPEs is achievable and valuable, but it’s not easy. Agility in business, management, and technical dimensions is essential, but it’s also no panacea. Additional research and engineering investment is needed to devise the appropriate methods, tools, and techniques that will enable agility at-scale, which is a key theme that we’ve emphasized throughout the SEI Agile Research Forum. Finally, we need your help. Achieving success with COPEs for the DoD isn’t something that can be done by any one group, institute, or government program. The SEI needs to help bring together researchers, developers, and managers from academia, industry, and government to conduct the appropriate work to help reduce risk and ensure the success of current and planned COPE initiatives. We also need to work closely with academia and industry to train the workforce and identify key requirements.  Likewise, we need to work with government to ensure effective oversight and acquisition dynamics to reduce the cycle time needed to acquire new systems, insert new technology into legacy systems, and sustain software-reliant systems at a lower cost over their lifecycles and across the DoD enterprise.  The stakes are high, and now is the time to make a difference! Additional Resources To learn more about the SEI’s work on common operating platform environments, please visit http://blog.sei.cmu.edu/archives.cfm/category/common-operating-platform-environments-copes To learn more about the SEI’s work on agile methods at-scale, please visit the SEI Agile Research Forum webinar site at http://www.sei.cmu.edu/go/agile-research-forum/.

By Robert Stoddard, ResearcherSoftware Engineering Measurement and Analysis Program As part of our research related to early acquisition lifecycle cost estimation for the Department of Defense (DoD), my colleagues in the SEI’s Software Engineering Measurement & Analysis initiative and I began envisioning a potential solution that would rely heavily on expert judgment of future possible program execution scenarios. Previous to our work on cost estimation, many parametric cost models required domain expert input, but, in our opinion, they did not address alternative scenarios of execution that might occur from Milestone A onward. Our approach, known as Quantifying Uncertainty in Early Lifecycle Cost Estimation (QUELCE), asks domain experts to provide judgment not only on uncertain cost factors for a nominal program execution scenario, but also for the drivers of cost factors across a set of anticipated scenarios. This blog post describes our efforts to improve the accuracy and reliability of expert judgment within this expanded role of early lifecycle cost estimation. Our work in cost estimation began two years ago, building upon a review of existing cost estimation and expert judgment research. As an example, we identified an industry consultant, Douglas Hubbard, whose book, How to Measure Anything, presents an approach known as "calibrating your judgment" (my colleague, Dave Zubrow, describes Hubbard’s technique in a recent blog post).  Hubbard’s focus on calibrating expert judgment using "trivial pursuit" exercises led to our team’s decision to pursue research into the use of domain-specific reference points to further improve the accuracy and reliability of expert judgment within cost estimation. Our research on early lifecycle cost estimation for the DoD consists of two tasks: Development of the QUELCE method, in which the probability of changes occurring in program execution is separated from the final assessment of the effects of such changes on the program cost estimate Critical thinking and designed experiments that would contribute to current research on expert judgment We hypothesized that a domain-specific approach to calibration training and development of reference points would be necessary to reduce unwanted variation in judgments rendered by experts participating in the QUELCE method. We decided to take a two-pronged approach to improving expert opinion. The first part of the approach involved data mining of DoD program execution experience.  The second part of the approach interviewed DoD experts about cost estimation and DoD program cost experience.  One of our goals is to create an online repository of domain reference points that embodies the historical DoD program cost experience.  The repository will include a searchable database of reference points that helps domain experts exercise better judgment during cost estimation.  Domain experts will be able to query the reference points using key words based on search technology. Search results will show the key reference points in relation to the domain and technology challenge. The domain expert(s) will then review those reference points before formulating judgement for the current cost estimation exercise.  At this point in the project, we are mining reference points from DoD and other open-source data examples. My colleague, James McCurley, has investigated DoD repositories for raw information that outlined why acquisition programs experience cost and schedule overruns.  Our team compiled domain reference points from McCurley’s data that identify selected changes associated with cost and schedule overruns. We are categorizing these changes into a set of common change drivers that are rooted in the various sources we have accessed. One of the first sources we accessed was the U.S. Navy’s Probability of Program Success (PoPS) program. The PoPS criteria came from studies of program performance used by the Navy to implement a step-by-step approval process for a program to continue, independent of—but aligned to—the DoD acquisition process.  PoPS identified a number of categories of reasons for cost and schedule overruns in government programs. PoPS was always seen as just one of many sources of programmatic factors that might provide information useful to QUELCE. The PoPS criteria are biased toward programmatic change issues (such as sponsorship, contractor performance, and program office performance) that are of primary concern to DoD sponsors and Program Executive Offices.  As expected, when we started this project, we are finding the need to supplement PoPS with more technical change issues, such as those related to system engineering and integration factors. Many technical change drivers may be seen in the Capability Based Assessment (CBA) activity performed by programs in preparation for the Milestone A decision. The CBA includes the Functional Area Analysis (FAA),  Functional Needs Analysis (FNA), and Functional Solution Analysis (FSA).  Another early source is the Analysis of Alternatives (AoA).  These and other early documents often include information that identifies technical and programmatic uncertainties not captured in the cost estimation process, but which can be incorporated as program change drivers in QUELCE method.  Consequently, many technical change drivers are rooted in artifacts that proposed programs must draft prior to Milestone A. Examples of change drivers we’ve identified from various sources include: Interoperability - a program is affected by changes from a dependent program Contractor Performance - a subcontractor must be replaced Obsolescence - a part is made obsolete before a program is operational Technical Performance - either a technology is not ready for use or a technology fails to achieve key performance goals Scope - the source of many changes, including new users, additional delivery targets, and extra platforms, all of which fall outside the realm of "code growth" Funding - funding may be increased  or decreased in DoD programs, often with little warning Applying Our Expert Opinion Approach In the coming year, our goal is to develop a database with information that supports experts implementing the QUELCE method.  We will publish our approach to improving expert judgment and increasing and structuring their involvement in cost estimation using procedures similar to Team Software Process (TSP) scripts.  Our goal is to ensure that domain experts have more active involvement with the cost estimation activity.  The problem today is that domain expert results are often loosely coupled with the cost estimates.  In contrast, QUELCE will facilitate domain experts systematically discussing change drivers and then mapping the change drivers explicitly to the cost driver inputs of traditional cost estimating models and estimating relationships (CERs). In our approach, the domain expert will be prompted at different points throughout QUELCE to access the reference point database. These activities will consist of just-in-time virtual training for calibration within a given domain.  For example, a domain expert may be participating in QUELCE to develop a cost estimate for a new communication system that involves satellite technology.  If the domain expert has not recently completed virtual calibration training for that domain, he or she may receive a refresher course consisting of a two- to four-hour online exercise. Our approach to improving expert opinion will help domain experts during the following three different points in the QUELCE method that depend significantly on expert judgment: Identifying pertinent change drivers. After completing the training, domain experts will be asked to participate in a workshop exercise that anticipates which change drivers will most likely be relevant to a particular program. In the workshop, domain experts will query for communication programs or specific technology names related to particular programs. In the example involving the new communication system above, the results should yield information related to historical communication programs or technologies and domain reference points, explaining why certain aspects went over budget or schedule. Populating the change driver cause-and-effect matrix. The second judgment point involves a change driver cause-and-effect matrix. The domain expert will evaluate each change driver and rate the probability, on a scale of 0 to 3, that the change driver will cause any other change drivers on the list to switch from a nominal to an off-nominal condition, thereby signaling the danger of cost and schedule overruns. This exercise requires judgment about the relationships between change drivers. The domain expert will get information from querying our repository before rendering this type of judgment. For example, reference points might include historical information about a change driver going off nominal and subsequently causing three other change drivers to go off-nominal. The reference points therefore give the domain expert a basis to understand the relationships between change drivers and help make them more accurate. Establishing probabilities for the Bayesian Belief Network (BBN). The BBN models the change drivers as nodes in a quantitative network, including probabilities that state changes in one node will create a state change in another node.  Every change driver has a parent table that presents all the possible scenarios resulting from different combinations of its parent change driver states. For example, in our BBN, we have change driver A and change driver B, and both have an influence on change driver C. If change driver A has nominal and off-nominal states and change driver B has nominal and off-nominal states, there are four different combinations of parent change driver states, e.g. scenarios that may affect change driver C. While our work to date has focused on calibrating expert judgment in the DoD cost estimation of program development, our approach could be applied to many situations beyond cost estimation. We envision this approach being used in domains such as portfolio management and strategic planning.  Summary Our research into the QUELCE method for pre-Milestone A cost estimation represents a significant advance by enabling the modeling of uncertain program execution scenarios that are dramatically different from the traditional cost factor inputs of cost estimation models currently employed later in the DoD acquisition lifecycle.  By synergizing the latest advancements in proven methods such as scenario planning workshops, cause-effect matrices, BBNs, and Monte Carlo simulation, we have created a novel and practical method for early DoD acquisition lifecycle cost estimation.  If you’re interested in helping us succeed in these efforts, please let us know by leaving a comment below. Additional Resources To read the SEI technical report, Quantifying Uncertainty in Early Lifecycle Cost Estimation (QUELCE) please visit www.sei.cmu.edu/library/abstracts/reports/11tr026.cfm. For more information about Milestone A, please see the Integrated Defense Life Cycle Chart for a picture and references in the "Article Library."

Second in a Two-Part Series By Lisa Brownsword Acquisition Support Program Major acquisition programs increasingly rely on software to provide substantial portions of system capabilities. All too often, however, software is not considered when the early, most constraining program decisions are made.  SEI researchers have identified misalignments between software architecture and system acquisition strategies that lead to program restarts, cancellations, and failures to meet important missions or business goals. This blog posting—the second installment in a two-part series—builds on the discussions in part one by introducing several patterns of misalignment—known as anti-patterns—that we’ve identified in our research and discussing how these anti-patterns are helping us create a new method for aligning software architecture and system acquisition strategies to reduce project failure. Identifying Anti-Patterns We used an interview-based approach to discover and document patterns of alignment among four key aspects: business and mission goals, architecture, quality attributes, and acquisition strategy.  We then analyzed the interview data looking for evidence of alignment or misalignment.  Since most of our data comes from troubled programs, we have primarily discovered evidence of misalignment—known as anti-patterns—to date.  Our initial set of anti-patterns includes undocumented business goals - the lack of well-documented business goals expressed as they apply to an acquisition program  unresolved conflicting goals - the lack of analysis and reconciliation of known goals failure to adapt - failure of an acquisition program to modify the architecture and the acquisition strategy in response to changing goals, priorities, or technology  turbulent acquisition environment - requested changes are so frequent and contradictory that an acquisition program cannot realistically accommodate them poor consideration of software - critical decisions made early in an acquisition program’s lifecycle have strong negative implications on the system’s software inappropriate acquisition strategies - the acquisition strategy fails to consider important software attributes overlooking quality attributes - a failure to define and use software quality attributes in the definition of the software architecture or acquisition strategy Let’s explore one of these anti-patterns to show how it might be used. The first anti-pattern—undocumented business goals—reflects a lack of precise, well-defined, and well-documented business goals for a DoD acquisition program. In the programs we examined, we found that these goals were seldom explicitly expressed (e.g., "replace legacy system") or they reflected high-level program constraints (e.g., "maximize competition") or policy regulations (e.g., "implement an open architecture"). When this anti-pattern is present, we found that mission requirements dominate the definition of the software architecture, often leading to an architecture contrary to the achievement of the unstated business goal. For instance, an architect might reasonably design a monolithic architecture that was an excellent fit for the mission goals for performance, but which could be strongly at odds with an implicit—but unspecified—business goal to avoid vendor lock. The lack of explicit business goals has a more direct impact on the acquisition strategy. In one program in our study, a key element for the program was to build a new system with significant new capabilities.  The acquisition strategy specified a slow, deliberate pace to ensure that the new capability was defined correctly.  A competing goal was to replace several "end-of-life" systems. Not stated in this goal was the urgent need to replace these failing system as quickly as possible. When the operators and maintainers of the legacy systems became aware of the intended acquisition strategy, they forced a major change in program focus. The sequence of acquisition activities required alteration, which caused a significant delay in meeting either goal. Creating a Method for Identifying Misalignments Discovering and documenting anti-patterns is only the beginning of our work in addressing the problems of misalignment.  The second phase of our project involves creating a method that helps acquisition programs avoid the anti-patterns we’ve discovered and provides options that could help programs better align their acquisition strategy and software architecture to satisfy stakeholder mission and business goals. We will then validate the utility of this method through the help of projects and programs outside the SEI.  Software-reliant systems are inherently social and technical endeavors. A key facet of our method, therefore, is thus its ability to bring disparate actors together—often for the first time—to rationally identify and discuss issues of mutual concern and be able to make hard and informed choices based on rational information. We plan to adapt and tailor existing methods where possible. We are currently exploring methods in the following areas: identifying salient stakeholders - There are many requirements elicitation and analysis methods. Unfortunately, most methods assume that it is possible to know which stakeholders will most affect or be most affected by the program. We are therefore considering Controlled Requirements Expression (CORE), which is a method that assists developers in identifying stakeholders related to a given acquisition to help develop a more complete list of stakeholders. defining business and mission goals - The Pedigreed Attribute eLicitation Method (PALM) discussed in part one of this blog series remains a central element of our new method.  PALM enables organizations to systematically identify the high-priority mission and business goals from system stakeholders.  The architectural implications of those goals are then captured by determining the quality attribute requirements they imply.  We are extending PALM to investigate the acquisition strategy implications of business or mission goals. analyzing quality attributes - Quality Attribute Workshops (QAW) are a widely accepted method for developing definitions of the quality attributes that form the basis for deriving the software architecture. We are using the same approach to derive attributes that should drive the acquisition strategy. trading off architecture and acquisition strategy options - analysis methods, such as Architecture Tradeoff Analysis Method (ATAM) or Cost Benefit Analysis Method (CBAM), are used to ensure consistency of software and system quality attributes. We are analyzing these methods to explore consistency between the acquisition strategy and its driving quality attributes. Looking Ahead Government acquisitions are more likely to succeed if a program can align its acquisition strategy and software architecture with each other and with respect to satisfying stakeholder mission and business goals. Our research has shown evidence of misalignments in the form of anti-patterns. Discovering the patterns a program should avoid is a key step toward our objective to develop a method that systematically supports business and mission goals by aligning the acquisition strategy and software architecture. We welcome opportunities to validate and expand the anti-patterns or pilot our emerging method. Please leave us feedback or questions about our research in the comments section below and we will follow up with you. Additional Resources For more information about the Pedigreed Attribute eLicitation Method (PALM), please visitwww.sei.cmu.edu/architecture/tools/establish/palm.cfm To read A Method for Controlled Requirement Specification, please visithttp://ss.hnu.cn/oylb/tsp/CORE-mullery.pdf To read the SEI technical report, Quality Attribute Workshops, please visithttp://www.sei.cmu.edu/library/abstracts/reports/03tr016.cfm For more information about the book, Evaluating Software Architectures: Methods and Case Studies, please visit www.sei.cmu.edu/library/abstracts/books/020170482X.cfm To read the SEI technical report, Making Architecture Design Decisions: An Economic Approach, please visitwww.sei.cmu.edu/library/abstracts/reports/02tr035.cfm

Final Installment in a Three-Part SeriesBy Bill Nichols, Senior Member of the Technical StaffSoftware Engineering Process Management This post is the third and final installment in a three-part series that explains how Nedbank, one of the largest banks in South Africa, is rolling out the SEI’s Team Software Process (TSP) throughout its IT organization. In the first post of this series, I examined how Nedbank addressed issues of quality and productivity among its software engineering teams using TSP at the individual and team level. In the second post, I discussed how the SEI worked with Nedbank to address challenges with expanding and scaling the use of TSP at an organizational level. In this post, I first explore challenges common to many organizations seeking to improve performance and become more agile and conclude by demonstrating how SEI researchers addressed these challenges in the TSP rollout at Nedbank. In a 10-year retrospective on agile methods, Jeff Sutherland, co-creator of the Scrum agile method, listed some of the major challenges and impediments for organizations that adopt agile methods, including demanding technical excellence promoting individual change leading organizational change organizing knowledge improving education We’ve encountered and addressed many of these challenges in our work with Nedbank, which is one of several large companies to successfully pilot TSP and undertake an organizational rollout. For a TSP project (or any project using a new process) to succeed, management and other stakeholders must often change their behavior to support the process; they all must use the process in a way appropriate for their roles. Their behavior must support the empowerment of the TSP project team. Conversely, if the stakeholders do not behave appropriately, they can undermine the empowerment of the team and its ability to complete the project successfully. Demanding Technical Excellence In our experience, the key to success is to train management on how to ask for technical excellence and to know it when they see it. Technical people often know the details far better than their managers. Managers must be provided training and guidance on how to set the right goals for these empowered teams and to track the right measures. For Nedbank, the most important outcome of technical excellence is low rates of fielded severity 1 defects, that is, defects that stop the system. With TSP, this quality goal is not just a matter just for quality assurance (QA), but for the development teams to manage.   In an organization with self-managed teams based on TSP principles and practices, the teams and individuals manage the technical work. The teams define their process, estimate their work, track progress, and collect data on all defects. Status reports do not include detailed data, but summaries. Managers who oversee technical excellence need to hold teams accountable, but must avoid micromanaging the details of the technical work. This trust, while sometimes hard for management to grant, is essential to team empowerment and commitment. At Nedbank, senior management reviews a laundry list of visible outcomes for the project including adherence to committed delivery date, cost estimation accuracy, defects in QA and production, and accuracy of status reports. The teams are thus held accountable for project outcomes, managing their work, and keeping management informed. This accountability eliminates the fear of the data being misused, empowers the teams to make decisions, and aligns incentives and goals. To expand TSP use beyond early adopters, the organization must make expectations explicit and non-threatening. Project scorecards must focus on publically visible outcomes rather than on detailed process data. TSP includes training for management to help with this transition. This training is just one aspect of managing organizational change. Promoting Individual Change and Leading Organizational Change Attempts to change organizational behavior often fail. At the SEI, we address this difficulty by providing change-management training to TSP coaches. This training is effective at the individual and team level, but more is needed on an organization-wide scale.To address this gap, we coach the executive team on change management and the logistics of rollout and sustainment. The organization needs to build these new ways of working into its processes for training project planning project evaluation personnel evaluations Nedbank is fortunate to have an executive who understands the need for change and the difficulties surrounding it, and has provided resources to ease the transition. Organizing Knowledge and Improving Education Education must be tailored to the target audience. TSP provides specific training for coaches, instructors, developers, non-developer team members, team leads, and executive management. This training, which also addresses another challenge referenced by Sutherland, helped all involved at Nedbank understand the principles behind the change and their role in making new organizational practices a success. Through the Center of Excellence (COE) presented in my second blog posting, Nedbank assures that training is performed and that everyone involved is prepared to work on a TSP project. The function of organizing knowledge remains a work in progress as the COE collects project data and experiences. Perhaps the greatest barrier to collection of TSP data at the organizational level is the lack of enterprise tools. We are teaming with two large partners to develop the next generation of TSP-data tools at the enterprise level, but they are still a work in progress.  Agile development organizations must also secure sponsorship, build capability, identify and train change agents, develop organizational support for the initiative, track progress, and highlight success to senior management. They must not associate long-term success with any single project, but rather with demonstrated and sustained improvement over many projects and many people. Nedbank, which is at the leading edge of TSP implementation, started down this path with a rollout strategy that included funding and time for training executives, project leads, coaches, and team members setting expectations about how projects will be planned and conducted gathering, reporting, and analyzing specific project data The SEI worked with the COE to help Nedbank pilot TSP and train instructors and coaches. TSP coaches continue to help Nedbank plan and implement its organizational rollout. Results: An Organization Changed Nedbank now begins all new software projects with TSP. This effort is led by the TSP COE, which estimates needs and supports teams with coaching and training. In addition to providing operational support, the Nedbank COE collects data to track organizational progress and provides relevant planning data for its teams. Relevant data includes ranges of project schedule and cost variances, scope growth, component size, and effort estimation accuracy, normalized defect levels in QA and user-acceptance test, schedule and cost ratios of development, testing, maintenance, cost of specific activities such as peer inspection, and cost of rework. Nedbank uses this data at the organization level to Assess the overall cost efficiency of work. The benefits in schedule predictability, quality, and overall cost containment are made explicit and real. Plan projects. The use of historic data of comparable projects means the planning parameters are no longer just guesses. We have data from similar projects at Nedbank to suggest how results might be affected by the duration of front-end or back-end projects, the duration in testing, the calendar time that resources must be committed to support testing, and changes in the number of staff. Improve performance. Technical excellence and quality are economic decisions. Improving cost, quality, or schedule performance requires change, (e.g., taking time to plan, training for inspections, and measuring quality early). It’s important to know what changes will occur and what those changes will cost. Only by understanding the current process can the organization set realistic goals. By combining realistic goals and a data-driven knowledge of the process, empowered teams can make specific changes to the way they do work and evaluate the results. Simple outcome measures include severity defects in production, schedule overruns, and cost per change. To improve performance, Nedbank measured how much effort and time were required for testing and how much effort was devoted to defect fixes. With reliable data consistently defined in projects across the organization, Nedbank is building performance benchmarks and can now begin to make data-based cost benefit analysis for process changes. When Nedbank introduced detailed project planning and tracking, the use of TSP made it possible to see how these changes altered time spent in specific activities and the associated results on schedule performance, cost containment, and quality at delivery. In the context of a defined process, a few simple measures—direct effort, defects, schedule, and size—transformed their ability to see and manage their software releases. Looking Ahead We are continuing to work with Nedbank (and other organizations) to share our TSP research. Nedbank’s COE currently focuses on growing capacity during the rollout. The process is arduous and will take several years to complete, which can lead to frustration. Shortcuts, such as incomplete training, lack of coaching support, or stakeholders who have not yet fully bought into the new approach, will degrade results and likely create resistance. When the rollout is complete, the emphasis will shift to sustainment. As shown in the COE’s pilot video, Nedbank is already seeing benefits from these changes. If you’re interested in learning more about TSP, please consider attending the upcoming TSP Symposium in St. Petersburg, Florida, USA. Additional Resources For more information about TSP, please visitwww.sei.cmu.edu/tsp For more information about the 2012 TSP Symposium, please visitwww.sei.cmu.edu/tspsymposium/2012/ To read the SEI technical report Deploying TSP on a National Scale: An Experience Report from Pilot Projects in Mexico, please visitwww.sei.cmu.edu/library/abstracts/reports/09tr011.cfm To read the Crosstalk article A Distributed Multi-Company Software Project by Bill Nichols, Anita Carleton, & Watts Humphrey, please visitwww.crosstalkonline.org/storage/issue-archives/2009/200905/200905-Nichols.pdf To read the SEI book Leadership, Teamwork, and Trust: Building a Competitive Software Capability by James Over and Watts Humphrey, please visit www.sei.cmu.edu/library/abstracts/books/0321624505.cfm To read the SEI book Coaching Development Teams by Watts Humphrey, please visitwww.sei.cmu.edu/library/abstracts/books/201731134.cfm To read the SEI book PSP: A Self-Improvement Process for Engineers by Watts Humphrey please visitwww.sei.cmu.edu/library/abstracts/books/0321305493.cfm

By Donald FiresmithSenior Member of the Technical StaffAcquisition Support Program Engineering the architecture for a large and complex system is a hard, lengthy, and complex undertaking. System architects must perform many tasks and use many techniques if they are to create a sufficient set of architectural models and related documents that are complete, consistent, correct, unambiguous, verifiable, usable, and useful to the architecture’s many stakeholders.  This blog posting, the second in a two-part series, takes a deeper dive into the Method Framework for Engineering System Architectures (MFESA), which is a situational process engineering framework for developing system-specific methods to engineer system architectures. In our previous blog entry, we introduced MFESA and its four components: the MFESA ontology defining the foundational concepts underlying system architecture engineering the MFESA metamodel defining the base superclasses of method components the MFESA repository of reusable method components the MFESA metamethod for creating project-specific methods using method components from the MFESA repository We also briefly discussed the applicability of MFESA and how it simultaneously provides the benefits of standardization and flexibility. In this blog posting, we will take a closer look at the four components comprising MFESA. The MFESA Ontology To create a complete and well-defined method for performing system architecture engineering, it is first necessary to understand terminology (technical jargon) and concepts underlying this area. This understanding goes beyond a mere glossary of terms to encompass an information model that also describes how these concepts relate to each other. The MFESA ontology defines the domain of system architecture engineering and is the foundation on which the rest of MFESA is build. Figure 1 below summarizes many of the most important contents of the MFESA ontology, an information model of the foundational concepts underlying system architecture engineering. Starting at the center of the diagram and moving to the left, we see that system architecture comprises a combination of architectural structures that can be static or dynamic as well as logical or physical architectural decisions that include the use of architectural styles, architectural patterns, and architectural mechanisms Starting at the center and moving to the right, we see that system architecture can be documented in many ways including in the form of architectural descriptions, some of which are various types of architectural documents as well as various types of architectural models that model the different kinds of architectural structures executable representations, such as architectural prototypes, architectural simulations, and executable architectures At the top of the diagram, we see that the architectural concerns of stakeholders are architectural drivers, including architecturally-significant requirements that drive the engineering of the system architecture. Architectural concerns are also, often quality focus areas (i.e., quality characteristics such availability, performance, portability, reliability, robustness, safety, security, and usability) and architectural support for these qualities can be organized in the form of architectural quality cases (a.k.a., assurance cases) that provide arguments and evidence that the architecture adequately supports the architecturally-significant requirements.   View larger Image   The MFESA Metamodel The second MFESA component is a process metamodel that is restricted to system architect engineering. Figure 2 shows the MFESA view of the concepts: process, method, and process metamodel.     The system architecture engineering processes that are performed on different projects are at the lowest level of Figure 2. Each process consists of components such as actual work products (e.g., architectural models and architecture documents) actual work units (e.g., instances of architecture engineering tasks and techniques) that are used to produce the work products actual workers (e.g., specific architecture teams and architects) who perform the work units to produce the work products. The middle layer consists of system architecture engineering process models that are the as-intended methods for engineering system architectures. These methods contain reusable components that describe their instances: the process components. MFESA thus recognizes that there is both a theoretical and practical difference between the methods documented in standards, procedures, and guidelines (middle layer) and the work people actually perform on their projects (bottom layer). At the top level of this three-level structure is the MFESA system architecture engineering process metamodel, which models the process model. This metamodel consists of metamethod components, which are the abstract subclasses that are specialized to produce the method components. For example, the MFESA metamethod component task is subclassed to produce 10 specific tasks of system architecture engineering that, in turn, are instantiated as actual task executions on the project. Figure 3 shows the four metamethod components within the MFESA process metamodel. They are the abstract classes of process components that are subclassed to create the MFESA method components.     The MFESA repository contains an extensive set of reusable method components derived via subclassing from the MFESA metamethod components.  Figure 4 depicts the first nine method components (abstract classes of process components). MFESA thus recognizes three types of architecture workers who perform three types of architectural work units to produce three types of architecture work products. The complete class hierarchy of method components is considerably larger and includes the concrete method components that are instantiated to produce the actual process components seen on real projects. Example reusable method components include subtypes of: architectural work products, such as architectural representations (which we saw in the MFESA ontology) that include both architectural documents (e.g., system architecture document, software architecture document, and architecture vision document, various types of architectural whitepapers and reports such as how the architecture handles concurrency or fault tolerance) and architectural models (e.g., class diagrams, sequence diagrams, statecharts, and data flow diagrams) architecture workers, such as the system architect, software architect, and architecture team, and various types of architecture modeling and documentation tools architecture work products, such as the many types of architecture tasks (e.g., identify the architectural drivers and maintain the architecture and its representations) and techniques (e.g., brainstorming and architectural patterns)   View larger Image   The MFESA Metamethod As mentioned in the previous blog entry, MFESA is not a method for engineering system architectures but rather a framework for creating methods for engineering system architectures. Figure 5 shows the MFESA metamethod for creating these methods. Each box in the figure is a step in the method. The first metamethod step determines the project’s needs regarding system architecture engineering methods. The second step determines the number of such methods that are needed, which for most projects is one. The third step determines whether a previously constructed method can be tailored to fit the specific needs of the project, in which case a method is selected and then tailored, or whether a new method needs to be constructed from the reusable method components, in which case the relevant method components are selected, tailored, and integrated to form the new method  In both cases, the resulting method(s) must be documented, typically in plans, standards, procedures, guidelines, templates, and user manuals. The document method(s) must also be verified as complete, consistent, correct, and usable. Finally, the verified method(s) must be approved and published.   View larger Image   Wrapping Up Systems, organizations, and contractual relationships between acquirers and developers are diverse and multi-dimensional. No single architecture engineering method is therefore sufficiently complete and tailorable enough so that it is appropriate for all situations. MFESA is not a method for engineering system architectures but rather a method framework that can be used by system architects and process engineers to produce appropriate system architecture engineering methods using situational method engineering. To accomplish this, MFESA consists of four, interrelated components: an ontology that defines the foundational concepts of system architecture engineering and the relationships between them a metamodel that defines the base classes of reusable method components from which all other method components are subclassed a repository of reusable method components (architectural work products to be produced, work units to be performed, and architectural workers) from which situation-specific system architecture engineering methods can be constructed a metamethod for selecting appropriate method components, tailoring them, and integrating them to produce the appropriate method Even if you are content using a single organizational system architecture engineering method, you may find it worthwhile to peruse the MFESA repository to see if your method is missing any important elements. Consultants, trainers, and educators may also find MFESA a good foundation on which to build training courses and classes on system architecture engineering. Finally, researchers in system architecture engineering methods and situational method engineering may find MFESA useful as a repository of reusable method components.   Additional Resources MFESA is primarily documented in the book, The Method Framework for Engineering System Architectures, published in 2009 by CRC Press. The book was co-authored by a six-member team from the SEI, MITRE, and the U.S. Air Force. The book was recently added to the Intel Corporation’s Recommended Reading List. To see a tutorial on MFESA presented at the 2011 IEEE International Systems Conference (ISC) in Montreal, Quebec, Canada please go to http://donald.firesmith.net/home/publications/publicationsbyyear/2011/2011-MFESA_ISC.pdf To see a tutorial on MFESA presented at the 21st System and Software Technology Conference (SSTC) in Salt Lake City, Utah, please go to http://donald.firesmith.net/home/publications/publicationsbyyear/2009/MFESA-SSTC.pdf

By Andrew P. MooreSenior Member of the Technical StaffThe CERT Program Since 2001, researchers at the CERT Insider Threat Center have documented malicious insider activity by examining media reports and court transcripts and conducting interviews with the United States Secret Service, victims’ organizations, and convicted felons. Among the more than 700 insider threat cases that we’ve documented, our analysis has identified more than 100 categories of weaknesses in systems, processes, people or technologies that allowed insider threats to occur. One aspect of our research has focused on identifying enterprise architecture patterns that protect organization systems from malicious insider threat. Enterprise architecture patterns are organization patterns that involve the full scope of enterprise architecture concerns, including people, processes, technology, and facilities. Our goal with this pattern work is to equip organizations with the tools necessary to institute controls that will reduce the incidence of insider compromise. This blog post is the second in a series that describes our research to create and validate an insider threat mitigation pattern language that focuses on helping organizations balance the cost of security controls with the risk of insider compromise. Our Approach The aim of our pattern work is to develop insider threat mitigation strategies that are scientifically and operationally valid. To create those strategies, we employ mixed-methods research, which combines both qualitative and quantitative approaches. Among the various types of insider crimes—IT sabotage, theft of intellectual property (IP), national security/espionage, and fraud—our work has initially focused on IP theft, which includes theft of an organization’s proprietary information. We have already established a mitigation pattern of IP theft that is based on the types of crime we’ve observed in our case database. This pattern is oriented around the observation that many IP thieves steal information close to announcing their resignation. This behavior gives organizations a window of opportunity for identifying and responding to insider IP theft activity. Since it’s costly and time-consuming for organizations to monitor departing employees 100 percent of the time, we directed our resources at the timeframe when it is the highest likelihood that IP theft will occur. Our pattern focused on this question: I am establishing a program that looks for evidence of insider theft of my organization’s IP. Review and analysis of employee activities is costly. How can I improve the efficiency of resources I direct at IP theft detection? To help answer this question, our research team decided to focus on the distribution of durations between the following two dates across our sample of insider threat cases: the date of the last confirmed theft of IP event prior to an insider’s termination and the date of the insider’s termination Past qualitative analyses of our insider threat data have suggested that the approach of a termination day accelerates the insider’s decision-making process in a nonlinear manner. Our primary hypothesis is therefore the following: Primary Hypothesis: The distribution of the times between an insider IP thief’s last confirmed theft of IP before termination and the date of the insider’s termination follows a nonlinear distribution. Preliminary Analysis To determine whether our data on insider theft of IP crimes supports this hypothesis, we collaborated with Dave Zubrow, acting chief scientist with the SEI’s Software Engineering Process Management Program and lead of the Software Engineering Measurement & Analysis Initiative. To test the hypothesis, we used Crystal Ball software to evaluate the best fit distribution for our data on 30 IP theft cases from the CERT database. The geometric distribution (with p=0.02) was the best fit to our data when compared with other candidate distributions. We also ran a Monte Carlo simulation that generated 1,000 resampled data sets from the best fit distribution. From that data set, we graphed the cumulative probability function. We found that about 70 percent of insider IP theft cases can be caught by reviewing for significant theft events by the insider during the last 60 days of employment. Perhaps more importantly, the graphed function provides a tool to help organizations adjust their review window in an informed way, based on their particular risk aversion for IP theft and the cost of insider activity review within the organization. It is important to emphasize the limitations of our data analysis to date. Our data analysis and results are preliminary in part because of the small number of cases in our data set. While the best-fit distribution was the geometric distribution (as compared to a wide variety of other distributions), the fit was statistically different from the theoretical distribution. While future research will continue to add additional cases to better identify the underlying distribution and refine our analysis, the resampling approach described above allowed us to use the data that we had to greatest effect.  Given that the bestfit for the data is the geometric distribution, we contend that this result provides at least prima facie evidence that the subject mitigation pattern will be effective in fighting insider theft of IP. Continuing research will strive to bolster this evidence. We expect that the patterns and pattern language developed through this research will enable coherent reasoning about how to design enterprise systems to protect against malicious insider activity. Instead of working with vague security requirements and inadequate security technologies, system designers will have a coherent set of patterns that enable them to develop and implement effective strategies against the malicious insider activity more quickly and with greater confidence. Looking Ahead In addition to collecting and analyzing new cases of insider theft of IP, our future work in this area will explore another critical question with regard to patterns of IP theft: How can organizations distinguish between insider theft activity and legitimate employee activity? An answer to this question will help an organization use the mitigation pattern cost effectively by reducing the chance of false positives during the review process. Evaluating the effectiveness of individual mitigation patterns is just one aspect of our work to help organizations bolster their defenses against malicious activity by insiders. We view our pattern work as a way of helping organizations integrate what we’ve learned into their existing enterprise architecture and practices. The first post in this series described our work to protect next-generation DoD enterprise systems against insider threats by capturing, validating, and applying enterprise architectural patterns.   In our upcoming post in this series, fellow researcher David Mundie will describe a pattern language that we’ve developed to help software architects better understand how to apply patterns in sequence to design a system that provides balanced protection against malicious activity by insiders. Additional Resources: To read the SEI technical report, A Pattern for Increased Monitoring for Intellectual Property Theft by Departing Insiders, please visitwww.sei.cmu.edu/reports/12tr008.pdf To read the SEI technical note, Insider Threat Control: Using Centralized Logging to Detect Data Exfiltration Near Insider Termination, please visit www.cert.org/archive/pdf/11tn024.pdf. To read the CERT Insider Threat blog, please visit www.cert.org/blogs/insider_threat/

By Suzanne Miller, Senior Member of the Technical StaffAcquisition Support Program All software engineering and management practices are based on cultural and social assumptions. When adopting new practices, leaders often find mismatches between those assumptions and the realities within their organizations. The SEI has an analysis method called Readiness and Fit Analysis (RFA) that allows the profiling of a set of practices to understand their cultural assumptions and then to use the profile to support an organization in understanding its fit with the practices’ cultural assumptions.  RFA has been used for multiple technologies and sets of practices, most notably for adoption of CMMI practices. The method for using RFA and the profile that supports CMMI for Development adoption is found in Chapter 12 of CMMI Survival Guide: Just Enough Process Improvement. This blog post discusses a brief summary of the principles behind RFA and describes the SEI Acquisition Support Program’s work in extending RFA to support profiling and adoption risk identification for Department of Defense (DoD) and other highly-regulated organizations that are considering or are in the middle of adopting agile methods. One of the fundamental principles of technology adoption is that of mutual adaptation. This principle asserts that a successful technology adoption by an organization usually requires adaptation of both the technology and the organization. The technology may adapt, for example, by being configurable - allowing switching on or off of different features - or by allowing localization to a different native language. The organization may adapt by changing some of its business workflows so they are more compatible with the technology or by changing the roles of the people involved in different processes that are affected by the technology.  This blog post is the latest in a series examining our work on the adoption of agile methods in U.S. DoD settings. In July of this year, we kicked off a series that highlighted key ideas and issues associated with applying agile methods to address the challenges of complexity, exacting regulations, and schedule pressures in the DoD. When an organization adopts a new set of practices, it sees many of the same issues associated with adopting a new hardware or software technology. The SEI has observed that when adopting new practices—as when adopting new technologies—the principle of mutual adaptation applies. One of our observations has been that the closer the organization’s culture is to the implied cultural assumptions of a set of practices, the easier it is for that organization to adopt those practices. As part of our research in the adoption of agile methods in U.S. DoD settings, we have adapted the RFA profiling technique to accommodate both the typical factors used in RFA and some factors that are more uniquely associated with the DoD acquisition environment. We found that only applying the commercial profile didn’t highlight enough of the issues that we were seeing in our interviews and observations of practice. A technical note on RFA factors for agile adoption in DoD will be published at a future date. In this post, we want to present the categories and factors that we have identified so far, with the help of our interviewees and our SEI Agile Collaboration Group. This latter group consists of over a dozen DoD and other federal government acquisition practitioners, plus several DoD contractor organization representatives who are all actively adopting various relevant Agile methods in their organizations. We have characterized the following six categories to profile for readiness and fit: business and acquisition - adoption factors related to business strategy, acquisition strategy, and contracting mechanisms organizational climate - adoption factors related to sponsorship, leadership, reward systems, values, and similar "soft" issues system attributes - adoption factors related to the actual characteristics of the system(s) being developed project and customer environment - adoption factors related to project management norms, team dynamics and support structures, and customer relationships and expectations technology environment - adoption factors related to the technologies that are in place or planned to support the selected agile methods practices - a taxonomy of agile practices that is used to understand which practices an organization plans to adopt so that other factors can be calibrated around those expectations If an organization has used RFA in other settings, the factors that were found in the original RFA are scattered among the business and acquisition, organizational climate, and project and customer environment categories. Each category has a set of attributes that can be characterized by a statement that would represent what you would expect to see if you were observing a successful agile project or organization operating in relation to that attribute. For example, an attribute of business and acquisition is stated as: Oversight mechanisms are aligned with agile principles. Oversight is an aspect of acquisition that can either support or disable an agile project. Alignment of oversight with agile principles thus reduces the risk that oversight will be counterproductive. The remainder of this blog posting describes key factors in the business and acquisition category. In future posts, we will explore other categories of factors that deal with issues that cause different challenges to adoption of agile mthods. The Business and Acquisition Category This category covers issues related to an organization’s business strategy or mission and some specific factors related to acquisition and contracting. Business strategy is an important fit element because many organization values and principles are tied to the strategy. If the strategy changes, the organization’s values may change, creating either a better or worse fit environment for a particular set of practices. Similarly, in DoD settings, certain contracting approaches are more aligned with particular sets of values and practices, and changing the way a contract is formulated can have a significant impact on the values and practices that will be needed to execute that contract. The following list has both a short title that summarizes the statement and a statement that provides a condition or behavior found in an organization successfully using engineering and management methods consistent with agile principles as published in the Agile Manifesto. Clear Program Goals.  Business or program goals are clear and reflect stakeholder concerns.From an agile methods perspective, the organization’s mission or business goals are one of the touchpoints for decision making. If they are not clear—or if they do not adequately reflect the concerns of the organization’s stakeholders—then lower level decision-making runs the risk of being misaligned with the organization’s focus. Defined Success Strategies. Success strategies (e.g., roadmaps, product portfolios, etc.) are defined and clearly communicated.From an agile methods perspective, being clear about the roadmaps, portfolios, etc. that an organization uses to define its productivity and successful completions is a key to understanding how an individual project fits into the broader organizational mission. Project Funding Secured. Funding for the project has been secured.This factor may seem obvious and one that is a success criterion for any project, which is true. Of particular importance when applying agile methods to DoD organizations, however, is that there are multiple ways to fund and contract for information technology products and services. Some steps in the formulation of a program can be executed prior to official funding, but there are many tasks that cannot be initiated until the funding allocation process has completed. Close Stakeholder/Developer Collaboration Enabled. Mechanisms are in place in the contract and acquisition strategy to allow close collaboration between developers and other stakeholders (e.g., certification and accreditation personnel, end users, and others).The fourth principle derived from the Agile Manifesto states, "Business people and developers must work together daily throughout the project." In a commercial environment, business people includes managers of the project and end users of the product being developed. In the DoD, these roles may be in different organizations, and there are multiple business-related stakeholder roles to account for--program office personnel, information assurance, independent verification and validation agents, end users, logisticians, trainers, and others. If the acquisition strategy and associated contract vehicles create barriers to collaboration among these roles and the developer, it will be hard to achieve the performance of shoulder-to-shoulder agile implementations. Interim Delivery Enabled. Mechanisms are in place in the contract and acquisition strategy that allow for interim demonstration and delivery between official releases.The first principle derived from the Agile Manifesto states, "Our highest priority is to satisfy the customer through early and continuous delivery of valuable software." DoD contracts can specify the cadence of delivery in the Statement of Work (SOW) and also in the way they apply different standards and define line items in their Contract Data Requirements List. If a contract specifies a single delivery of the software, other mechanisms may be in place to prevent productive early demonstration and re-orienting of priorities or focus. Oversight Supports Agile Principles. Contract oversight mechanisms are aligned with agile principles.As with delivery enablement, the contract is the mechanism wherein program office technical and management oversight is specified. Contracts for large acquisition programs typically mandate document-centric capstone reviews, such as Preliminary Design Reviews (PDRs) and Critical Design Reviews (CDRs). These reviews analyze requirement, preliminary design (PDR), and detailed design (CDR) documentation; software development does not begin until after all these documents have been approved following the CDR. This linear lifecycle model is not as productive an oversight strategy for contracts employing agile methods, where contracting language enables incremental, more frequent (and less formal) progress reviews. Beyond the contract language itself, the expectations of reviewers and oversight personnel must also be set appropriately. Clear Alignment of Software Goals/Program Goals. The alignment of software-related goals with program-level goals is clear.This factor is also important in non-agile settings, but its urgency in agile settings comes from the fact that software will be available earlier to test and interact with the other parts of the system. For systems engineers unaccustomed to this early access, provisioning test beds consisting of hardware emulators and simulation environments may not get the attention needed to ensure the software part of the program can take advantage of incremental deliveries. Appropriate Contract Type. Contract type accounts for use of agile or lean methods in the program.This factor may seem obvious, but it’s actually quite a challenge for DoD program offices. Almost any contract type (firm fixed price, indefinite deliver/indefinite quality, time and materials, level of effort, cost plus incentive fee, etc.) can be used to effectively support development using agile methods. For each contract type, however, the way the agreement is framed determines how effective it will be. The contract type and the acquisition strategy must therefore be aligned to support agile methods implementation. Appropriate Lifecycle Activities. Lifecycle activities that are planned in the acquisition strategy are compatible with agile methods.It’s not enough that the contract vehicle be written correctly. It’s also important that the life cycle activities are specified in a way that can leverage the iterative and incremental nature of agile software development. For example, building test support equipment and test suites early in the life cycle is essential if test-driven development is an agile method being applied. Agile at-Scale Enabled.  The acquisition strategy takes into account the use of agile methods at the scale needed for the program.The most prevalent use to date for agile methods has been on smaller projects, but even in the DoD there have been successful projects with dozens of developers. To appropriately express the agile principles, stakeholders must consider communication mechanisms, architectural patterns, and layered management approaches. If these factors are not taken into account in the acquisition strategy, larger agile implementations may not be resourced effectively. Looking Ahead Upcoming blog entries in this series will describe factors in organizational climate, system attributes, and technology environment. We will also describe factors in the project and customer environment and practices categories that form a picture of the organization planning to adopt agile practices to help them understand where they are likely to have challenges adopting the desired practices.  From there, risk mitigation and issue management strategies can be defined to minimize the probability and/or impact of the adoption risks that have been identified. We welcome feedback and comments on both the concept and the content of the model so far. We solicit your feedback, especially if you are a practitioner that is being asked to adapt your own work to accommodate agile methods. Additional Resources For more about the readiness and fit analysis method, please visitwww.sei.cmu.edu/sos/consulting/sos/readinessandfit.cfm For more information on management and acquisition considerations in using agile methods in DoD environment, please our first two technical notes in the Agile Acquisition series: Agile Methods: Selected DoD Management and Acquisition Concernswww.sei.cmu.edu/library/abstracts/reports/11tn002.cfm An Acquisition Perspective on Product Evaluationwww.sei.cmu.edu/library/abstracts/reports/11tn007.cfm

By Kurt WallnauSenior Member of the Technical StaffResearch, Technology, and System Solutions and CERT Science of Cyber-Security For more than 10 years, scientists, researchers, and engineers used the TeraGrid supercomputer network funded by the National Science Foundation (NSF) to conduct advanced computational science. The SEI has joined a partnership of 17 organizations and helped develop the successor to the TeraGrid called the Extreme Science and Engineering Discovery Environment (XSEDE). This posting, which is the first in a multi-part series, describes our work on XSEDE that allows researchers open access—directly from their desktops—to the suite of advanced computational tools and digital resources and services provided via XSEDE. This series is not so much concerned with supercomputers and supercomputing middleware, but rather with the nature of software engineering practice at the scale of the socio-technical ecosystem. Background: Bringing Disciplined Engineering to TeraGrid From 2001 to 2011 the NSF’s network of supercomputers, services, and middleware—known as the TeraGrid—played a key role in establishing and supporting the emerging discipline of computational science.  Computational scientists used the TeraGrid to study the atomic-level structure of protein needed for sound perception, investigate how gas giant formations like Jupiter and Saturn are formed, and gain new insights into the early stages of HIV-1 infection to prevent the virus’ ability to "hijack" other healthy cells. By 2008, however, it was becoming clear to NSF’s Office of Cyberinfrastructure (OCI) that TeraGrid’s success had generated new demands from computational scientists that could not be satisfied by the existing infrastructure.  While TeraGrid had provided technologies and services to integrate the nation’s most advanced supercomputing resources, by 2008 there was a growing perception among computational scientists that not only did the national cyberinfrastructure ecosystem need to provide access to supercomputing resources, there needed to be a easier way to integrate a much broader array of digital assets and services in conjunction with much greater stability and other quality attributes in order to support top-tier research programs. In 2008, NSF/OCI issued a call for proposals for TeraGrid Phase III: eXtreme Digital (XD) Resources for Science and Engineering. As expected, the Phase III solicitation called for a continuation of and substantial improvements to core TeraGrid technologies, as well as to its outreach, education, consulting, and infrastructure-management services.  The solicitation also included something surprising—a strong emphasis on software and system engineering in Phase III.  NSF reinforced this new emphasis in a series of town hall meetings for potential submitters. Among the takeaways from those meetings, one thing was certain: The winning proposal would need disciplined engineering processes. But which processes?  It was this question that Pittsburgh Supercomputing Center director Michael Levine posed to the SEI. The SEI’s foundational role in identifying "process" as a lever for improving practice—and in achieving industrial-scale adoption of software engineering process through the Capability Maturity Model Integration (CMMI) and its many derivatives—made it a natural choice for Levine and his team.  Levine, a key leader in TeraGrid, also understood that NSF’s culture and the TeraGrid community in particular required something different than the conventional notion of "process." It was this understanding that ultimately led Levine to the SEI’s Research, Technology, and Systems Solution (RTSS) program. After a few rounds of discussion, RTSS joined a team that included the National Center for Supercomputing Applications (NCSA), Pittsburgh Supercomputing Center (PSC), Texas Advanced Computing Center (TACC), and the National Institute for Computational Sciences (NICS).  The team assembled by Towns at the NCSA ultimately developed the winning proposal. The successor to TeraGrid would now be known as the eXtreme Science and Engineering Discovery Environment (XSEDE). Our Challenge: Creating an Engineering Culture NSF decided to award one-year planning grants to the two most competitive preliminary proposals: XSEDE and XROADS. NSF had awarded the planning grant with the stipulation that the results were to be shovel-ready implementation plans, ready on Day 1. That is, the winning proposal would need to go operational with no disruption of service to any current customer of TeraGrid.  The XSEDE team recognized the following challenges of developing a credible management and technical plan: For the management plan, this meant completely working out all the details about project structure and governance. For the technical plan, this meant defining the software and system architecture needed to support immediate implementation. The SEI team believed that the challenge was of a crosscutting nature: Project governance, software and system architecture, and engineering practice must be mutually reinforcing.  Our first challenge was to persuade the XSEDE team to abandon the belief (as it turns out, a belief that was widely held by both XSEDE and XROADS teams) that "doing" software engineering was nothing more than following a process, and, by extension, the SEI could define "The Process." To overturn this mistaken belief would require persistence.  We undertook two efforts to do so: We developed a traditional System Engineering Management Plan (SEMP) that described top-level engineering processes and organizations. We piloted architecture-based engineering practices to help the architects engage internal stakeholders to decide what mattered most in the architecture. As expected, the XSEDE team was mostly indifferent to the content of the SEMP, except to note that it lacked narrative "pizazz." They were surprised, however, by the effectiveness of techniques—such as the Mission Thread Workshops led by the SEI’s Mike Gagliardi—at shedding light on the organizational implications of engineering decisions.  Most importantly, the team began to appreciate that the steps of "The Process" do not define most of what we regard as software engineering. Instead, real software engineering practices reside within these steps—the parts that often aren’t written down.  Our New Challenge: Doing it Again, In Operation With the one-year planning grant complete, in 2011 XSEDE and XROADS teams submitted and briefed their plans to the NSF Review Panel.  When asked what worried him most about the review process, John Towns (XSEDE PI) replied he was concerned that the panel would decide to fund part of each of the XSEDE and XROADS efforts. That funding model would violate the by-now integrated management, technical, engineering and governance structures that XSEDE had developed in its plan. As it turned out, Towns’ fears were partly realized when NSF announced its intent to award XD to XSEDE with the proviso that it incorporate the most meritorious elements  of the XROADS effort and team.  We were now confronted with the need to revisit planning assumptions and decisions made with a full year of deliberations—in effect to "re-plan" the effort in the span of a few short weeks. In hindsight it is clear that NSF made a wise decision in merging the XSEDE and XROADS proposals, both in terms of avoiding the schism in the community that might have arisen from a "winner-take-all" outcome, and in terms of incorporating several exciting architectural ideas introduced by XROADS, such as delivering cloud resources as an alternative to leadership class "big iron" computers, and adopting "software as a service" as a business model and product delivery channel. On the other hand, much of what we had accomplished in terms of introducing the seeds of an engineering culture in XSEDE had been thoroughly disrupted. After a year of engaging the XSEDE team on the nuances of software engineering methods, these methods had begun to pay dividends. Of course this progress relied on the skills and perspectives of just one of the two teams with which we had engaged.  Now our engineering teams would be reconstituted with people from both teams.  Not only had we potentially lost a "common sense" of software engineering, but we now had engineering teams with "competing views" of what to accomplish.   We had to reconstitute an engineering culture while simultaneously realigning our technical objectives. Moreover, because the TeraGrid effort was coming to its contractual end, we needed to do this "live," which is like having to redesign and rebuild an aircraft while in flight.  To be fair, although we knew that the merger was going to be challenging, we also knew that we were substantially further along than we had been one year earlier. Both teams had spent their planning efforts wisely, and their overall approaches were more similar than different, and where different, they were mostly complementary.  There was also something at work that is difficult to objectively quantify: The members of the XSEDE and XROADS teams were mostly well known to one another, and there was a universal desire to move beyond the competition of the past years, roll up our sleeves, and get to work on the real mission of advancing the nation’s computational science. Results after One Year What did we manage to accomplish?  Did we build a redesigned aircraft while in mid-flight without crashing and burning?  After one year, we can safely claim a good measure of success, the results of which are summarized in the figure below. Figure 1 shows a portion of the XSEDE program that is engaged in engineering work on a day-to-day basis, as described below: The SEI helped introduce an explicit and rigorous treatment of software architecture as a keystone of the XSEDE engineering approach. We reinforced concrete practices for documenting software architecture and for understanding, documenting, and using quality attributes to design and evaluate how well an architecture serves its strategic aims. The SEI helped introduce a range of concrete software development practices, beginning with unit, integration, and acceptance testing, which had been completely lacking in TeraGrid.  We augmented these practices with a rudimentary software lifecycle that included basic development phases and verification gates for detailed design and various test phases. The SEI also helped XSEDE leadership evolve to a more realistic and nuanced understanding of "process."  XSEDE had evolved from its early quest for The Process to a program that appreciated the importance of establishing an engineering culture, as well as the need for that culture to define and evolve its own engineering processes. It is fair to say that on Day 1 of the XSEDE/XROADS union, none of the practice areas reflected in the graphic and outlined above were in place, and indeed, few if any had any legacy in TeraGrid. This substantial progress is due not to the SEI but to XSEDE leadership and staff.  Our plans for Year 2 combine two main thrusts Consolidation by documenting and applying practices and making small improvements as needed. Planning for new and potentially groundbreaking practices to scale architectural design to allow contributions from hundreds to thousands of members of the XSEDE community and from the larger computational science ecosystem of science grids, communities, and technology providers. Upcoming Topics in the XSEDE Thread My next post will explore establishing an engineering culture within an emerging social-technical system. Other topics that I plan to post about are: What are the quality attributes of a socio-technical system that will produce not just one intended system but many systems? Can the web of cultural norms and incentives that make up a socio-technical system be described? Can their mutual alignments be measured or influenced?  Can these measures predict the quality of engineering decisions? What is the role of an integrating technology platform in the formation and maintenance of a socio-technical ecosystem?  How can a socio-technical ecosystem be established to design the platform for another ecosystem? (In fact, this is the XSEDE case.) Are persistent institutions required to sustain engineering practice in a socio-technical ecosystem?  Can a practice be sustained without such institutions? In addition to the above posts, I’d like to enlist the help of other contributors to the XSEDE effort, as well as researchers of engineering practices in and for socio-technical systems.  Please feel free to leave your thoughts in the comments section below. Due Credit Before Closing While most of the credit belongs to XSEDE leadership and staff, I will take a few moments here to give credit to SEI colleagues who have contributed to this result: Mike Gagliardi introduced XSEDE to large-scale architecture thinking though Mission Thread Workshops. Joe Batman demonstrated mature systems engineering practices by showing the connectedness of engineering decision-making and project governance. Paul Clements, and later Felix Bachmann, introduced disciplined approaches for documenting architecture in a way that serves stakeholder interests. Scott Hissam defined the Software Development and Integration (SDI) approach (see Figure 1) to integration testing, acceptance testing, and version management, and invaluable assistance in environment integration. Rhonda Brown and Mike Konrad drew on their experience in CMMI, TSP and more recently, the combination of TSP with software architecture practices, to improve the agility of SDI’s overall approach. Additional Resources For more information about the XSEDE project, please visit https://www.xsede.org/.For more information about the SEI’s work in architecture centric engineering (ACE), please visit www.sei.cmu.edu/about/organization/rtss/ace.cfmFor more information about the SEI’s work in system-of-systems engineering, please visit www.sei.cmu.edu/sos/.For more information about the SEI’s work in ultra-large-scale systems, please visit www.sei.cmu.edu/uls/.

By Bill Pollak, Transition Manager Research, Technology, & System Solutions A search on the term "software architecture" on the web as it existed in 1992 yielded 88,700 results. In May, during a panel providing a 20-year retrospective on software architecture hosted at the SEI Architecture Technology User Network (SATURN) conference, moderator Rick Kazman noted that on the day of the panel discussion—May 9, 2012— that same search yielded 2,380,000 results. This 30-fold increase stems from various factors, including the steady growth in system complexity, the increased awareness of the importance of software architecture on system quality attributes, and the quality and impact of efforts by the SEI and other groups conducting research and transition activities on software architecture. This blog posting—the first in a series—provides a lightly edited transcription of the presentation of the first panelist, Linda Northrop, director of the SEI’s Research, Technology, & System Solutions (RTSS) Program at the SEI, who provided an overview of the evolution of software architecture work at the SEI during the past twenty years. 20 Years of Architecture at the SEI as Presented by Linda Northrop In 1992, the SEI was working on structural modeling and applying this concept to flight simulators for the purpose of re-architecting Air Force systems that had become brittle and unmodifiable. Others at the SEI were then studying what we called "quality attributes," looking at such attributes as performance, reliability, and modifiability and trying to determine what the underlying analytic models were. Soon after, we started a software architecture project at the SEI that studied architecture description languages, definitions of software architecture, and ways to evaluate software architectures. Around 1995, we were asked by the U.S. Department of Defense (DoD) to write something about software architecture. In response, Paul Clements and I wrote Software Architecture: An Executive Overview. Then Len Bass, Paul Clements, and Rick Kazman wrote the first edition of the book Software Architecture in Practice. We considered a number of existing definitions of software architecture, but in the end formulated our own:  The software architecture of a program or computing system is the structure or structures of the system, which comprise software elements, the externally visible properties of those elements, and the relationships among them. Architecture is concerned with the public side of interfaces; private details of elements—details having to do solely with internal implementation—are not architectural. This work constituted the beginning of a cohesive body of SEI research and transition efforts focused on software architecture.   We started our research with architecture evaluation, because we reasoned that by evaluating software architectures early, we could alleviate subsequent testing and integration problems. We developed the Architecture Tradeoff Analysis Method (ATAM ) and from this grew many other methods based on questions that people asked during our application of the ATAM , such as, "What are the significant quality attributes?" (Quality Attribute Workshop) and "How should an architecture be documented?" (Views & Beyond method). Over time, we created a repertoire of methods, technologies, and techniques that grew into a discipline that we now call architecture-centric engineering (ACE). Characteristics of ACE discipline include an explicit focus on quality attributes direct linkage of architecture to business and mission goals explicit involvement of system stakeholders a grounding in state-of-the-art quality attribute models and reasoning frameworks and a firm foundation in how these frameworks can guide architectural decisions Other organizations, such as Bosch, Raytheon, Siemens, and Lockheed Martin, were also exploring and embracing practices in software architecture during these years. In 2006, I led a study on ultra-large-scale (ULS) systems, which we defined as a system at least one of whose dimensions is of such a large scale that constructing the system using development processes and techniques prevailing at the start of the 21st century is problematic. This study included panelist Doug Schmidt and a number of other experts from an assortment of disciplines. We studied the characteristics of software-reliant systems of the future and learned that these systems would be inherently decentralized that you’d never be able to fully know the diverse range of requirements, many of which will conflict that they’d be continuously evolved, developed, and deployed -- in other words they can’t be completed shutdown and restarted they’d be heterogeneous and developed and governed by many organizations and stakeholders failure will be a normal part of system operation people would be a part of the system, as opposed to being "mere" users. ULS systems were a complete mind change for me. I was thinking that, if we are going to architect these systems effectively, what aspects of our current body of knowledge on software architecture could be applied and what new breakthroughs would be required? Much of what we described in the study report has come to pass; there are ULS systems today with the characteristics we identified. In addition, several game-changing technologies, such as cloud computing, social media, mobile computing, and multicore have emerged that can be used to improve ULS system qualities, as well as to gain competitive advantage. Many people here at the conference have talked about what’s next. Jeromy Carriere said that you can’t nail everything down about the architecture when you have an inherently distributed team and everyone has to operate autonomously, but you can agree on the hard decisions and consistent boundaries. So in the future we have to understand What are appropriate architectures for socio-technical, cyber-physical systems (especially ones that form critical parts of systems-of-systems and ULS systems)? What are quality attributes of these systems that apply today? We need to consider attributes like privacy and we need to incentivize humans in these system to do the right things. How to get stakeholders involved when they are so numerous and widely distributed we can’t get them in a room to hold a Quality Attribute Workshop or conduct an ATAM? Perhaps we can exploit social media - "crowdsource" the applications of our methods.    Our charge as software architects is to take all the ULS system characteristics and challenges and figure out how we architect successfully in that space. What’s Next In my next blog posting, I’ll provide a transcript of the presentation by Douglas C. Schmidt, former CTO of the SEI and currently a professor of computer science at Vanderbilt University, who will talk about advances in software architecture practice for distributed real-time and embedded systems.  Subsequent blog postings will present the talks by Ian Gorton research and development R&D lead Data Intensive Computing at Pacific Northwest National Lab; Bob Schwanke of Siemens Corporate Research; and Jeromy Carrière, chief architect at X.commerce, a subsidiary of eBay. Additional Resources The abstract and slides for Linda Northrop’s talk are available at www.sei.cmu.edu/library/abstracts/presentations/northrop-panel-saturn2012.cfm. The Ultra-Large-Scale Systems report is available at www.sei.cmu.edu/uls/. For more information about the book Software Architecture in Practice, please visitwww.sei.cmu.edu/library/abstracts/books/0321154959.cfm. SATURN 2013, presented in collaboration with IEEE Software magazine, will be held April 29 through May 3, 2013 in Minneapolis, Minnesota. For more information see the SATURN 2013 software architecture conference website at www.sei.cmu.edu/saturn/2013.