By Ipek OzkayaSenior Member of the Technical StaffResearch, Technology, and System Solutions Managing technical debt, which refers to the rework and degraded quality resulting from overly hasty delivery of software capabilities to users, is an increasingly critical aspect of producing cost-effective, timely, and high-quality software products. A delicate balance is needed between the desire to release new software capabilities rapidly to satisfy users and the desire to practice sound software engineering that reduces rework. A previous post described the practice of strategically managing technical debt related to software architecture, which involves deliberately postponing implementation of some architectural design choices to accelerate delivery of the system today and then rearchitecting at a later time. This blog post extends our prior post by discussing how an architecture-focused analysis approach helps manage technical debt by enabling software engineers to decide the best time to rearchitect—in other words, to pay down the technical debt. Our architecture-focused approach for managing technical debt is part of the SEI’s ongoing research agenda  on Agile architecting, which aims to improve the integration of architecture practices within Agile software development methods. This project is investigating which measures a software development team can apply to effectively monitor changing qualities of software, such as degrading modifiability and extensibility of the system at each iteration in an iterative and incremental lifecycle like Agile, for example. We initially investigated a particular metric—propagation cost—that measures the percentage of system elements affected when a change is made to a randomly chosen element. A high propagation cost is an indication of tight coupling, such that when a change is made in the system, many parts of the system will be affected. We focused on propagation cost due to the rich set of existing static analysis techniques that evaluate code and design quality by measuring software coupling and cohesion, such as whether there are cycles within parts of the software system, whether there is code duplication, and so on.  Most existing static analysis techniques focus on code quality and code-level technical debt. For example, a high percentage of duplicate code and cycles in the code indicates a high level of technical debt. In contrast, we applied propagation cost at the architecture level to calculate the impact of the dependencies looking at architectural elements, rather than calculating every dependency between different classes. The goal of this approach is to reduce complexity and provide insights, even when an implementation is not complete. Our work explores the relationship between propagation cost and technical debt. In particular, we use propagation cost as one indication of increasing technical debt. We assess the potentially increasing rework—which is effectively the impact of paying back technical debt—based on monitoring increasing propagation cost of the system. Reasoning about quality by modeling rework as a proxy for technical debt requires objective—and repeatable—representation of architectural properties (such as module dependencies and changing interfaces) for the model to work. We therefore modeled the dependencies of the architectural elements by means of a technique called design structure matrices (DSMs). DSMs can be used to visualize which elements use or depend on others at each iteration and to calculate propagation cost. Our research on the propagation cost metric examined a real-world case study regarding a building automation control system. The research team had at its disposal the software engineering artifacts of the project, including the software architecture, code, functional and quality attribute requirements, and project management plan. Using these artifacts we generated the design structure matrix of the system at each of the project iterations and completed "what-if" studies. This what-if analysis focused on calculating accumulating rework based on allocating functionality and architectural tasks with different orders to the iterations. We applied propagation cost measurement to calculate the rework and to assess what the team could have done differently in terms of delivering functionality at different times and calculated the overall impact on rework and the lifecycle costs.  Our goal was to demonstrate how different allocations of both functionality and critical architectural tasks to iterations can enable developers to respond to changes quicker and use technical debt to their advantage by monitoring the accruing rework. The results of our studies showed that focusing on architectural dependencies, as well as using propagation cost as a proxy to indicate the level of changing complexity and rework, provided good insight into quantifying technical debt at the architecture level. This insight helps software architects, developers, and managers decide the best time to pay back technical debt or determine if technical debt is accumulating in the first place. To make these measurement and analysis techniques practical, however, they should be integrated seamlessly into engineers’ integrated development environments. For example, tools should have the ability to group classes into module view architectural elements and specify design rules, such as one element can or cannot access another element. New generation tools, such as Lattix, Sonargraph, and Structure101, are starting to explore such issues, though there is still room for improvement. In some instances, architectural dependencies should be also integrated with architectural design decisions. For example, when using a mediator to decouple interfaces from the data model, the mediator communicates with all the interface and data model elements. When applying the propagation cost metric consistently across the system—including all dependencies to and from the controller element—a high propagation cost emerges. This high cost indicates a greater risk of technical debt and change propagation, potentially requiring rework when new features must be added. Although higher propagation costs are generally associated with higher risk, in this case introducing a mediator to decouple the data model and the interface may be a good architectural decision since it localizes the changes. From a reliability perspective, however, the controller is a single point of failure. So in this case high propagation cost may not necessarily be negative from a modifiability standpoint, but it is still a reliability risk. Our studies revealed that enhancing propagation cost measurements with architectural information provides more insightful analysis of the actual implications of technical debt and rework. Studying rework using propagation cost helps improve the integration of architecture practices within Agile software development methods. For example, when teams are developing software in an Agile context, they typically embrace Scrum as a project-management technique. It is often hard for teams to determine how to subdivide large architectural tasks and allocate them to small two- to four- week sprints. Our research demonstrated that by focusing on iteration-to-iteration analysis— rather than trying to time box distribution of functionality to sprints where each time box/sprint has the same duration—it is possible to show customers how the quality of the software changes with each release, such as how increasing propagation cost could impact rework. Our next steps are to examine the scalability of assessing rework by focusing on architecture metrics.  We developed two real-life case studies using dependency analysis to operationalize the measurement of propagation cost. While our approach works fine with 100- to 200 software architecture elements, we are now evaluating how well our approach scales up to a higher number of elements. When our analysis focused on software architecture as opposed to code to quantify technical debt, we observed a magnitude of reduction of dependencies analyzed from about 200 to two dozen, and we were able to pinpoint the potential emerging rework, which is a significant reduction of complexity. Architecture-level analysis of technical debt enables a team to gauge the status of the system quickly and make decisions on whether to rework the system or not. Code-level analysis then enables the team to define specific tasks for developers. Our research on an architecture-focused measurement framework for managing technical debt is informed by real-world examples gathered from Technical Debt Workshops.  These workshops engage practitioners and researchers in an ongoing dialogue to improve the state of techniques for managing technical debt. The 2011 Managing Technical Debt Workshop co-located with the International Conference on Software Engineering (ICSE) revealed an increasing interest in managing technical debt proactively. As a result, we will conduct a third workshop—again collocated with ICSE on June 4, 2012. Our research team will also guest-edit the November/December 2012 issue of IEEE Software  on the same theme and is accepting papers until April 1, 2012. We welcome any individuals who have experiences in this area to submit a paper for consideration in IEEE Software or the 3rd International Workshop on Managing Technical Debt. This research was conducted in collaboration with Dr. Philippe Kruchten, professor of software engineering at the University of British Columbia, and Raghu Sangwan, associate professor of software engineering at Penn State University, and with support from Lattix, a leading provider of software architecture management solutions. Additional Resources N. Brown, P. Kruchten, R. Nord, and I. Ozkaya. Managing technical debt in software development: report on the 2nd international workshop on managing technical debt, held at ICSE 2011. ACM SIGSOFT Software Engineering Notes 36 (5): 33-35 (2011). N. Brown, Philippe Kruchten, R. Nord, and I. Ozkaya. Quantifying the Value of Architecting Within Agile Software Development via Technical Debt Analysis. 2011. N. Brown, R. Nord, I. Ozkaya, and M. Pais. Analysis and Management of Architectural Dependencies in Iterative Release Planning. In Proceedings of the 2011 Ninth Working IEEE/IFIP Conference on Software Architecture (WICSA '11). IEEE Computer Society, 103-112.

By Mary Ann LaphamSenior Member of the Technical StaffAcquisition Support Program Over the past several years, the SEI has explored the use of Agile methods in DoD environments, focusing on both if and when they are suitable and how to use them most effectively when they are suitable. Our research has approached the topic of Agile methods both from an acquisition and a technical perspective. Stephany Bellomo described some of our experiences in previous blog posts What is Agile? and Building a Foundation for Agile. This post summarizes a project the SEI has undertaken to review and study Agile approaches, with the goal of developing guidance for their effective application in DoD environments. The SEI’s Agile project began in 2009 in response to our recognition of the growing awareness that Agile methods help alleviate key challenges facing the DoD, such as providing competitive capabilities to warfighters in a timely manner that minimizes collateral damage and loss of lives and property. We also observed an emerging consensus that Agile methods can be applied to create systems who functionality and quality attributes can be adapted more readily over time, which may help reduce total ownership costs over long acquisition program lifecycles. Within this context, the primary activities of our project included reviewing relevant literature, interviewing programs that are using or have used Agile methods, identifying criteria to be used to determine if a program is a candidate for Agile methods and what risks exist for implementing the Agile methods, and creating guidelines to be used in implementing agile methods. We initially focused on the question of whether Agile could be used in DoD acquisition programs, which historically follow the DoD 5000 series of guidelines that have been associated with so-called waterfall methods. An early finding of our project was that no prohibitions preclude the use of Agile in the DoD 5000 series. This result is important since some skeptics have asserted that Agile is not suited for the DoD due to inherent conflicts between Agile methods and DoD policies and regulations. Given that there is no one size fits all Agile method, however, we also found that implementations of Agile methods must be tailored to fit the situation and context. In other words, Agile is not a silver bullet. Transitioning DoD systems—and their associated socio-technical ecosystems—to Agile therefore requires considerable work from DoD agencies and the defense industry base, and is not without hurdles. For example, we found that adapting Agile into the DoD acquisition life cycle presents unique challenges and opportunities. The challenges are in meeting existing DoD milestone and regulatory criteria when there is little if any guidance available on how to do so when using Agile methods. The opportunities, however, provide the capability to accomplish development with frequent results. Other hurdles identified by DoD programs we interviewed include providing the right team environment allowing access to end users determining how to train and coach the government staff instituting suitable oversight methods adapting rewards and incentives to the Agile environment adjusting to a different team structure These hurdles are due to a different approach than business as usual and a general lack of specific training and guidance on Agile concepts and approaches within the DoD. Overcoming these hurdles requires changes to the waterfall-centric organizational culture that is common within DoD acquisition programs. These culture changes also require mindset changes because the underlying paradigm for implementing Agile methods is different from that used for the waterfall method. After studying the topics described above to gain a preliminary understanding of the use of Agile within the DoD, we then studied other management, acquisition, and technical topics, as described below. Agile Management and Acquisition Topics Our project also is studying the following management and acquisition topics relevant to the effective adoption of Agile methods in the DoD: Being Agile in the DoD. Agile methods provide promising techniques for streamlining the acquisition process for systems within the DoD. To meet the challenges of adopting Agile methods, however, DoD program management offices must take specific actions to assist in Agile adoption and even enable it. For example, to ensure successful Agile adoption, DoD organizations must plan for it, train for it, anticipate changes in their environments and business models to ensure the benefits of Agile become a reality. Managing and contracting for Agile programs. Managing large-scale, complex software-reliant systems is always hard, but management in Agile programs takes on some added dimensions. For example, program managers not only must be leaders, they must also be coaches, expeditors, and champions. If they do not personally perform these roles, someone in their organizations must responsible for them. These additional roles are needed due to the paradigm associated with using Agile methods and the lack of any significant experience by current DoD personnel in that arena. A particular management concern is the selection and implementation of appropriate contracting vehicles to support the types of practices that successful Agile projects exhibit. Technical milestone reviews. One sticking point in employing Agile methods is how to accommodate large capstone events, such as the preliminary design review (PDR), critical design review (CDR), and others. While many concerns exist in this area, it’s important to focus on the purpose of holding these reviews in the first place: to evaluate progress on and/or review specific aspects of the proposed software solution. Expectations and criteria must therefore reflect the level and type of documentation that is acceptable for the milestone, which is no different from business as usual. The key, however, is to define the level and type of documentation required for the specific program while working within an Agile environment. Estimating in DoD Agile acquisition. Estimation done on Agile projects is typically not the same as the traditional methods used on legacy systems within DoD. Traditional methods tend to focus on estimating details up front; these details are then modified as more information is obtained. In contrast, Agile estimates are often "just-in-time," with high-level estimates that are refined to create detailed estimates as knowledge of the requirements matures. Some tools within the traditional estimation community are now adding modules to address Agile estimation. Moving toward adopting Agile practices. Change is hard—especially for large DoD ecosystems—and understanding the scope of changes is essential. Organizational change methods must be employed to help DoD organizations successfully adapt to applying Agile. There are multiple adoption factors (such as business strategy, reward system, sponsorship, values, skills, structure, history, and work practices) that must all be addressed. Change-management best practices include understanding the adopter population, understanding the cycle of change, understanding the adoption risks, and building transition mechanisms to mitigate adoption risks. Organizations we studied that had successfully adopted Agile methods typically achieved the following goals: Found and nurtured good sponsors for Agile adoption Understood the adoption population they were dealing with Conducted a readiness assessment that addressed organizational and cultural issues Analyzed what adoption support mechanisms were needed for a particular context and built or acquired them before proceeding too far into an Agile adoption SEI Agile work continues and the following additional documents are—or will soon be—available: Case Study of Successful Use of Agile Methods in DoD: Patriot Excalibur 2011-TN-019 Agile Methods: Changing the Viewpoint of Government Technical Evaluation 2011-TN-026 A Closer Look at 804: A Summary of Considerations for DoD Program Managers 2011 SR-015 "DoD Agile Adoption—Necessary Considerations, Concerns, and Changes" in the Jan/Feb 2012 issue of CrossTalk In addition, the SEI has created an Agile Collaboration Group to advise, review, enhance, and validate SEI acquisition work. The SEI is working with this group to create, calibrate, and validate a contingency model that will help acquisition professionals determine when to use Agile techniques, as well as how to identify potential risks if Agile methods are adopted. We are also creating guidelines that summarize best practices and instruct users of Agile methods on how to apply these methods effectively in DoD environments. Agile Technical Topics Agile software development has historically succeeded in small-scale (largely IT-based) commercial environments due largely to its easy-to-apply practices for tracking project status and allocating the development resources to those activities that deliver the most potential customer value. A key technical challenge for DoD projects, however, involves balancing the short-term and long-term needs. In particular, the cliché "You aren’t gonna need it (YAGNI)" is a principle in eXtreme Programming (XP) that implies developers should not add functionality until it is necessary, thereby eliminating a considerable amount of unused code in a system. The YAGNI principle rarely seems to apply, however, in large-scale DoD environments, where systems must operate for decades with continual flux with respect to evolving requirements, technology upgrades, new partners, and different contractors. The SEI is conducting the following technical work on successfully creating and applying Agile methods for the DoD: Agile at scale. This work focuses on providing methods and techniques for applying Agile software development practices for large-scale DoD programs, with improved visibility into the release plan and the quality of the system. One of our activities to address the use of Agile development at scale was conducting a field study with organizations that deal with the challenges of Agile and architecture practices at scale. Based on our observations, we developed a readiness, best practices, and risk analysis technique. Striking the proper balance between developing the system and the architecture in an agile manner while providing enhancement agility for maintaining the system is key to success. We observed that tactics that help organizations to succeed within an Agile environment include paying close attention to architecture-centric practices where a balance of feature development and architecture development is achieved. Technical debt. Our work in technical debt analysis again focuses on architecture and looks at strategically incurring technical debt (such as applying architectural short-cuts) to improve agility in the short-term. This work focuses on developing techniques to monitor and respond to emerging rework, as well as the need to refactor or rearchitect the system to pay back the debt. The need to refactor or rearchitect DoD systems arises in several ways. For instance, system quality degradation, such as unacceptable end-to-end performance, might require refactoring. Such quality degradation-related rework can appear if the development teams focused solely on feature-oriented decomposition of the system to deliver features at early iterations, but didn’t provide the necessary architecture for the infrastructure in a timely manner. Refactoring would require the restructuring of the existing body of code to alter its internal structure (architecture) but not change its external behavior to address the decrease in quality. Modeling decision impact on agile development. Our work also provides guidance and techniques that enhance the applicability of mainstream Agile and lean software development methods to DoD stakeholders by balancing their acquisition and technical needs. In a recently started project, for example, we are investigating acquisition and architecture activities during the pre-engineering manufacturing and development phase of the acquisition lifecycle. This work closely examines modeling decision dependencies and analyzes their impact on the ability to conduct effective Agile system development. This work targets the perspective of reducing integration risks in large-scale DoD systems. In summary, our projects have found that Agile methods can indeed provide both tactical and strategic benefits in the DoD. The tactical benefits of lower cost, on-time delivery, and increasing quality are clearly important as the DoD places a growing emphasis on greater efficiency in its acquisition processes. The strategic benefits of responsiveness and more rapid adaptability to the current situation, however, may be of even greater value in today’s world, where the DoD must get results faster and be better aligned with changing needs to prepare for an uncertain future. As our work progresses, we will periodically post our progress, ask questions, and request feedback. If you have any questions or feedback on the current work, please post in the comments below. Additional Resources Please see the following SEI technical reports and notes for more on Agile development:Considerations for Using Agile in DoD Acquisition Agile Methods: Selected DoD Management and Acquisition Concerns  Documenting Software Architectures in an Agile World CMMI or Agile: Why not Embrace Both!   Incorporating Security Quality Requirements Engineering (SQUARE) into Standard Lifecycle Models  Secure Software Development Life Cycle Processes—A Technology Scouting Report Integrating Software-Architecture-Centric Methods into Extreme Programming (XP)

By Douglas C. SchmidtVisiting Scientist We use the SEI Blog to inform you about the latest work at the SEI, so this week I'm summarizing some video presentations recently posted to the SEI website from the SEI Technologies Forum. This virtual event held in late 2011 brought together participants from more than 50 countries to engage with SEI researchers on a sample of our latest work, including cloud computing, insider threat, Agile development, software architecture, security, measurement, process improvement, and acquisition dynamics. This post includes a description of all the video presentations from the first event, along with links where you can view the full presentations on the SEI website. Paul Nielsen, director of the SEI, gave the opening remarks, which summarized the presentations in the SEI Technologies Forum, focusing on the SEI’s leadership role in software, security, and resiliency technologies and methods that help address the complexities of software-reliant systems. You can watch the opening presentation here. My presentation described the SEI’s strategic plan to advance the practice of software engineering for the DoD, federal agencies, industry, and academia through research and technology transition. I motivated and summarized the following four major areas of software engineering and cyber security work at the SEI: Innovating software for competitive advantage. This area focuses on producing innovations that revolutionize development of assured software-reliant systems to maintain the U.S. competitive edge in software technologies vital to national security. Securing the cyber infrastructure. This area focuses on enabling informed trust and confidence in using information and communication technology to ensure a securely connected world to protect and sustain vital U.S. cyber assets and services in the face of full-spectrum attacks from sophisticated adversaries. Advancing disciplined methods for engineering software. This area focuses on improving the availability, affordability, and sustainability of software-reliant systems through data-driven models, measurement, and management methods to reduce the cost, acquisition time, and risk of our major defense acquisition programs. Accelerating assured software delivery and sustainment for the mission. This area focuses on ensuring predictable mission performance in the acquisition, operation, and sustainment of software-reliant systems to expedite delivery of technical capabilities to win the current fight. You can watch a video of my presentation here. The remainder of this blog posting summarizes the forum presentations, which are grouped under the four major research areas outlined above. Innovating Software for Competitive Advantage The presentation on Architectural Implications of Cloud Computing by Grace Lewis defined cloud computing, explored different types of cloud computing environments, and described the drivers and barriers for cloud computing adoption. It also focused on examples of key cloud architecture and design decisions, such as data location and synchronization, user authentication models, and multi-tenancy support. This topic is important since cloud computing is being adopted by commercial, government, and Department of Defense (DoD) organizations, driven by a need to reduce the operational cost of their information technology resources. From an engineering perspective, cloud computing is a distributed computing paradigm that focuses on providing a wide range of users with distributed access to virtualized hardware and/or software infrastructure over the internet. From a business perspective, it is the availability of computing resources that are scalable and billed on a usage basis (as opposed to acquired resources) that lead to potential cost savings in IT infrastructure. From a software architecture perspective, having resources in the cloud means that some elements of the software system will be outside the organization, and the control over these elements depends on technical aspects such as the provided resource interface, and on business aspects such as the service-level agreement (SLA) with the resource provider. Systems must therefore be designed and architected to account for lack of full control over important quality attributes. You can watch a video of Grace’s presentation here. Ipek Ozkaya made a presentation on Agile Development and Architecture: Understanding Scale and Risk. This presentation examined tactics that can help identify and mitigate key risks of large-scale, complex software development when there is a need to use Agile development and architecture-centric practices in concert.  This topic is important because Agile software development and software architecture practices have received increasing attention from both industry and government over the past decade. The complementary nature of Agile development and software architecture practices is also increasingly better recognized and appreciated. Applying Agile development with a concurrent focus on architecture, however, is still experimental and experiential rather than a proven practice based on sound engineering techniques. This presentation described how SEI researchers are helping organizations using Agile techniques deal with increased system software size and increased complexity in orchestrating larger engineering and development teams, to ensure that the systems they develop will be viable in the market for decades. You can watch a video of Ipek’s presentation here. Securing the Cyber Infrastructure A presentation by Randy Trzeciak on The Insider Threat: Lessons Learned from Actual Insider Attacks described the technical and behavioral aspects of insider threats, focusing on the types of insiders who committed the crimes, their motivation, organizational issues surrounding the incidents, methods of carrying out the attacks, impacts, and precursors that could have served as indicators to organization in preventing incidents or detecting them earlier. It also conveys the complex interactions, relative degree of risk, and unintended consequences of policies, practices, technology, insider psychological issues, and organizational culture over time. This presentation stemmed from a decade of work by the Insider Threat Center at CERT, which has been researching insider threats since 2001 and has built an extensive library and comprehensive database containing more than 700 actual cases of insider cybercrimes. This presentation describes findings from our analysis of three primary types of insider cybercrimes: IT sabotage, theft of information, and fraud. You can watch a video of Randy’s presentation here. The Smart Grid Maturity Model: A Vision for the Future of Smart Grid presentation by David White offered insight into the past year’s use of the Smart Grid Maturity Model (SGMM), which is a management tool for the utility industry to plan a reliable, secure energy supply that is vital to our economy, our security, and our well-being. The smart grid represents a new framework for improved management of electricity generation, transmission, and distribution. With the support of the U.S. Department of Energy, the SEI is the steward of the SGMM. This presentation described the release of the SGMM V1.2 Product Suite and showed how utilities are working with the model. As more utilities around the globe participate and the SGMM experience base grows, the SGMM has become an increasingly valuable resource for helping inform the industry’s smart grid transformation. You can watch a video of David’s presentation here. Julia Allen’s presentation was on Measuring Operational Resilience. This presentation suggested the strategic measures for an organization’s an operational resilience management (ORM) program, which defines an organization’s strategic resilience objectives (such as ensuring continuity of critical services in the presence of a disruptive event) and resilience activities (such as the development and testing of service continuity plans). Traditional operational security metrics such as number of machines patched, vulnerability scan results, number of incidents, and number of staff trained are easy to collect and can be useful. If an organization’s objectives are to inform decisions, affect behavior, and determine control effectiveness in support of business objectives, however, they must consider a set of more strategic resilience measures. These ten strategic measures derive from lower-level measures at the CERT Resilience Management Model (RMM) process area level, including average incident cost by root cause type and number of breaches of confidentially and privacy of customer information assets resulting from violations of provider access control policies. You can see a video of Julia’s presentation here. Advancing Disciplined Methods for Engineering Software CMMI-SVC: The Strategic Landscape for Service, by Eileen Forrester, described the current state of CMMI for Services (CMMI-SVC).  It also explored the larger strategic choices available to organizations in markets where superior service can improve work and business results. CMMI-SVC is important because the global economy is increasingly based on services, rather than manufacturing or trading of tangible goods. Even the development of goods and systems increasingly takes on the character of services. Innovative CMMI-SVC approaches that are already working in the United States, Latin America, Asia, and Europe can be tailored to meet the needs of other organizations and markets. You can watch a video of Eileen’s presentation here. James McHale gave A Brief Survey of the Team Software Process (TSP). This presentation briefly described training and introduction of TSP practices, including the Personal Software Process (PSP), the results and benefit potentials inherent in the methods, and the common use of TSP methods in combination with other popular practices, including Agile (Scrum, TDD, XP), architecture, secure coding, RUP, Six Sigma, and CMMI. TSP has been identified as one of the most effective practices for software developers by Capers Jones in his 2010 book Software Engineering Best Practices. You can see a video of James’s presentation here. Accelerating Assured Software Delivery and Sustainment for the Mission The presentation on Software Acquisition Program Dynamics by William ("Bill") Novak described analysis the SEI is doing on data collected from more than 100 independent technical assessments of software-reliant acquisition programs. This analysis has produced insights into the most common ways that acquisition programs encounter difficulties. Programs regularly experience recurring cost, schedule, and quality failures, and progress and outcomes often appear to be unpredictable and unmanageable. Moreover, many acquisition leaders and staffers neither recognize these recurring issues nor realize that known solutions exist for many of these problems. This presentation explains how the SEI is working to mitigate the effects of misaligned acquisition program organizational incentives and adverse software-reliant acquisition structural dynamics by improving program staff decision-making. To do this, SEI researchers are modeling and analyzing both the adverse acquisition dynamics that we have encountered in actual programs, as well as candidate solutions to resolve those dynamics. You can watch a video of Bill’s presentation here. Next Event February 28 A second virtual event, Architecting Software the SEI Way, is planned for February 28.  This event focuses on moving toward using architecture practices more effectively to build better systems more efficiently and productively by understanding the fundamentalsof software architecture, improving practice through architecture evaluation guidelines, and bridging technical and business goals by applying architecture methods to analyze, and evaluate enterprise software architectures. We look forward to "seeing" you there.  If you have any questions or thoughts on any of the presentations please feel free to leave your comments below. Additional Resources SEI Technologies ForumArchitecting Software the SEI Way

By Dave Zubrow, Manager Software Engineering Measurement and Analysis Initiative The SEI has been actively engaged in defining and studying high maturity software engineering practices for several years.  Levels 4 and 5 of the CMMI (Capability Maturity Model Integration) are considered high maturity and are predominantly characterized by quantitative improvement. This blog posting briefly discusses high maturity and highlights several recent works in the area of high maturity measurement and analysis, motivated in part by a recent comment on a Jan. 30 post asking about the latest research in this area. I’ve also included links where the published research can be accessed on the SEI website. At CMMI level 3, work is proactively managed and standard processes are used. Beyond level 3, process performance needs to be understood quantitatively. High maturity means you have the data to understand how the process is performing, how variation in the implementation and execution of the process affect performance, and what the likely costs and benefits of any change will be. A high-level description of the benefits, by process area, is shown below. In past years some CMMI users said they felt high maturity was not well-defined, an issue addressed by its clarification in CMMI v1.3. The CMMI community has also debated the benefits of moving up to high maturity, and asked for more examples of high maturity process implementations. Some challenges organizations face when striving for high maturity include developing an insightful set of measures, creating predictive models for process performance, project management, and product quality, and knowing which tools and methods to use for modeling and analysis. The SEI has worked to address these concerns and provide needed resources through courses, case studies, and other publications about the implementation of high maturity practices. Publications defining and describing high maturity measurement and analysis practices include: CMMI and TSP/PSP: Using TSP Data to Create Process Performance ModelsBy Shurei TamuraThis report describes the fundamental concepts of process performance models (PPMs) and describes how they can be created using data generated by projects following the Team Software Process (TSP). PPMs provide accurate predictions and identify factors that projects and organizations can control to better ensure successful outcomes, helping organizations move from a reactive mode to a proactive, anticipatory mode. PPMs are fundamental to the implementation of the high maturity process areas of CMMI and are specifically required in the Quantitative Project Management and Organizational Process Performance process areas. The three examples in this report demonstrate how data generated from projects using TSP can be combined with data from other sources to produce effective PPMs.www.sei.cmu.edu/library/abstracts/reports/09tn033.cfm Approaches to Process Performance Modeling: A Summary from the SEI Series of Workshops on CMMI High Maturity Measurement and AnalysisRobert W. Stoddard & Dennis R. GoldensonOrganizations are increasingly striving for and achieving high maturity status, yet there is still an insufficient shared understanding of how best to implement measurement and analysis practices appropriate for high maturity organizations. A series of twice-yearly workshops organized by the SEI allows organizations to share lessons learned to accelerate the adoption of best measurement and analysis practices in high maturity organizations. This report summarizes the results from the second and third high maturity measurement and analysis workshops. The participants' presentations described their experiences with process performance models; the goals and outcomes of the modeling; the x factors used; the data collection methods; and the statistical, simulation, or probabilistic modeling techniques used. Overall summaries of the experience and future plans for modeling also were provided by participants.www.sei.cmu.edu/library/abstracts/reports/09tr021.cfm CMMI High Maturity Measurement and Analysis Workshop Report: March 2008Robert W. Stoddard II, Dennis R. Goldenson, Dave Zubrow, & Erin HarperOrganizations are increasingly looking for guidance on how to implement CMMI high maturity practices effectively and how to sustain their momentum for improvement. As high maturity organizations work to improve their use of measurement and analysis, they often look to examples of successful implementations for guidance. In response to the need for clarification and guidance on implementing measurement and analysis in the context of high maturity processes, members of the SEI’s Software Engineering Measurement and Analysis (SEMA) initiative organized a workshop at the 2008 SEPG North America conference to bring leaders in the field together at a forum on the topic. Other workshops will be held as part of an ongoing series to allow high maturity organizations to share best practices and case studies.www.sei.cmu.edu/library/abstracts/reports/08tn027.cfm The following reports describe results from surveys conducted related to the implementation and impacts of high maturity practices. Performance Effects of Measurement and Analysis: Perspectives from CMMI High Maturity Organizations and AppraisersBy James McCurley & Dennis R. GoldensonThis report describes results from two recent surveys conducted by the SEI to collect information about the measurement and analysis activities of software systems development organizations. Representatives of organizations appraised at maturity levels 4 and 5 completed the survey in 2008. Using a variant of the same questionnaire in 2009, certified high maturity lead appraisers described the organizations that they had most recently coached or appraised for the achievement of similar high maturity levels. The replies to both surveys were generally consistent even though the two groups are often thought to be quite different. The results of the surveys suggest that the organizations understood and used CMMI-based process performance modeling and related aspects of measurement and analysis a great deal. Both the organizational respondents in 2008 and the appraisers in 2009 reported that process performance models were useful for the organizations. The respondents in both surveys also judged that process performance modeling is more valuable in organizations that understood and used measurement and analysis activities more frequently and provided organizational resources and management support. In addition, results from the 2009 survey of lead appraisers indicate that organizations that achieved their appraised high maturity level goals also found measurement and analysis activities more useful than those organizations that did not achieve their targets.www.sei.cmu.edu/library/abstracts/reports/10tr022.cfm Use and Organizational Effects of Measurement and Analysis in High Maturity Organizations: Results from the 2008 SEI State of Measurement and Analysis Practice SurveysBy Dennis R. Goldenson, James McCurley, and Robert W. Stoddard IIThere has been a great deal of discussion about what organizations need to attain high maturity status and what they can reasonably expect to gain by doing so. Clarification is needed along with good examples of what has worked well and what has not. This clarification is particularly needed with respect to measurement and analysis. This report contains results from a survey of high maturity organizations conducted by the SEI in 2008. The questions center on the use of process performance modeling in those organizations and the value added by that use. The results show considerable understanding and use of process performance models among the organizations surveyed; however there is also wide variation in the respondents’ answers. The same is true for the survey respondents’ judgments about how useful process performance models have been for their organizations. As is true for less mature organizations, there is room for continuous improvement among high maturity organizations. Nevertheless, the respondents’ judgments about the value added by doing process performance modeling also vary predictably as a function of the understanding and use of the models in their respective organizations. More widespread adoption and improved understanding of what constitutes a suit-able process performance model holds promise to improve CMMI-based performance outcomes considerably.www.sei.cmu.edu/library/abstracts/reports/08tr024.cfm We hope you find this information useful. If there are any other areas of SEI research that you would like us to highlight, please leave a comment below. Additional Resources: An article in the January/February 2012 issue of CrossTalk, The Journal of Defense Software Engineering, titled "High Maturity - The Payoff" discusses the value and benefits realized by organizations that adopt high maturity CMMI Level 4 and 5 software processes. Measuring the Software Process: Statistical Process Control for Software Process ImprovementBy William A. Florac & Anita D. CarletonThis book was one of the first published works that explicitly addressed how to use statistical process control methods to manage and improve software processes within an organization.  It remains a key reference for how to implement high maturity measurement and analysis.  It explains how quality characteristics of software products and processes can be quantified, plotted, and analyzed, so that the performance of software development activities can be predicted, controlled, and guided to achieve both business and technical goals.www.sei.cmu.edu/library/abstracts/books/0201604442.cfm To learn more about the SEI’s work in Measurement and Analysis, please visitwww.sei.cmu.edu/measurement/index.cfm  

By Linda Parker GatesSenior Member of the Technical StaffAcquisition Support Program The appeal of Agile or lightweight development methods has grown steadily in the software development community.  Having spent a number of years investigating strategic planning approaches, I’ve recently been thinking about whether Agile principles can be—and should be—applied to strategic planning.  This blog post examines the applicability of Agile principles to strategic planning. Strategic planning is the process of defining an organization’s plans for achieving its mission. The purpose is to outline a broad approach to achieving mission-aligned goals derived through a process of analyzing the full organizational environment, taking into account the organization’s vision, goals, objectives, enablers, barriers, and values. Although descriptions and analysis of the present situation are included, a strategic plan doesn’t merely endorse the status quo; it is directional in nature and directs change of some kind. As such, strategic planning is a critical foundation for executing work. It sets the stage for division- and unit-level planning, as well as enterprise architecture, process improvement, risk management, portfolio management, and any other enterprise-wide initiatives. Strategic planning typically follows this type of sequence: Scope the strategic planning effort. Build a foundation of organizational information. Define goals and objectives in terms of the organizational need and desired outcome. Identify potential strategies for achieving the objectives. Develop action plans. Identify project measures. Execute the work. Track progress. My February 2011 blog post, Strategic Planning with Critical Success Factors and Future Scenarios, proposes the integration of the Critical Success Factor (CSF) Method and future scenario planning into the strategic planning process. CSFs are a group of factors that determine group or company success, including key jobs that must be done exceedingly well. Future scenarios are a tool for exploring multiple, possible "futures" and developing decisions or strategies that will serve the uncertain future well. Together, these two techniques help foster strategic thinking, which is a strong complement to strategic planning. They also provide some leverage points for making strategic planning more nimble. The values presented in the Agile Manifesto focus on the following four principles: individuals and interactions over processes and tools working software over comprehensive documentation customer collaboration over contract negotiation responding to change over following a plan The manifesto is careful to point out that a balance is required; that is, the counter values cannot be ignored. So a key question becomes: Can organizations benefit from pursuing strategic planning approaches that embody these Agile values?  The rest of this blog posting addresses this question. Individuals and Interactions over Processes and ToolsA strong process is vital to effective strategic planning. Nonetheless, placing value on interactions between leadership and staff and on face-to-face conversations certainly enhances the value of the strategic planning process. Both the CSF method and scenario planning rely heavily on individuals and interactions. These conversations themselves bring value to strategic planning. Working Software/Tangible Results over Comprehensive DocumentationA common criticism of strategic planning is that it over-emphasizes deconstruction of the past and present while creating the illusion that we can anticipate the future. If we replace "working software" with "tangible results" (to accommodate the strategic planning domain), I contend it is productive to focus strategy efforts on short-term accomplishments over thoroughly documented commitments about the future. Customer Collaboration over Contract NegotiationWhile contract negotiation is not particularly pertinent to strategic planning, a strategic plan serves as an informal contract with the organization. The idea of involving customers is also intriguing. In my technical report, Strategic Planning with Critical Success Factors and Future Scenarios, published in November 2010, I recommend involving customers in scenario planning since they bring an external perspective that can be critical to getting quality results. There may even be a greater role for customers in the development of strategy. Responding to Change over Following a PlanThis principle offers great potential for improving the way strategic planning is conducted and the results realized in implementation. A good strategic-planning process has always done more than just produce a plan—it supports ongoing strategic thinking, discussion, and behavior. Strategic thinking focuses on finding and developing organizational opportunities and creating dialogue about the organization’s direction—these are the foundation of an organization’s readiness to respond to change. Strategic planning is enhanced by strategic thinking, which makes planning adaptive and in sync with an evolving environment. So is the planning activity itself still needed? Yes! Effective strategic work cannot be accomplished without it. If nothing else, the divergent results of strategic thinking must be operationalized through a convergent planning activity. So, the next question is, how do we adjust the strategic planning process to make it more agile?  The most effective adjustments to make are to apply the Agile values to steps 3 and 8 of the strategic planning process outlined above. Step 3 (Define goals and objectives) benefits from a focus on the first value (individuals and interactions) and the third value (customer collaboration).  Step 8 (Track progress) is enhanced through a focus on the second value (which I am calling "tangible results") and the fourth value (responding to change).  Interestingly, the CSF and scenario planning methods provide opportunities to integrate all four Agile values into a strategic planning process as follows: In terms of individuals and interactions, CSFs are derived through interviews with managers, a critical aspect of the technique that involves one-on-one conversations with people very closely acquainted with operational issues. Scenario planning is a team-based exercise that relies heavily on interactions among a representative cross-section of organizational staff. The interview-based method for developing scenarios that Kees van der Heijden describes in his 1996 book, Scenarios: The Art of Strategic Conversation, can enhance the role of individuals in setting strategy. As noted above, scenario planning provides a good opportunity for customer collaboration. Scenario planning relies heavily on monitoring for early warning signs, which are indicators that a particular future is unfolding. These indicators help planners make adjustments to strategies or their execution. Combined with shorter cycles, monitoring techniques are critical for delivering short-term, tangible results and responding to change.  It is important to understand the nature of planning.  The fourth agile value emphasizes responding to change over following a plan.  Without a strong plan, response to change is simply reaction, not agility. The Agile principles assert values in terms of preferences over the counter values. In the strategic planning domain, the ability to perform in accordance with Agile values requires significant strength in the counter values. Agility emerges from skill and strength in the counter values. It doesn’t replace them. Agile strategic planning would best serve an organization that is applying agile methods.  If development teams are already using lightweight methods, leadership should consider adopting agile processes to move the organization toward its goals. In general, agile strategic planning can offer value to organizations that are complex or self-organizing and that focus on adaptive, iterative delivery. Additional Resources: To read or download a copy of the SEI technical report, Strategic Planning with Critical Success Factors and Future Scenarios: An Integrated Strategic Planning Framework, please visitwww.sei.cmu.edu/library/abstracts/reports/10tr037.cfm To read the blog post, Strategic Planning with Critical Success Factors, please visit http://blog.sei.cmu.edu/post.cfm/strategic-planning-with-critical-success-factors-and-future-scenarios

By Bjorn Andersson, Senior Member of the Technical StaffResearch, Technology & System Solutions Many DoD computing systems—particularly cyber-physical systems—are subject to stringent size, weight, and power requirements. The quantity of sensor readings and functionalities is also increasing, and their associated processing must fulfill real-time requirements. This situation motivates the need for computers with greater processing capacity. For example, to fulfill the requirements of nano-sized unmanned aerial vehicles (UAVs), developers must choose a computer platform that offers significant processing capacity and use its processing resources to meet its needs for autonomous surveillance missions. This blog post discusses these issues and highlights our research that addresses them. To choose a computer platform that offers greater capacity, it is necessary to observe the major trends among chip makers. Historically, advances in semiconductor miniaturization (a.k.a., Moore's Law) periodically yielded microprocessors with significantly greater clock speeds. Unfortunately, microprocessor serial processing speed is reaching a physical limit due to excessive power consumption. As a result, semiconductor manufacturers are now producing chips without increasing the clock speed, but instead are increasing the number of processor cores on a chip, which results in multicore processors. For nearly a decade, the use of homogeneous multicore processors (which are chips with identical processing cores) gave us some headroom in terms of power consumption and allowed us to enjoy greater computing capacity.  This headroom is diminishing, unfortunately, and is about to vanish, forcing semiconductor manufacturers to seek new solutions. We are currently witnessing a shift among semiconductor manufacturers from homogeneous multicore processors with identical processor cores to heterogeneous multicore processors. The impetus for this shift is that processor cores tailored for a specific class of applications behavior have the potential to offer much better power-efficiency. AMD Fusion and NVIDIA Tegra 3 are examples of this shift. Intel Sandybridge, which has a graphics processor integrated onto the same chip as the normal processor, also reflects this shift, as well. In a heterogeneous multicore environment, the execution time of a software task depends on which processor core it executes on. For example, a software task performing computer graphics rendering, simulating physics, or estimating trajectories of flying objects runs much faster on a graphics processor than on a normal processor. Conversely, some software tasks are inherently sequential and cannot benefit from the graphics processor; they execute much faster on a normal processor. For example, a software task with many branches and no inherent parallelism runs much faster on a normal processor than on a graphics processor. Ideally, each task would be assigned to the processor where it executes with the greatest speed, but unfortunately the workload is often not perfectly balanced to the types of processor cores available. Efficient use of processing capacity in the new generation of microprocessors therefore requires that tasks are assigned to processors intelligently. In this context, "intelligently" means that the resources requested by the program are the ones possessed by the processor. Moreover, the desire for short design cycles, rapid fielding, and upgrades necessitates that task assignment be done automatically—with algorithms and associated tools. THE TASK ASSIGNMENT PROBLEM The problem of assigning tasks to processors can be described as follows: A task (such as computer graphics rendering or a program determining whether the process half-or-triple-plus-one reaches one with a known starting value) is described with its processor utilization, but it has different processor utilizations for different processors. For example, if a given task is assigned to a graphics processor, then the task will have a utilization of 10 percent. If the task is assigned to a normal processor, the task will have a utilization of 70 percent. We are interested in assigning each task to exactly one processor such that for each processor, the sum of utilization of all tasks assigned to this processor will not exceed 100 percent. If we can find such an assignment then it is known that if tasks have deadlines described with the model implicit-deadline sporadic tasks—and if the scheduling algorithm Earliest-Deadline-First (EDF) is used—then all deadlines will be met at run-time (with a minor modification, we can use Rate-Monotonic scheduling as well). PREVIOUS APPROACHES FOR TASK ASSIGNMENT The task assignment problem belongs to a class of problems that are computationally intractable, meaning that it is highly unlikely to be possible to design an algorithm that finds a good assignment and always runs fast. So we should either create an algorithm that always finds a good assignment or one that always runs fast. For the goal of designing an algorithm that always finds a good assignment, task assignment can be modeled as integer-linear programming (ILP) as follows: Minimize zsubject to the constraints that    for each processor p: x1,p*u1,p + x2,p*u2,p + … + xn,p*un,p <= zand    for each task i: xi,1 + xi,2 + … + xi,m = 1and    for each pair (i,p) of task i and processor p: xi,p is either 0 or 1 In the optimization problem above, n is the number of tasks and m is the number of processors and ui,p is the utilization of task i if it would be assigned to processor p. xi,p is a decision variable with the interpretation that it is one if task i is assigned to processor p; zero otherwise. Unfortunately, solving this integer linear program takes a long time. To design an algorithm that always runs reasonably fast, there are several algorithms, as described in a research paper by Sanjoy K. Baruha, that transform the ILP into a linear program (LP) and then perform certain tricks. Although an LP runs faster than the ones based on ILP, they still have to solve an optimization problem, which can be time-consuming. To design algorithms that run faster, we would like to perform task assignment in a way that does not require solving LP. OUR APPROACH FOR TASK ASSIGNMENT Previous work on task assignment for homogeneous multicore processors where all processor cores are identical is based on a framework called bin-packing heuristics. Such algorithms work approximately as follows: 1.Sort tasks according to some criterion. 2.for each task do 3.for each processor do 4.if the task has not yet been assigned and it is possible to assign the task to the processor so that the sum of utilization of tasks on the processor does not exceed 100 percent then 5.Assign the task on the processor 6.end if 7.end for 8.end for Our approach involves adapting bin-packing heuristics to heterogeneous multicore processors. We believe it is possible to modify the algorithm structure outlined above so we can also assign tasks to processors even when the utilization of a task depends on the processor to which it is assigned. One can show that the use of bin-packing can perform poorly if processors and tasks are not considered in any particular order. Specifically, for a set of tasks that could be assigned, such an approach can fail even when given processors that are "infinitely" faster. One of our main research challenges is therefore to determine how to sort tasks (step 1) and in which order we should consider processors (in step 3). We are evaluating our new algorithms in the following ways: We plan to prove (mathematically) the performance of our new algorithms. Specifically, we are interested in proving that if it is possible to assign tasks to processors then our algorithm will succeed in assigning tasks to a processor if a given processor is x times as fast.  Given that x is our performance metric; the lower its value, the better. We also plan to evaluate the performance of our algorithms by applying the algorithms on randomly-generated task sets. This will demonstrate the "typical" behavior of the algorithms. CONCLUSION Most semiconductor manufacturers are shifting towards heterogeneous multicore processors to offer greater computing capacity while keep power consumption sufficiently low. But using a heterogeneous multicore efficiently for cyber-physical systems with stringent size, weight, and power requirements requires that tasks are assigned properly. This blog post has discussed the state-of-art and summarized our ongoing work in this area. ADDITIONAL RESOURCES To read the paper "Partitioning real-time tasks among heterogeneous multiprocessors" by Sanjoy Baruah, please visithttp://ieeexplore.ieee.org/xpls/abs_all.jsp?arnumber=1327956 To read the proceedings "Assigning Real-Time Tasks on Heterogeneous Multiprocessors with Two Unrelated Types of Processors", please visithttp://www.cister.isep.ipp.pt/activities/seminars/%28S%28otidyq454nfyy255alnrdb3m%29%29/GetFile.aspx?File=%2Fspringseminars2011%2Frtss10_het2.pdf

By Mike Phillips Principal ResearcherAcquisition Support Program In my preceding blog post, I promised to provide more examples highlighting the importance of software sustainment in the US Department of Defense (DoD). My focus is on certain configurations of weapons systems that are no longer in production for the United States Air Force, but are expected to remain a key component of our defense capability for decades to come, and thus software upgrade cycles need to refresh capabilities every 18 to 24 months. Throughout this series on efficient and effective software sustainment, I will highlight examples from each branch of the military. This second blog post describes effective sustainment engineering efforts in the Air Force, using examples from across the service’s Air Logistics Centers (ALCs). A Brief History of Software Sustainment From its earliest days, the military has provided facilities to maintain the functionality of its various weapon systems. The descriptive terms for these units have included "arsenals,"  "depots," and "logistics centers," to name a few. In the 1990s the Air Force consolidated its depot maintenance capabilities into three centers: Warner-Robins ALC in Georgia, Oklahoma City ALC in Oklahoma, and Ogden ALC at Hill Air Force Base (AFB) in Utah.  A 2012 initiative within the Air Force Material Command will centralize the leadership of the ALC’s into a single entity, although the three sites will remain as sister units in a single "super ALC" headquartered at the Oklahoma City site. Within this geographical framework, we can overlay the increased importance of software in our weapon systems. Lloyd Mosemann, the deputy assistant secretary of the Air Force for communications, computers and support systems, was a visionary leader for software sustainment in the Air Force logistics arena in the 1980s. He recognized that the various ALCs would "inherit" responsibility for the various weapon systems as production waned and that "software sustainment" should be a well-developed capability within the organic structure of the ALCs. The first demand for an organization to achieve a maturity level was in a memorandum from Mosemann to the centers, directing them to achieve a maturity level 2 against the Capability Maturity Model for Software and then, level 3 a few years later (many think the "Mosemann letter" was aimed at industry, but it was not). This blog posting highlights examples of effective sustainment of software intensive systems across the three sites and recognizes that the successes achieved are the results of improvement efforts across the domain, well beyond the process domain. The workforce has grown in its technical competence, and a modern systems engineering environment has been developed. With that historical perspective, we can explore some of the examples evident today. In the mid 1970s, the Air Force created a "block" strategy as a way to pursue modernization while maintaining a relatively stable production approach. In the past, letter identifiers after the aircraft number showed a change to a new capability, including major hardware changes. With increasing software content, a block upgrade represented major functionality changes with relatively modest hardware configuration changes. The block strategy has become a common practice across the DoD, where software updates are released as a block of changes at regular intervals to avoid too many variations being sent to the field.  By applying a block strategy, the DoD assures that there is a regular, anticipated opportunity to have the freshest capabilities deployed. The US Air Force F-16 program is an excellent example of the block strategy, with more than 4,400 F-16’s being produced and flown by 26 countries. A key program decision made early in the development of the F-16 was the deliberate strategy for evolving the F-16’s capabilities. This strategy was called the F-16 Multinational Staged Improvement Program (MSIP) and involved the continual enhancement of new aircraft and the retrofit of previous aircraft with system hardware and software capabilities. The Ogden ALC’s experience with the F-16 provides a good example of the success of the MSIP being applied in a sustainment environment. From a sustainment perspective, the focus has been on the F-16C/D, Block 30, since the later blocks (40 and 50) have remained in production. Continuous Improvement Initially the Ogden ALC used the CMM and then CMMI to guide its improvement efforts. More recently, they have joined their process improvement partners at the Navy Air Warfare Center at China Lake, Calif. in using the Team Software Process (TSP). The metrics produced by the Ogden team illustrate the following quality performance: Major upgrades continue to enjoy higher and higher productivity due to the quality emphasis contained in the disciplined approach to software production. Rework effort is down below 10 percent. Efficient Upgrade Planning Part of the effective planning highlighted above was an effort to provide reasonably sized updates on a regular schedule. For the F-16, the size has been about 500 thousand of source/software lines of code (KSLOC) using an upgrade cycle aimed at 18 to 24 months. The upgrade team has been able to accommodate high priority upgrades, such as a rapid introduction of the AIM-9X, by balancing sustainment and modernization loads effectively. The F-16 team has just begun to assimilate the transition of the later F-16 blocks 40 and 50. The foundation established with the Block 30 experience should enable the ALC sustainment team to master the learning curve for the more complex and diverse fleet being transitioned. The Airborne Early Warning & Control Systems (AWACS), which are supported by the Oklahoma City ALC, provide another example. These systems fit in a Boeing 707-based aircraft with a large saucer shaped antenna housing radar systems that can be upgraded to improved hardware and software capabilities. Reflective of many DoD platforms, the AWACS system entered operations in 1976 and its critical mission systems are continually upgraded to extend its capabilities. The multi-source integration function that is part of the most recent AWACS upgrade used open systems and lean architecture approaches, which are advocated by the SEI and allow more rapid software updates to the fleet without requiring extensive hardware changes. Another weapon system sustained and continuously improved upon at Oklahoma City is the venerable B-52 strategic bomber. If planned modernization efforts of this weapon system extend the life as currently proposed, the aircraft will have been in the active Air Force inventory for almost 90 years upon its retirement in 2040. The improvement to the software architecture expands its versatility with "smart weapons" and its network communications capability. In an era that demands a focus on affordability, extending the lifetime of the B-52—while enhancing its capability—demonstrates the power of software-focused modernization. The Warner-Robins ALC has responsibility for electronic warfare and radar systems, as well as weapon systems, including the F-15, C-130 (almost as old as the B-52) and the C-5. Across this collection of software-reliant systems, the center continues to improve the quality and timeliness of its upgrades. Of 246 software releases in fiscal year 2010, the ALC delivered 99 percent at or below expected cost, and 98 percent on or ahead of schedule. Only three systems contained defects that were discovered in the field after release and required rework. As with its two sister ALCs, Warner-Robins has committed to continuous improvement of its software capability. While we at the SEI are pleased with the unit’s involvement with our CMMI models, the organization has complemented that effort with attention to Lean Six Sigma and the Air Force Smart Operations for the 21st Century (AFSO21). The results are notable. The leadership of the software organization at Warner Robins has noted that the "old image" of software sustainment as small "bug fixes" has been replaced by recognition that with software-reliant systems, the opportunity arises to quickly develop innovative capabilities to solve the challenges facing their DoD—and allied nation—customers today. Each of the services has a wide variety of software-reliant systems, and organic capabilities to complement the major contractors who create sturdy platforms like the B-52 that can last nearly a century. The next installment in this series on efficient and effective software sustainment will examine the Naval Air Weapons Station at China Lake. Additional Resources To read the SEI technical report, Sustaining Software Intensive Systems, please visit www.sei.cmu.edu/library/abstracts/reports/06tn007.cfm?DCSext.abstractsource=SearchResults

Experience from Financial SystemsBy Bill Nichols, Senior Member of the Technical Staff Software Engineering Process Management In his book Drive, Daniel Pink writes that knowledge workers want autonomy, purpose, and mastery in their work. A big problem with any change in processes is getting the people who do the work to change how they work. Too often, people are told what to do instead of being given the information, autonomy, and authority to analyze and adopt the new methods for themselves.  This posting—the first in a two-part series—describes a case study that shows how Team Software Process (TSP) principles allowed developers at a large bank to address challenges, improve their productivity, and thrive in an agile environment. In 2009, Nedbank, one of the four largest banks in South Africa, launched a TSP pilot program in collaboration with the Johannesburg Centre for Software Engineering (JCSE) at the University of Witwatersrand and the SEI. After TSP pilot teams at Nedbank were given a toolkit—including training on how to measure, analyze, and communicate—their behaviors changed.  Not only were the developers (a mixed team of elite and average programmers) able to work more autonomously, they implemented methods that resulted in higher quality work in less time. The Nedbank TSP pilot involved two software-intensive projects: one maintenance project and one new development project. Nedbank had faced software process improvement challenges in the past. They therefore sought to address the issues of quality and productivity among their software engineering teams by implementing a methodology that would improve the teams' performance through planning, tracking their work, establishing goals, and providing teams with the tools to take responsibility for their processes and plans. Engage the Team The Nedbank TSP pilots began with separate training for everyone involved: senior managers, team leads, software developers, and non-developer team members.  The groups learned how the process change would affect them and their relationship with others in the organization. They learned that working on a TSP project required them to behave differently in terms of reporting, data collection, expectations regarding other team members, and interactions with other team members. Developers learned the Personal Software Process with a focus on software engineering discipline, measurement, and planning. Non-developers, such as testers, business analysts, and documenters, learned to work within the new team and project structure. Throughout the courses, Nedbank team members learned what a measured software process looks like and how to measure the software process. They also learned to communicate with a focus on the data, project, and work, not on personal issues. Launch the Project TSP projects begin with a structured launch to plan the work, including nine meetings plus a post-mortem meeting to satisfy specific project planning needs. The Nedbank teams began by understanding project goals from a management standpoint and then identified supporting team goals and personal objectives for each participant. These goals helped frame their subsequent decisions on strategy, process, support, and work breakdown. The project was then broken down into small pieces, and work was assigned among team members. Each member was then able to commit to the plan and his or her specific interim goals, and understand how those goals support the organization's larger goals for the project. The key to success was that the team had the autonomy to plan and manage the work and determine how it would be done. For example, early in the planning phase, the team working on the maintenance project realized that the schedules of the designers and the developers were not aligned. The designs wouldn't be ready when the developers needed them. Working together, the team approached the schedule problems in the following two ways: The team lead went to management and negotiated priorities, using his team's progress data to support the schedule needs. The team lead was then able to convince the designers to prioritize their work so that they could supply the developers with the designs needed to proceed. The senior developers next took the load off the designers by taking on some of the design work. In addition, the team identified modules that would not have to be changed as well as sets of programs that would be easy to update. This change in approach not only helped meet project needs, but also helped satisfy the management goal that teams gain a big picture view of the project. Deal with Mid-Project Issues Early in the development process, both projects fell behind schedule. Discipline and measurement became critical. By tracking their time and following their defined steps, team members were able to identify their status precisely and understand how they got there. The team lead made sure the measurements and data were discussed in the status meeting. The coach helped the team understand what the measurements and data meant and how these facts affected the work. With this data-driven feedback, the teams saw an increase in the number of task hours (direct time on project tasks) per week, as well as a sharp reduction in defect rates. With the coach, the teams periodically analyzed their data and improved their processes. By the end of each pilot project, the quality of each team's work had improved significantly. They were able to find and remove defects earlier in the process. After the initial delivery, no further defects were found in system tests or production in the remaining three cycles. There were zero defects in deliveries two through four. This improvement required months of effort, training, management support, coaching, worker motivation and engagement, and meaningful data-based feedback. See the Results In the end, the TSP pilot teams at Nedbank made significant behavioral changes that not only improved the quality of the software but also improved team members' work lives by decreasing the need for evening and weekend overtime. The teams were able to make these improvements because they had project-specific measurements to guide their decisions, and they had the authority to implement those decisions. Based on the results of the pilots Nedbank decided to implement TSP throughout the organization. To learn more about Nedbank's views on TSP, see their rollout video below: This video requires the Adobe Flash Player Plug-in please go to the following URL to download: http://get.adobe.com/flashplayer/ Expanding and scaling any process comes with challenges. These topics will be discussed in the second post of this series: Achieving Quality and Speed with TSP Organization-Wide.   Additional Resources: To read the SEI technical report Deploying TSP on a National Scale: An Experience Report from Pilot Projects in Mexico, please visitwww.sei.cmu.edu/library/abstracts/reports/09tr011.cfm To read the Crosstalk article A Distributed Multi-Company Software Project by Bill Nichols, Anita Carleton, & Watts Humphrey, please visitwww.crosstalkonline.org/storage/issue-archives/2009/200905/200905-Nichols.pdf To read the SEI book Leadership, Teamwork, and Trust: Building a Competitive Software Capability by James Over and Watts Humphrey, please visit www.sei.cmu.edu/library/abstracts/books/0321624505.cfm To read the SEI book Coaching Development Teams by Watts Humphrey, please visitwww.sei.cmu.edu/library/abstracts/books/201731134.cfm To read the SEI book PSP: A Self-Improvement Process for Engineers by Watts Humphrey please visitwww.sei.cmu.edu/library/abstracts/books/0321305493.cfm

By Robert Nord, Senior Member of the Technical StaffResearch, Technology, & System Solutions New acquisition guidelines from the Department of Defense (DoD) aimed at reducing system lifecycle time and effort are encouraging the adoption of Agile methods. There is a general lack, however, of practical guidance on how to employ Agile methods effectively for DoD acquisition programs. This blog posting describes our research on providing software and systems architects with a decision making framework for reducing integration risk with Agile methods, thereby reducing the time and resources needed for related work. The DoD chief information officer (CIO)’s office has recently released a 10 Point Plan to reform DoD information technology (IT).  Point number 4 is "Enable Agile IT." The key tenets of Agile IT include Emphasize incremental introduction of mature technology by delivering useful capability every 6 to 12 months to reduce risk through early validation to users. Require tight integration of users, developers, testers, and certifiers throughout the project life cycle to meet Agile IT’s promise of rapid delivery in lieu of extensive up front planning. Leverage common development, test and production platforms and enterprise products to deliver IT capabilities faster, cheaper, and more interoperable, without redundant infrastructure and documentation.  Establish a change-tolerant design environment enabled by discovery learning that promotes decisions based on facts rather than forecasts. Program managers and acquisition executives are responding to this plan by applying industry practices, such as Agile methods. At the same time, related DoD guidelines encourage system development via a more open, modular software architectural style, such as loosely-coupled service-oriented architectures. The impact of these architectural decisions becomes more critical in an agile context because they promote or inhibit the ability to achieve agile goals, such as rapid feedback, responding to change, and team communication. Architectural decisions influence agile development properties, such as the length of an iteration, the allocation of stories to iterations, and the structure of the teams and their work assignments with respect to the architecture. What is needed, therefore, is research on eliciting and employing these properties and determining their relative importance in promoting rapid lifecycle development. To address this need, I and other SEI technologists, Ipek Ozkaya, Stephany Bellomo, and Mary Ann Lapham, are conducting research that focuses on the implications of decisions made over the course of the software and systems lifecycle. We are examining when these decisions are made and the time when the implications surface to validate the following hypotheses: The fundamental early decisions made during the pre-engineering and manufacturing development (pre-EMD) phase have an impact throughout the lifecycle. Acquisition programs lack the techniques needed to elicit and preanalyze the implications of which fundamental architectural decisions will enable or inhibit their ability to adopt an agile lifecycle. For example, architectural decisions that relate to decomposition and dependencies influence teaming structure, the capability to rapidly and confidently deliver features, and the ability to rapidly integrate new components among other factors. The implications of the early decisions often surface in the final stages of the lifecycle, downstream from development.  For example, unexpected rework related to correcting integration defects. When these problems are discovered further downstream from where they were injected in the lifecycle, they are more expensive to correct and this often causes cost overruns and delays project completion.   Identifying Critical Properties A primary objective of our research is identifying critical properties of Agile software development influenced by architectural decisions that reduce lifecycle time.  Identifying these properties is an important first step to give stakeholders the tools they need to make informed decisions that manipulate the settings of the properties to control for better outcomes. Our approach involves examining the implications of possible change scenarios. One such scenario examines the impact of introducing an emerging multi-core technology to the mission processor software of the Apache helicopter, which is a US Army/Boeing program. Applications are increasingly favoring multi-core processors, a single integrated computing element with two or more independent processors, called "cores," allowing for greater processing capacity. The scenario explores questions that include Will the multi-core technology change the architectural design (e.g., patterns, tactics, and architectural approaches)? If so, which architectural decisions will change? Are there engineering practices (such as Agile) that must be customized as a result of architectural decisions in support of a rapid lifecycle? How will continuous integration be ensured? How will team communication be impacted? A key component of our work involves determining the critical decisions that will influence Agile practices. These decisions provide the rationale for how software and system architects design an architecture.  To expand our view, we will again collaborate with Philippe Kruchten, a professor of software engineering in the Department of Electrical and Computer Engineering of the University of British Columbia, who is active within Agile development and architecture decision making research communities. Another facet of our approach involves interviewing DoD programs and gathering data from members of the SEI Agile Collaboration Group.  Creating a Model Rework must be considered early when reasoning about how to enable rapid lifecycle development. After we review the findings identified in the first phase of our work with collaborators, we will create an architectural decision model that allows software or systems architects to analyze the ramifications of their decisions. Based on our prior work, we anticipate that highly impactful architectural decisions will include Decisions about interfaces and how the parts of the system are connected. Decisions about structuring the systems to achieve quality attribute requirements. The impact of these decisions typically spans multiple areas within the system and is not localized within a single module. We plan to use the Multiple Domain Matrix (MDM) to represent the decision model and to analyze the impact of architectural decisions on rapid lifecycle development. The MDM approach considers decision dependencies that provide visibility into how the ordering of decisions influences when development can be started and how changes propagate and may require rework of software elements. This approach will allow us to test our hypothesis that modeling architectural decisions during early stages of development (similar to pre-EMD) will reduce cycle time.  Cycle time could be reduced upstream by enabling an earlier start of development, thus minimizing the time spent at the pre-EMD phase or reduced downstream by decreasing rework costs attributed to architectural decisions that affect integration. Another component of our research will explore strategies for improving the relationship between architecture decision making and complex collaborative team interaction. A barrier that often arises is that decisions made by architects about partitioning the architecture are not aligned with the networks of agile teams at scale and for the kinds of systems relevant to the DoD. Another barrier is alignment with the teams that span the lifecycle beyond the development teams traditionally associated with agile and include system engineering, testing, validation and verification, certification and accreditation. We have observed the ramifications of this misalignment during the integration of components built by different teams, where incompatibilities lead to significant rework. To map, analyze, and support the architectural decisions of industry collaborators, we plan to map our MDM approach into a conceptual model developed by Kevin Sullivan of the University of Virginia, who is working on creating a cyber-social conceptual model. Sullivan’s work focuses on the social networks and the value of relationships between decision makers in a system. Challenges A primary technical challenge that we face in this approach involves scaling. As a practice, Agile has been successful in helping to solidify the efficiency of development teams when projects involve small teams. Applying these same concepts to large-scale distributed systems— including the rest of the organization that has priorities larger than the development team—will be critical for success, but it will also present some of the greatest challenges. To address the challenge of scaling, we are looking at the influence architecture exerts on managing teams and how to provide practical guidance on what amount of architecture is needed and when. On the one hand, early or overproduction of architecture can create delay. On the other hand, not enough production of architecture can result in integration defects leading to rework. The focus of lean thinking on improving cycle time by eliminating waste, in the form of delay or unnecessary rework, in conjunction with architecture, has shown great potential for improving management of software development projects and increasing flow of value delivered to the customer. Next Steps We are in the process of conducting a survey of critical agile development properties and will write about the results in a future blog posting. Additional Resources:    To read the SEI technical report, Agile Methods: Selected DoD Management and Acquisition Concerns, please visit www.sei.cmu.edu/library/abstracts/reports/11tn002.cfm  

By Dean Sutherland Senior Member of the Technical StaffThe CERT Program Many modern software systems employ shared-memory multi- threading and are built using software components, such as libraries and frameworks. Software developers must carefully control the interactions between multiple threads as they execute within those components. To manage this complexity, developers use information hiding to treat components as "black boxes" with known interfaces that explicitly specify all necessary preconditions and postconditions of the design contract, while using an appropriate level of abstraction to hide unnecessary detail. Many software component interfaces, however, lack explicit specification of thread-related preconditions. Without these specifications, developers must assume what the missing preconditions might be, but such assumptions are often incorrect. Failure to comply with the actual thread-related preconditions can yield subtle and pernicious errors (such as state corruption, deadlock, and security vulnerabilities) that are intermittent and hard to diagnose. This blog post, the first in a series, describes our ongoing research towards solving this problem for a variety of languages, including Java and C11. Previous Work In previous work, we introduced the concept of "thread usage policy," which is a group of often unspecified preconditions used to manage access to shared state by regulating which specific threads are permitted to execute particular code segments or to access particular data fields. The concept of thread usage policy is not language specific; similar issues arise in many programming languages, including Java, C11, C#, C++, Objective-C, and Ada. The preconditions contained in thread usage policies can be hard to identify, poorly thought out, unstated, poorly documented, incorrectly expressed, out of date, or simply hard to find. These problems inspired us to devise a means of specifying these preconditions in a form that developers would find both useful and acceptable. We developed a simple, formal specification language for modeling thread usage policies, which we call the language of thread role analysis. We devised appropriate abstractions of the key semantic building blocks of thread usage policies (including thread identity, concrete code segments, and data fields) so that developers can build a model of the thread usage policy (hereafter referred to as simply "policy") by expressing preconditions as simple precise annotations in program code. Current Work My focus is on bringing thread role analysis to bear on programs written in C11, which is the current standard of the C programming language (the core analysis is also similar to the analysis for Java). To ensure relevance to practicing programmers, we integrate our analysis with widely used programming tools—in this case, the CLANG/LLVM open-source compiler suite. The C11 work is still in its early stages, so I’ll present an example from our previous work in Java. Electric Had a Problem The developers of the Electric VLSI design tool had re-implemented it in Java for their V8.0 release, in part to support a change to a multi-threaded architecture. Users of the new version of the tool encountered seemingly random internal errors and crashes. These crashes were rarely repeatable, however, and usually disappeared when developers tried to debug the program. Since they were experienced developers, they quickly realized that these symptoms were probably caused by concurrency errors. This diagnosis seemed especially likely, since the major new feature in the most recent release was the implementation of a multi-threaded architecture for performance. The developers needed to quickly identify the problem in their 140 KLOC program, and we were able to do that in less than 8 hours using thread role analysis. We analyzed version 8.0.1, which contained roughly 140 KLOC in 44 Java packages. We describe the process in the form of a reverse-engineering exercise, partly because that’s how we addressed the problem and partly because it helps explain thread role analysis. Note, however, that this reverse-engineering-focused description makes thread role analysis sound relatively time-consuming and hard. In case studies, we found that developers working on their own code could easily identify policies, answer questions about intended thread usage, and locate key points for annotation. Identifying Thread Usage Policies The first step in thread role analysis is to identify relevant pre-existing policies. We’ll use the Electric tool from our example because the developers had a pre-existing written thread usage policy, which appears here in a slightly edited form: There is one persistent user thread called the DatabaseThread, created in com.sun.electric.tool.Job. This thread acts as an in-order queue for Jobs, which are spawned in new threads. Jobs are mainly of two types: Examine and Change. These jobs interact with the Database, which are objects in the package hierarchy com.sun.electric.database The Rules are Jobs are spawned into new Threads in order of receipt. Only one Change job can run at a time. Many Examine jobs can run simultaneously. Change and Examine jobs cannot run at the same time. Examine jobs can only look at the Database. Change jobs can look at and modify the Database. Because only one Change job can run at a time, the Change job is run directly in the DatabaseThread. Examine jobs are spawned into their own threads, which terminate when the Examine job is done. We thus identified one of the two key policies that needed to be expressed; the other is the policy for Java’s AWT/Swing GUI framework, used by Electric’s graphical user interface (GUI). In less than one day of effort, we expressed models of these two policies and used the findings produced by our analysis tool to diagnose a set of "seemingly random intermittent failures" experienced by the development team and their users. Policy for the GUI The Electric GUI uses the AWT and Swing frameworks, which share a single thread usage policy. The GUI framework implementation is not multi-threaded; rather, it executes in its own "event thread" and prescribes rules for how non-GUI threads may interact with it through the framework APIs. The salient points of the policy (according to Bowbeer) are as follows: There is at most one "AWT thread" at a time per application. There may be any number of separate "Compute threads." A Compute thread is forbidden to paint or to handle AWT data structures or events. Failure to comply can lead to exceptions from within the AWT, because the AWT avoids both potential deadlock and data races by accessing its internal data structures from within a single thread, without the use of locks. Extended computation on the AWT thread is forbidden; "brief" computation is acceptable. While the thread is computing, it cannot respond to events or repaint the display; this "freezes" the GUI until the computation finishes. It is important to note that the AWT data structures mentioned above explicitly include any fields of user-defined classes that extend the library-defined AWT and Swing framework classes. Thread Role Versus Thread Identity The policies expressed in prose above allow us to highlight an important feature of thread role analysis—focusing on thread roles rather than thread identities. The GUI policy above speaks of the "AWT thread"; although it fails to mention that, in some implementations, its identity changes from time to time. We don’t care which specific thread is the AWT thread right now; we care only that it performs the role of "AWT thread." Similarly, Electric’s thread usage policy permits multiple Examine jobs, each with its own thread. We care about the "Examine" role but not about the identity of the various threads that perform it. Expressing Thread Usage Policy The first step is to declare any needed thread roles. The @ThreadRole annotation declares names for thread roles. Roles are opaque identifiers that have their own namespace. The Electric policy mentions two thread roles: one for examining the Database and one for changing it (n.b.: Electric’s "Database" is a large data structure that represents the circuit the user is designing; it is not a database in the sense of MySQL or other relational databases). Here are the declarations of the thread roles for Electric, which are described as Java comments: /** **@ThreadRole DBChanger , DBExaminer **@MaxRoleCount DBChanger 1 **@IncompatibleRoles DBChanger, DBExaminer, AWT **/ Likewise, here are the declarations of the thread roles for the GUI: /** * @ThreadRole AWT, Compute * @MaxRoleCount AWT 1 * @IncompatibleRoles AWT, Compute **/ We will use the DBChanger role for all change Jobs and the DBExaminer role for all Examine jobs. Similarly, in the GUI policy, we see two thread roles: the "AWT thread" and "Compute threads." We declare these roles using the annotation @ThreadRole AWT, Compute, thus capturing the roles described in rules 1 and 2 of the GUI policy. We use the AWT  role for the AWT thread and the Compute  role for all other threads. Next, we declare global constraints on thread roles. For example, an important aspect of the GUI policy indicates that the "AWT thread" and the "Compute threads" are distinct; it is forbidden for any thread to perform both roles at once. Similarly, the Electric policy implies that the Change and Examine roles are distinct. In each case, this incompatibility property allows us to conclude that a thread performing one of these roles necessarily excludes all of the others. Incompatibility is one of the few postconditions in our language for thread role analysis. We state this incompatibility for the GUI framework by writing @IncompatibleRoles AWT, Compute for its API. The @IncompatibleRoles annotation in the snippet above specifies the incompatibility for Electric’s thread roles, as well as for the GUI’s AWT thread. These annotations capture the third rule of the GUI policy and the analogous—but implicit—rule for Examine and Change jobs from the Electric policy. Note that if we had followed the Electric policy literally, our annotation might have incorrectly omitted the AWT thread. Finally, both the GUI policy and the Electric policy identify thread roles that may be performed by, at most, one thread at a time. We document this with @MaxRoleCount AWT 1 for the GUI and the similar annotation in the snippet above. Because "any number" is the default, we omit @MaxRoleCount annotations for the other thread roles. Wrap Up and Look Ahead We’ve now reached the halfway point in the process of expressing the thread usage policies. So far, we’ve identified the relevant thread usage policies along with the thread roles needed to express those policies.  We’ve written annotations to declare the thread roles and to express the few global constraints on those roles—most notably incompatibility.  In the second installment in this series, we’ll finish the thread usage policy by associating thread roles and thread role constraints with code segments, and show how thread role analysis enabled us to diagnose in 8 hours a violation that took "multiple" Electric developers "weeks of time" to fix independently. In the third and final post, we’ll discuss bringing thread role analysis to C11, along with techniques to improve adoptability by reducing required programmer effort and supporting analysis of partially annotated code. Additional Resources To read the paper, Composable Thread Coloring (which was an earlier name for the technique we now call thread role analysis) by Dean Sutherland and Bill Scherlis, please go towww.fluid.cs.cmu.edu:8080/Fluid/fluid-publications/p233-sutherland.pdf.