By Douglas C. SchmidtPrincipal Researcher To deliver enhanced integrated warfighting capability at lower cost across the enterprise and over the lifecycle, the Department of Defense (DoD) must move away from stove-piped solutions and towards a limited number of technical reference frameworks based on reusable hardware and software components and services. There have been previous efforts in this direction, but in an era of sequestration and austerity, the DoD has reinvigorated its efforts to identify effective methods of creating more affordable acquisition choices and reducing the cycle time for initial acquisition and new technology insertion.  This blog posting is part of an ongoing series on how acquisition professionals and system integrators can apply Open Systems Architecture (OSA) practices to decompose large monolithic business and technical designs into manageable, capability-oriented frameworks that can integrate innovation more rapidly and lower total ownership costs. The focus of this posting is on the evolution of DoD combat systems from ad hoc stovepipes to more modular and layered architectures. Motivating the Need for Technical Reference Frameworks DoD programs face a number of challenges in this era of increasing threats and constrained budgets. As nation state actors become more sophisticated, the nature of threats becomes asymmetric. It is therefore critically important that the DoD be able to respond quickly to risk with new technologies, while delivering enhanced integrated warfighting capability at lower cost. The DoD faces several challenges in achieving these goals. Chief among them is addressing the decades-long, stove-piped, ad hoc approach to developing software that results in vendor-locked legacy systems, each of which maintains its own proprietary software, computers, networks, and operating systems. A promising solution is OSA, which combines technical practices designed to reduce the cycle time needed to acquire new systems and insert new technology into legacy systems and business models for creating a more competitive marketplace and a more effective strategy for managing intellectual property rights in DoD acquisition programs. The SEI is helping the DoD craft its OSA strategy and an implementation plan to deliver better capabilities to warfighters withinthe fiscal constraints of sequestration. A working group has been established to help the DoD move away from stove-piped software development models to Common Operating Platform Environments (COPEs) that embody OSA practices. As part of this effort, I am involved with a task area on "published open interfaces and standards" that aims to help program managers and other acquisition professionals avoid vendor lock-in, encourage competition, and spur innovation by defining a limited number of technical reference frameworks that breakdown traditional stove-piped solutions. These frameworks are integrated sets of competition-driven, modular components that provide reusable combat system architectures for families of related warfighting systems. Despite substantial advances in technical reference frameworks during the past decade, widespread adoption of affordable and dependable OSA-based solutions has remained elusive. It is therefore important to look at past open-systems efforts across the DoD  to understand what worked, what hasn’t, and what can be done to make it more successful this time. To achieve this historical perspective, I—along with fellow SEI researcher Don Firesmith and Adam Porter from the University of Maryland Department of Computer Science—have been documenting the evolution of DoD combat systems with respect to their adoption of systematic reuse and the OSA paradigm described above, as shown in the following diagram. To view a larger image of the diagram, please click on the image above The ad hoc architectures in the columns on the left are highly stove-piped, course-grained, and exhibit little or no shared capabilities that are critical to warfighter, including communications, radars, launchers, etc. The increasingly advanced architectures from left to right are intentionally designed to share more capabilities at finer levels of granularity in DoD systems, including Infrastructure capabilities, such as internetworking protocols, operating systems, and various layers of middleware services, such as identity managers, event disseminators, resource allocators, and deployment planners, and Common data and domain capabilities, such as trackers, interoperable data models, and mission planners involving battle management (BM), control, and interaction with sensors and weapons in C4ISR systems, and External interfaces, such as across the global information grid (GIG) to external weapon systems, as well as information sources and users. In practice, of course, production combat systems vary in terms of their progression along the continuum shown in the figure and descriptions above. This discussion is intended to provide a birds-eye view of the design space of DoD combat systems with respect to architectural evolution. The remainder of this posting describes the first four epochs in the diagram shown above. The remaining four epochs will be described in the next blog post in this series. Ad hoc Architectures involve the separate development of each warfighter’s capability (such as BM/C4I, sensors, weapons etc. ) in a vertically stove-piped manner that lacks crisply defined module boundaries. This approach is characterized by vertical integration and tight coupling from higher-level domain-specific capabilities down to hardware and system-level infrastructure capabilities.  Ad hoc architectures are widely used in DoD legacy combat systems for various reasons. For example, the tight coupling between system components has historically been deemed essential for mission- and safety-critical DoD programs that need to extract maximum performance to meet stringent end-to-end quality attributes. The stove-piped nature of these ad hoc architectures has also often been perceived as risk prudent, since these architectures enable a single program office and system integrator to maintain tight control over every facet in the solution. Despite their pervasive use historically, however, ad hoc architectures have become prohibitively expensive to develop and sustain over the software and system lifecycle. A key problem is the tight coupling common in ad hoc architectures, which typically locks the DoD into sole-source contracts that limit the benefits of open competition and impede innovations. These innovations include the ability to leverage commodity hardware and/or software platform advances, such as multi-core and distributed-core cloud computing environments, that would otherwise occur during periodic technical refresh insertion points. Modular Architectures define some crisp boundaries within their stove pipes and began to transition away from top-down algorithmic decomposition to a more object-oriented and component-based decomposition. This approach is characterized by designs whose components are less externally coupled and more internally cohesive than the earlier generation of ad hoc architectures.Although ad hoc architectures have not been economically viable for many years, sequestration has renewed the interest of government and defense industry leadership in modular architectures. Ironically, interest in modular architectures for DoD combat systems began several decades ago, as acquisition programs began to define module boundaries more crisply within their stove-pipes to move away from top-down, algorithmic decomposition (which yields tightly coupled point solutions) to a more object-oriented decomposition (which emphasizes modular, loosely coupled components that can be understood and tested more readily in isolation and thus reused more effectively).In the early phases of DoD software development, developers tended to write software using function-oriented programming languages (such as FORTRAN, JOVIAL, and C) and algorithmic decomposition and structured design methods, which focus on optimizing computing performance. In the 1990s, developers began to adopt object-oriented programming languages (such as C++, Java, and Ada95) and object-oriented design methods, which focus more on optimizing developer productivity. This shift occurred, in part, due to advances in hardware and software technologies (such as faster processors and networks, larger storage capabilities, and better compilers).  It also coincided with the defense cutbacks stemming from the breakup of the U.S.S.R. (the so-called "Peace Dividend") and the end of the first Gulf War, which motivated the defense industry to rethink the economics of their development models.For example, as component providers and system integrators recognized they couldn’t pass along the costs of these complex systems to their government customers, they begin modularizing their stove-pipes. They deemed the myriad dependencies and accidental complexity of their traditional ad hoc architectures too costly in terms of development and sustainment effort. A drawback of the first generation of modular architectures, however, was that they were still largely stove-piped and lacked the ability to share components across different warfighting capabilities. For instance, a software module could not be initially targeted for a radar system and then subsequently reused in a launcher system. Modular Open Systems Architecture with Standard Key Interfaces (MOSA) stemmed from a well-defined public standard approach that was both a business and technical strategy for developing new systems or modernizing existing ones.  This approach was characterized by designing systems with modular interfaces, designated key interfaces, and select open standards with the goal of providing acquisition programs a choice of vendors when a system needs to be updated.With the advent of the modular architecture approach described above, the DoD began to reap some benefits of module reuse, including easier testing and porting to new environments. The drawback of this approach, however, was that each module was still largely proprietary. While the end result was a more efficient architecture, it modules were too tightly coupled, which increased sustainment costs and encouraged vendor lock-in.To help overcome these limitations with earlier modularity approaches, MOSA was devised to make it easier for DoD acquisition programs to replace modules from one architecture with modules from another. The resulting architecture provided acquisition programs with a wider choice of vendors when a system underwent upgrades since developers could create a new module with the same interface as the one being replaced. The key difference between MOSA and earlier modularity approaches was that module were connected via standardized and openly published interfaces and integration models. Layered Architectures emerged as commercial off-the-shelf (COTS) software began to mature and DoD acquisition programs began to purchase them directly from vendors and use them to layer systems so that they were no longer entirely built by a single integrator, even in a modular way. This approach was characterized by a horizontal partitioning of a system’s functionality according to a (sub) system-wide property, such that each group of functionality is clearly encapsulated and can evolve independently. The specific partitioning criteria can be defined along various dimensions, such as abstraction, granularity, competitive market size, hardware encapsulation, and rate of change.During the mid-1990s, as the MOSA approach was growing in popularity, the DoD also began to reconsider its stance on COTS hardware and software technologies, such as CPUs, storage devices, networking elements, programming languages, and operating systems. Prior to this point, the DoD had considered COTS to be incompatible in terms of safety, maturity, and dependability for mission-critical combat systems. The constraints and demands of the DoD environment had instead fostered a system in which contractors were building both vertically integrated systems and the underlying system infrastructure, such as programming languages, compilers, operating systems, networking protocols, and networking standards.As COTS technologies began to mature, however, DoD programs began to purchase them directly from vendors and use them to layer certain portions of their systems, particularly domain-independent infrastructure capabilities layer(s).  Examples include COTS products based on open standards such as TCP/IP, POSIX, CORBA, DDS, and Web Services. Consequently, these infrastructure capabilities were no longer built by integrators, even in a modular way. One benefit of layered architectures is that, because of industry competition, DoD programs were using technologies that were much more current than those they were able to obtain through traditional, stove-piped systems. Likewise, commercial industry tends compete and innovate more rapidly than traditional defense contractors due to leveraged funding from a range of customers, including DoD, government, and enterprise/consumer users. Wrapping Up and Looking Ahead Over the past several decades the advances in DoD combat system architectures presented above have had several beneficial effects.  For example, modularity has helped integrators increase the flexibility of their proprietary solutions. Likewise, layering has increased the adoption of domain-independent COTS and open-standards infrastructure as the basis for many DoD combat systems. While these advances are a step in the right direction, they have not yet significantly reduced the development and sustainment costs of DoD combat systems. One reason for this limited impact on lifecycle costs is that these earlier architecture advances did not address key business model drivers, but instead focused on standardized infrastructures and codified architectures, which account for a relatively small portion of the total ownership costs of combat systems. The next post in this series will describe the other four epochs of the architectural evolution of the DoD combat system shown in the diagram above.  These epochs focus more on domain-specific architectural layers that address business and economic issues, as well as technical concerns. Subsequent posts in this series will explore a research effort to help one Navy program obtain accurate estimates of the cost savings and return on investment for both the development and lifecycle of several product lines built using a common technical reference framework. Additional Resources To read the SEI technical report, A Framework for Evaluating Common Operating Environments: Piloting, Lessons Learned, and Opportunities, by Cecilia Albert and Steve Rosemergy, please visit http://www.sei.cmu.edu/library/abstracts/reports/10sr025.cfm To read the SEI technical note, Isolating Patterns of Failure in Department of Defense Acquisition, by Lisa Brownsword, Cecilia Albert, David Carney, Patrick Place, Charles (Bud) Hammons, and John Hudak, please visithttp://www.sei.cmu.edu/library/abstracts/reports/13tn014.cfm

By Julien Delange Member of the Technical Staff    Software Solutions Division Safety-critical avionics, aerospace, medical, and automotive systems are becoming increasingly reliant on software. Malfunctions in these systems can have significant consequences including mission failure and loss of life. So, they must be designed, verified, and validated carefully to ensure that they comply with system specifications and requirements and are error free. In the automotive domain, for example, cars contain many electronic control units (ECU)—today’s standard vehicle can contain up to 30 ECUs—that communicate to control systems such as airbag deployment, anti-lock brakes, and power steering. The design of tightly-coupled software components distributed across so many nodes may introduce problems, such as early or late data delivery, loss of operation, or concurrent control of the same resource. In addition, errors introduced during the software design phase, such as mismatched timing requirements and values beyond boundaries, are propagated in the implementation and may not be caught by testing efforts. If these problems escape detection during testing, they can lead to serious errors and injuries, as evidenced by recent news reports about problems with automotive firmware. Such issues are not specific to a particular domain and are very common in safety-critical systems. In fact, such problems are often found when reviewing code from legacy systems designed and built more than 20 years ago and still operating, as in the avionics and aerospace domains. This blog post describes an effort at the SEI that aims to help engineers use time-proven architecture patterns (such as the publish-subscribe pattern or correct use of shared resources) and validate their correct application. Architecture Design and Analysis: Why it Matters Today's safety-critical systems are increasingly reliant on software. Software architecture is an important asset that impacts the overall development process: for example, good software architecture eases system upgrade and reuse while bad architectures can lead to unexpected rework when trying to modify a component. This trend will continue, especially because software size continues to grow at a significant rate and the early—and intentional—design of software architecture is an important tool in managing this complexity. Software architecture also helps system stakeholders reason about the system in its operational environment and detect potential flaws. Beyond these benefits, the early design and review of a software architecture can help avoid common software traps and pitfalls prior to implementation. A study by the National Institute of Standards and Technology found that 70 percent of software defects are introduced during the requirements and architecture design phases. What exacerbates the problem is the fact that 80 percent of those defects are not discovered until system integration testing or even later in the development lifecycle. Fixing these issues later has an adverse impact on product delivery schedule and also on development costs, In their paper "Software Reduction Top 10 List" software engineering researchers Barry Boehm and Victor Basili wrote that "finding and fixing a software problem is 100 times more expensive than finding and fixing it during the requirements and design phase." A group of SEI researchers have started an effort that details strategies for avoiding software architecture mistakes by using appropriate architecture patterns (such as the ones from the NASA reports) and validating their correct application. Specifically, we are working on tools to analyze software architecture, detect pattern usage, and check that system characteristics cannot undermine the benefits of the pattern. This approach promotes use of well-known methods to improve software quality, such as decoupling functions or reducing variable scope to make the software more modular. In the long term, such methods can help designers avoid common architecture traps and pitfalls from the beginning as well as the incurrence of potential rework later in the development process. From a practical perspective, this approach makes use of the Architecture Analysis and Design Language (AADL) for specifying an architecture pattern. We implemented a new analysis function in the Open Source AADL Tool Environment (OSATE) to validate correct use of the pattern and analyze pattern consistency with the other components. In particular, such a tool can detect any characteristic from the system environment that might impact use of the pattern. For example, in the case of the publish-subscribe pattern (a component sending data periodically to a receiver), one common mistake is a mismatch between the execution frequency of the publisher and subscriber, such as when the publisher sends data faster than the subscriber can handle it. Our validation tool analyzes the application of such a pattern and checks for timing mismatch, ensuring that the subscriber has enough time and resources to receive and handle all incoming data. Using and Validating Architecture Patterns The publish-subscribe pattern introduced above can be illustrated by a simplified weather station with two components: a temperature sensor (publisher) that periodically sends a value (temperature) to a monitor (subscriber) that computes statistics about the value including maximum, minimum, and average. Each component (the sensor and the monitor) is periodic: each executes at a fixed and predefined rate (for example, each second). Figure 1 illustrates the publish-subscribe pattern. As shown in this figure the communication uses a connection between two components. When the sensor publishes data it is stored in a buffer to make it available to the monitor that subscribes to the data. As both tasks are running at the same rate (1 second(s)), no data is lost or read twice. Figure 1. The publish-subscribe pattern without queued communication Changing the components’ characteristics may have important side-effects. For example, changing the execution rate of the sensor so that it is executed more frequently than the monitor causes data loss. The second execution of the sensor will overwrite the actual buffer on the monitor and replace the previous unread value. The consequences will be that some values are not processed by the monitor and that the result (minimum, maximum, and average temperature) is not accurate. A common workaround for this issue uses communication queues that can store several values. In our current example, we change the buffer dimension of the monitor so that it can handle two pieces of data. We illustrate such an architectural change in Figure 2. Figure 2 - The publish-subscribe pattern with queued communication In this case, the sensor is executed faster (500 milliseconds (ms)) than the monitor (1s). No data is lost because the monitor can contain two data values and read all of them when it is executed. A new problem may appear, however, if the buffer size or the execution period is modified. This type of issue may not be important to your system, and checking the correct application of the pattern depends on your system requirements. If the data being exchanged is of any particular importance, however, you must check that the pattern is applied correctly in the architecture. In this example (the publish-subscribe pattern), validating the correct application of the pattern requires that: without queued communication, the monitor is executed faster than the sensor with queued communication, the components’ periods and queue size are configured consistently to avoid data loss Timing and resource-dimension issues are among many in a 2011 NASA report that identifies issues related to an unintended acceleration problem in automotive software. The report states that software analysis tools detected more than 900 buffer overruns when the tools were used to analyze the automotive software that was experiencing the problem. The use of software architecture ensures that these types of issues can be detected and avoided during system design and not propagated to subsequent development stages. For that reason, it is important to not only make use of good architecture patterns, but to analyze an architecture to ensure correct pattern application and use. For our publish-subscribe example, we describe the architecture using  AADL. Our validation tool checks its correctness by analyzing the components’ characteristics. The following figures show our validation framework, with the left part illustrating the validation of a correct architecture and the right showing an error, highlighting a software architecture defect (inconsistent timing properties). To view a larger image of this figure, please click here. Use of architecture validation tool, by validating a correct architecture (left) or detecting inconsistent use of an architecture pattern (right). The Take Away Recent news reports illustrate the value of architecture analysis for improving software development, reducing potential rework costs, and avoiding delivery delays. In that context, SEI researchers are promoting the use of software architecture patterns in conjunction with analysis tools to check their correct application and thus, avoid typical architecture design trap and pitfalls. Our analysis tools look for architecture defects using validation rules, such as: Variable Scope. Variable scope defines what entities might read or write in a variable. An improperly defined variable scope limits software reuse (too many components depend on a shared global variable) or limits analysis by making it hard to trace what tasks read from or write to the variable. To avoid such defects, architects must analyze software architecture and check if variables are declared and used at the appropriate scope. From a technical perspective, our validation tool checks whether variables are declared with the appropriate scope according to their use (tasks or subprograms that accesses it) and advocate architecture changes when appropriate. Such an approach would avoid unnecessary use of global variables, which is usually a design mistake, as evidenced by a recent report from the National Highway Transportation Safety Administration on unintended acceleration in Toyota vehicles. The same report illustrates that this is a common trend and states that some automotive software can contain more than 2,200 global variable declarations with different types. Concurrency. Many software architectures include tasks that access shared resources (such as services, resources, data, etc.). A common mistake is to share data among several components that read and write new values without controlling concurrent access, which can lead to potential consistency issues. To overcome this problem, we advise using a concurrency control mechanism (such as semaphore or mutex) to avoid value inconsistencies and related race conditions. On the other hand, if only one task writes to the data, the concurrency mechanism might be avoided. Inappropriate use of multi-tasking features and locking mechanisms is the source of many software issues, as evidenced by the Flight Software Complexity Report issued by NASA. Using the appropriate mechanism is important in the context of safety-critical systems, as they may have limited resources, and use of such mechanisms introduces potentially unnecessary overhead. Examples of rules to check correct use of shared resources include: If more than two tasks write into shared data, the data must be associated with a locking mechanism (mutex, semaphore, etc.). If only one task writes into shared data, no locking mechanism is mandatory. We are working on several validation rules for analyzing the use of global variables and refactoring the software architecture so that software is decomposed into modules that can be reused and deployed on separate processing nodes variable assignment and modification are restricted to a limited scope (so that a variable cannot be modified anywhere.) data flow is clearly defined and bounded to a specific scope An outline of this effort, and our progress in developing this approach, is available online. All the validation technology is included in OSATE, our Eclipse-based AADL modeling framework under a free license. We invite you to use and test our approach, and then send us feedback. To improve existing patterns and add new ones, we also plan to interview safety-critical system engineers and designers so that we may adapt our work to existing industrial issues, expectations, and needs. If you are a software engineer or designer who would be interested in participating, please send an email to info@sei.cmu.edu. Additional Resources To read more about the approach that we are developing, please visithttps://wiki.sei.cmu.edu/aadl/index.php/Good_Software_Architecture_Practices_with_AADL To read the NASA Study on Flight Software Complexity, please visit http://www.nasa.gov/offices/oce/documents/FSWC_study.html To read the National Highway Transportation Safety Administration Study of Unintended Acceleration in Toyota Vehicles, please visit http://www.nhtsa.gov/UA

By Robert Nord, Senior Member of the Technical StaffSoftware Solutions Division (This blog post was co-authored by Ipek Ozkaya) As the pace of software delivery increases, organizations need guidance on how to deliver high-quality software rapidly, while simultaneously meeting demands related to time-to-market, cost, productivity, and quality. In practice, demands for adding new features or fixing defects often take priority. However, when software developers are guided solely by project management measures, such as progress on requirements and defect counts, they ignore the impact of architectural dependencies, which can impede the progress of a project if not properly managed. In previous posts on this blog, my colleague Ipek Ozkaya and I have focused on architectural technical debt, which refers to the rework and degraded quality resulting from overly hasty delivery of software capabilities to users. This blog post describes a first step towards an approach we developed that aims to use qualitative architectural measures to better inform quantitative code quality metrics. Technical debt is an increasingly critical aspect of producing cost-effective, timely, and high-quality software products. Recently, our research has focused on going beyond debt as a metaphor to investigating which measures a software development team can apply to effectively monitor changing qualities of software. These measures can take advantage of code quality if the goal is to optimize development qualities. Existing code measures alone, however, do not provide insight into overall architectural improvements due to the increasing complexity and context dependencies of software systems. We must investigate a range of measures to provide a multi-view architectural perspective of design time, run-time, and deployment time qualities. The goal of our research is to provide an architectural measurement framework that can assist in monitoring and improving high architectural risk areas of a system.  Informing Quantitative Metrics with Qualitative Measures Developers can apply off-the-shelf tools (such as Lattix, SonarGraph, SonarQube, Structure101) to understand architectural dependencies for change impact or rework analysis that rely on code metrics (such as stability, coupling, cohesion, cyclicity, complexity, etc.). These metrics are often helpful in improving code quality and can provide structural information about architectural dependencies and modifiability. Recent research has demonstrated, however, that such metrics fall short of providing overall architectural system guidance when used as they are. The question we ask is whether the relevance and use of such metrics can be improved to provide architectural guidance, as well. To address this question, we developed an approach to contextualize and focus the application of dependency analysis and architecture-relevant code quality and system modifiability metrics using architecture evaluations. Scenario-based architecture analysis offers a broad understanding of how a software-reliant system evolves over time and can form a basis for assessing the amount of rework that may be necessary in the foreseeable future. Using the architectural risks identified during scenario-based architecture analysis, we clarified the level of system decomposition where code quality metrics reveal relevant information. As outlined in our January 2012 blog post, An Architecture Focused Measurement Framework for Managing Technical Debt, our research on this topic is informed by real-world examples gathered from technical debt workshops. That blog post, authored by my colleague and co-author, Ipek Ozkaya, noted that an architecture-focused analysis approach helps manage technical debt by enabling software engineers to decide the best time to re-architect, thereby reducing the technical debt. CONNECT Our earlier work is part of an ongoing SEI research agenda to improve the integration of architecture practices within agile software development methods. To test our approach, we evaluated CONNECT, which is an open-source software system to exchange patient information among different healthcare providers at the local and national level. The developers of CONNECT used Scrum as their agile project management approach. They hold biweekly sprints (117 as of late 2013) and periodically release updates on the system (typically every quarter.) Code development is outsourced and takes place in different offices. Periodic code sprints bring all the developers together to synchronize their work. We chose to focus a portion of our research in the context of CONNECT since a team of SEI researchers was asked in November 2011 to complete an evaluation of the system that focused on quality attribute goals using the Architecture Tradeoff Analysis Method (ATAM). This analysis yielded a list of potential risks that the project needed to address. Among the risks cited, researchers referenced Adapter/Gateway separation: CONNECT initially separates the handling of messages from the integration with other systems. However, these roles have become confused over time, and it is not clear how the roles should be separated. The remainder of this post focuses on this risk theme that the ATAM identified as an area of major concern. While the ATAM provided a list of risks the project needed to address, we considered it to be a point-in-time representation of the project. A common response is to focus on short-term fixes at the expense of underlying causes for those risks, which are typically architectural in nature. In our examination of CONNECT, we tried to understand how architectural scenarios, developed during the architectural evaluation, impacted project narrative in the next major release, which included changes implemented as a result of the ATAM. In the case of CONNECT, we examined the JIRA issue tracker, which contained sprint and product backlogs. We looked at the feature requests and improvements in the backlog both before and after the ATAM and saw that it had an impact on what the development team worked on as there was a 22 percent increase in risk-related issues created after the ATAM was conducted. The CONNECT system underwent a significant re-architecting effort to reduce its dependency between the Adapter and the Gateway. Next, we sought to reconcile the risk themes identified in the ATAM with the dependencies extracted from the code and the automatically generated code quality measures. To analyze whether common modularity code metrics reflect the impact of changes related to the architectural risks and recommendations, we compared the baseline version of  CONNECT with the next version released after the developers re-architected the Adapter/Gateway dependency to mitigate the risk. We analyzed the code at three levels of decomposition in the code hierarchy: system-level decomposition. We analyzed the code of the entire CONNECT system, including middleware and all of its third-party dependencies. Overall, the modifiability metrics show improvements in atom count, internal dependencies, average impact, system stability, connectedness, connected strength, coupling, coupling strength, and system cyclicity. software-level decomposition. We analyzed the code from gov.hhs including packages, interfaces, and classes for CONNECT middleware, with the exception of third-party libraries. We found that the metrics demonstrate that the numbers of atoms and internal dependencies decreased, indicating that the package contained fewer code artifacts. System cyclicity and connectedness increased, indicating a higher likelihood of change. Given that system stability measured 98 percent, the system appears to be highly stable regardless of the architecture change at the package level of decomposition. For this case, where the stability metric did not change, a lower connectedness strength in the next release suggests the design would be architecturally less sensitive to change. package-level decomposition. At this lowest level of decomposition, we focused on the document query package gov.hhs.fha.nhinc.docquery, which reflected the architectural changes based on Adapter-Gateway integration risk. At this level of decomposition, the system size is significantly reduced as is the average impact. This metric indicates that changes to the document query system will be less likely to resonate throughout the rest of the system. While the system stability level at this metric improved, we noted a decline in connectedness and connected strength. The stability metric, which provides a system’s overall sensitivity to changes, reliably reflected the system’s architecture when appropriately focused on the CONNECT system’s selected decomposition level, and excluding the rest of the system. Navigating down the hierarchy in this manner allowed us to see where the code metrics show any significant change, thereby indicating whether the architecture has improved or deteriorated during its evolution. Spotting areas of rework requires understanding the context provided by architecture and quality concerns that influence architectural evolution. When focused on the problematic elements of the system, at a suitable level of decomposition in the package structure, selected code-based metrics for assessing ripple effects reflect the improvements on the architecture where rework has been done. When we included the entire system in our stability assessment, however, the system stability measurement reported that it was close to 100 percent, given the large number of elements that falsely indicated a stable system.  We found that applying existing metrics consistently requires the ability to choose appropriate elements of the system at a suitable level of decomposition because dependency analysis is quite sensitive to the size of the graph and its context, as our results demonstrated. Looking Ahead Our research aims to bring the architecture analysis and developer environments closer together. Our goal is to create repeatable analysis and validation on metrics that provide architectural information and scale to systems of realistic size. On another front, we have joined forces with members of the architecture and metrics community to host a workshop on software architecture metrics. Our aim is to improve measurement techniques for architecture that yield reliable, consistent, and repeatable results by discussing progress on architecture metrics, measurement, and analysis gathering empirical evidence on the use and effectiveness of metrics identifying priorities for future research The workshop, which will be held in April 2014, will bring together a cross-section of experts in academia and industry in the areas of dependency analysis, architecture metrics, analysis and evaluation, software analytics, empirical software engineering and measurement. Additional Resources For more information about the First International Workshop on Software Architecture Metrics, which will be held April 7, 2014 in conjunction with the Working IEEE/IFIP Conference on Software Architecture (WICSA), or to submit a paper, please visit  www.sei.cmu.edu/community/sam2014.

By Will DormannSenior Member of the Technical StaffCERT Vulnerability Analysis Team Occasionally this blog will highlight different posts from the SEI blogosphere. Today we are highlighting a recent post by Will Dormann, a senior member of the technical staff in the SEI’s CERT Division, from the CERT/CC Blog. In this post, Dormann describes how to modify the CERT Failure Observation Engine (FOE),when he encounters apps that "don’t play well" with the FOE. The FOE is a software testing tool that finds defects in applications running on the Windows platform.

By Douglas C. Schmidt Principal Researcher As part of our mission to advance the practice of software engineering and cybersecurity through research and technology transition, our work focuses on ensuring that software-reliant systems are developed and operated with predictable and improved quality, schedule, and cost. To achieve this mission, the SEI conducts research and development activities involving the Department of Defense (DoD), federal agencies, industry, and academia. As we look back on 2013, this blog posting highlights our many R&D accomplishments during the past year. Before turning to our accomplishments, it’s important to note that 2013 brought the arrival of Kevin Fall as deputy director and chief technology officer. In the blog post, A New CTO and Technical Strategy for the SEI, Fall provided some background on his experience, as well as his technical goals for the SEI: Develop an even higher quality and more cohesive research program Increase collaboration with Carnegie Mellon University and other academic researchers Enhance accessibility to the SEI’s work Kevin leads R&D at the SEI, which benefits the DoD and other sponsors by identifying and solving key technical challenges facing developers and managers of current and future software-reliant systems. The R&D work at the SEI presented in this blog focused on a range of software engineering and cybersecurity areas, including Securing the cyber infrastructure. This area focuses on enabling informed trust and confidence in using information and communication technology to ensure a securely connected world to protect and sustain vital U.S. cyber assets and services in the face of full-spectrum attacks from sophisticated adversaries. Advancing disciplined methods for engineering software. This area focuses on improving the availability, affordability, and sustainability of software-reliant systems through data-driven models, measurement, and management methods to reduce the cost, acquisition time, and risk of our major defense acquisition programs. Accelerating assured software delivery and sustainment for the mission. This area focuses on ensuring predictable mission performance in the acquisition, operation, and sustainment of software-reliant systems to expedite delivery of technical capabilities to win the current fight. Innovating software for competitive and tactical advantage. This area focuses on safety-critical avionics, aerospace, medical, and automotive systems, all of which are becoming increasingly reliant on software. Other posts in this area highlight innovations that revolutionize development of assured software-reliant systems to maintain the U.S. competitive and tactical edge in software technologies vital to national security. What follows is a sampling of the SEI’s R&D accomplishments in each of these areas during 2013, with links to additional information about these projects. Securing the Cyber Infrastructure Some cybersecurity attacks against DoD and other government organizations are caused by disgruntled, greedy, or subversive insiders, employees, or contractors with access to that organization’s network systems or data. Over the past 13 years, researchers at the CERT Insider Threat Center have collected incidents related to malicious activity by insiders from a number of sources, including media reports, the courts, the United States Secret Service, victim organizations, and interviews with convicted felons. In a series of blog posts, members of the research team have presented some of the 26 patterns identified by analyzing the insider threat database. Through our analysis, insider threat researchers have identified more than 100 categories of weaknesses in systems, processes, people, or technologies that allowed insider threats to occur. One aspect of their research focuses on identifying enterprise architecture patterns that organizations can use to protect their systems from malicious insiders. Now that we’ve developed 26 patterns, our next priority is to assemble these patterns into a pattern language that organizations can use to bolster their resources and make them more resilient against insider threats. The blog post, A Multi-Dimensional Approach to Insider Threat, is the third installment in a series that described research to create and validate an insider threat mitigation pattern language to help organizations balance the cost of security controls with the risk of insider compromise. Exposed vulnerable assets make a network a target of opportunity, or low-hanging fruit for attackers. According to the 2012 Data Breach Investigations Report, of the 855 incidents of corporate data theft reported in 2012, 174 million records were compromised. Of that figure, 79 percent of victims were targets of opportunity because they had an easily exploitable weakness, according to the report. The blog post Network Profiling Using Flow highlighted recent research in how a network administrator can use network flow data to create a profile of externally-facing assets on mid- to large-sized networks. New malicious code analysis techniques and tools being developed at the SEI will better counter and exploit adversarial use of information and communication technologies. Through our work in cybersecurity, we have amassed millions of pieces of malicious software in a large malware database. Analyzing this code manually for potential similarities and identifying malware provenance is a painstaking process. The blog post Prioritizing Malware Analysis outlined a research collaborative with CMU’s Robotics Institute aimed at developing an approach to prioritizing malware samples in an analyst’s queue (allowing analysts to home in on the most destructive malware first) based on the file’s execution behavior. Another blog post, Semantic Comparison of Malware Functions, described research aimed at helping analysts derive precise and timely actionable intelligence to understand and respond to malware. The approach described in the post uses the semantics of programming languages to determine the origin of malware. The blog post Analyzing Routing Tables highlighted another aspect of our work in cybersecurity.  The post detailed maps that a CERT researcher developed using Border Gateway Protocol (BGP) routing tables to show the evolution of public-facing autonomous system numbers (ASNs). These maps help analysts inspect the BPG routing tables to reveal disruptions to an organization’s infrastructure. They also help analysts glean geopolitical information for an organization, country, or a city-state, which helps them identify how and when network traffic is subverted to travel nefarious alternative paths to place communications deliberately at risk. Exclusively technical approaches toward attaining cybersecurity have created pressures for malware attackers to evolve technical sophistication and harden attacks with increased precision, including socially engineered malware and distributed denial of service (DDoS) attacks. A general and simple design for achieving cybersecurity remains elusive, and addressing the problem of malware has become such a monumental task that technological, economic, and social forces must join together to address this problem. The blog post Deterrence for Malware: Towards a Deception-Free Internet, detailed a collaboration between the SEI’s CERT Division and researchers at the Courant Institute of Mathematical Sciences at New York University. Through this collaboration, researchers aim to understand and seek complex patterns in malicious use cases within the context of security systems and develop an incentives-based measurement system that would evaluate software and ensure a level of resilience to attack. Our security experts in the CERT Division are often called upon to audit software and provide expertise on secure coding practices. The blog posting Using the Pointer Ownership Model to Secure Memory Management in C and C++, described a research initiative aimed at eliminating vulnerabilities resulting from memory management problems in C and C++.  Memory problems in C and C++ can lead to serious software vulnerabilities including difficulty fixing bugs, performance impediments, program crashes (including null pointer deference and out-of-memory errors), and remote code execution. Advancing Disciplined Methods for Engineering Software New data sources, ranging from diverse business transactions to social media, high-resolution sensors, and the Internet of Things, are creating a digital tidal wave of big data that must be captured, processed, integrated, analyzed, and archived. Big data systems that store and analyze petabytes of data are becoming increasingly common in many application areas. These systems represent major, long-term investments requiring considerable financial commitments and massive scale software and system deployments. With analysts estimating data storage growth at 30 to 60 percent per year, organizations must develop a long-term strategy to address the challenge of managing projects that analyze exponentially growing data sets with predictable, linear costs. The blog post, Addressing the Software Engineering Challenges of Big Data, described a lightweight risk reduction approach called Lightweight Evaluation and Architecture Prototyping (for Big Data). The approach is based on principles drawn from proven architecture and technology analysis and evaluation techniques to help the DoD and other enterprises develop and evolve systems to manage big data. The post Architecting Systems of the Future is the first in a series highlighting work from the SEI’s newest program, the Emerging Technology Center. This post highlighted research aimed at creating a software library that can exploit the heterogeneous parallel computers of the future and allow developers to create systems that are more efficient in terms of computation and power consumption.  Accelerating Assured Software Delivery and Sustainment for the Mission SEI researchers work with acquisition professionals and system integrators to develop methods and processes that enable large-scale software-reliant government systems to innovate rapidly and adapt products and systems to emerging needs within compressed time frames and within constrained budgets. To deliver enhanced integrated warfighting capability at lower cost across the enterprise and over the lifecycle, the DoD must move away from stove-piped solutions and towards a limited number of technical reference frameworks based on reusable hardware and software components and services. There have been previous efforts in this direction, but in an era of sequestration and austerity, the DoD has reinvigorated its efforts to identify effective methods of creating more affordable acquisition choices and reducing the cycle time for initial acquisition and new technology insertion. In 2013, we published two postings as part of an ongoing series on Open Systems Architecture (OSA) that focused on: Affordable Combat Systems in the Age of Sequestration expanded upon earlier coverage of how acquisition professionals and system integrators can apply OSA practices to decompose large monolithic business and technical designs into manageable, capability-oriented frameworks that can integrate innovation more rapidly and lower total ownership costs. The Architectural Evolution of DoD Combat Systems described the evolution of DoD combat systems from ad hoc stovepipes to more modular and layered architectures. Despite substantial advances in technical reference frameworks during the past decade, widespread adoption of affordable and dependable OSA-based solutions has remained elusive. It is therefore important to look at past open-systems efforts across the DoD to understand what worked, what hasn’t, and what can be done to make the development of systems more successful in the future. Government agencies, including the departments of Defense, Veteran Affairs, and Treasury, are being asked by their government program offices to adopt Agile methods. These organizations have traditionally used a waterfall life cycle model (as epitomized by engineering "V" charts). Programming teams in these organizations are accustomed to being managed via a series of document-centric technical reviews that focus on the evolution of the artifacts that describe the requirements and design of the system rather than its evolving implementation, as is more common with Agile methods. As a result of the factors outlined above, many organizations struggle to adopt Agile practices. For example, acquisition professionals often wonder how to fit Agile measurement practices into their progress tracking systems. They also find it hard to prepare for technical reviews that don’t account for both implementation artifacts and the availability of requirements/design artifacts. A team of SEI researchers is dedicated to helping government programs prepare for and, if appropriate, implement Agile. In 2013, the SEI continued its series of blog posts on the Readiness & Fit Analysis (RFA) approach, which helps organizations understand the risks involved when contemplating or embarking on the adoption of new practices, in this case Agile methods. Blog installments published in the series thus far outlined factors to study when considering agile adoption including business and acquisition (discussed in the first post in this series) organizational climate (discussed in the second post and continued in the third post) project and customer environment (discussed in the fourth post) The verification and validation of requirements are a critical part of systems and software engineering. The importance of verification and validation (especially testing) is a major reason that the traditional waterfall development cycle underwent a minor modification to create the V model that links early development activities to their corresponding later testing activities. A blog post published in November introduced three variants on the V model of system or software development that make it more useful to testers, quality engineers, and other stakeholders interested in the use of testing as a verification and validation method. A widely cited study for the National Institute of Standards & Technology (NIST) reports that inadequate testing methods and tools annually cost the U.S. economy between $22.2 billion and $59.5 billion, with roughly half of these costs borne by software developers in the form of extra testing and half by software users in the form of failure avoidance and mitigation efforts. The same study notes that between 25 percent and 90 percent of software development budgets are often spent on testing. In April, we kicked off a series on common testing problems that highlighted results of an analysis that documents problems that commonly occur during testing. Specifically, this series of posts identifies and describes 77 testing problems organized into 14 categories; lists potential symptoms by which each can be recognized, potential negative consequences, and potential causes; and makes recommendations for preventing them or mitigating their effects. The first post in the series explored issues surrounding the reality that software testing is less effective, less efficient, and more expensive than it should be. The second posting highlighted results of an analysis that documents problems that commonly occur during testing. Innovating Software for Competitive and Tactical Advantage Mission- and safety-critical avionics, aerospace, defense, medical, and automotive systems are increasingly reliant on software. Malfunctions in these systems can have significant consequences including mission failure and loss of life, so they must be designed, verified, and validated carefully to ensure that they comply with system specifications and requirements and are error free.  Ensuring these properties in a timely and cost-effective manner is also vital to ensure competitive advantage for companies who produce these technologies. In March, we kicked off a series of blog posts that explored recent developments with the Architecture Analysis Design Language (AADL) standard, which provides formal modeling concepts for the description and analysis of application systems architecture in terms of distinct components and their interactions. The series aimed to highlight how the use of AADL helps alleviate mismatched assumptions between the hardware, software, and their interactions that can lead to system failures. The series has included the following posts thus far Detecting Architecture Traps and Pitfalls in Safety-Critical Software highlighted an effort at the SEI that aims to help engineers use time-proven architecture patterns (such as the publish-subscribe pattern or correct use of shared resources) and validate their correct application. AADL: SAVI and Beyond described the use of AADL in the aerospace industry to improve safety and reliability. AADL in the Medical Domain detailed how AADL is being used in medical devices and highlights the experiences of a practitioner whose research aims to address problems with medical infusion pumps.  AADL Tools: Leveraging the Ecosystem provided an overview of existing AADL tools and highlights the experience of researchers and practitioners who are developing and applying AADL tools to production projects.  Introduction to the Architecture Analysis and Design Language, the first post in the series, detailed the initial foundations of AADL, which defines a modeling notation based on a textual and graphic representation that is used by development organizations to conduct lightweight, rigorous—yet comparatively inexpensive—analyses of critical real-time factors, such as performance, dependability, security, and data integrity. Another post highlighting our work on safety-critical systems introduced the Reliability Validation and Improvement Framework that will lead to early defect discovery and incremental end-to-end validation. The Advanced Mobile Systems Initiative at the SEI focuses on helping soldiers and first responders, whether they are in a tactical environment (such as a war zone) or responding to a natural disaster. Both scenarios lack effective, context-aware use and adaptation of tactical resources and the ability to get relevant information when they critically need it. Software and system capabilities do not keep pace with these users’ changing needs and must be adapted at the operational edge, or periphery, of the network. Posts describing research in this area include the following Situational Awareness Mashups at the Tactical Edge detailed efforts to create the Edge Mission-Oriented Tactical App Generator (eMontage), a software prototype that allows warfighters and first responders to rapidly integrate geotagged situational awareness data from multiple remote data sources. National Deployment of the Wireless Emergency Alerts Systems described how the SEI’s work on architecture, integration, network security, and project management is assisting in implementing the WEA system, so it can handle a large number of alert originators and provide an effective nationwide wireless emergency warning system. Building Next-generation Autonomous Systems focused on a new research effort at the SEI called Self-governing Mobile Ad-hocs with Sensors and Handhelds (SMASH) that is forging collaborations with researchers, professors, and students with the goal of enabling more effective search-and-rescue crews.  Application Virtualization for Cloudlet-based Cyber Foraging at the Edge is the latest in a series that recounted research aimed at exploring the applicability of application virtualization as a strategy for cyber-foraging in resource-constrained environments. Concluding Remarks As you can see from this summary of accomplishments, 2013 has been a highly productive and exciting year for the SEI technical staff. Moreover, this blog posting just scratches the surface of SEI R&D activities. Please come back regularly to the SEI Blog for coverage of these and many other topics we’ll be doing in the coming year. As always, we’re interested in new insights and new opportunities to partner on emerging technologies and interests. We welcome your feedback and look forward to engaging with you on the blog, so please feel free to add your comments below. Additional Resources For the latest SEI technical reports and papers, please visitwww.sei.cmu.edu/library/reportspapers.cfmFor more information about R&D at the SEI as well as opportunities for collaboration, please visitwww.sei.cmu.edu/research/

Second in a Series By Charles B. WeinstockDistinguished Principal ResearcherSoftware Solutions Division Software used in safety-critical systems—such as automotive, avionics, and healthcare applications, where failures could result in serious harm or loss of life—must perform as prescribed. How can software developers and programmers offer assurance that the system will perform as needed and with what level of confidence? In the first post in this series, I introduced the concept of the assurance case as a means to justify safety, security, or reliability claims by relating evidence to the claim via an argument. In this post I will discuss Baconian probability and eliminative induction, which are concepts we use to explore properties of confidence that the assurance case adequately justifies its claim about the subject system. The Case for Confidence Assurance cases are now becoming a standard practice in the development and deployment of safety-critical systems. In May 2013, at the First International Workshop on Assurance Cases for Software-Intensive Systems, it was stated that Several certification standards and guidelines, e.g., in the defense, transportation (aviation, automotive, rail), and healthcare domains, now recommend and/or mandate the development of assurance cases for software-intensive systems. In the first post in this series, I discussed our research on assurance cases and how they can be used when developing software-intensive systems. It is not enough, however, to simply have an assurance case. It is also important to understand why you should have confidence in the assurance case. Achieving confidence is important to all stakeholders in the subject system, including those acquiring the system, those producing the system, those using the system, and (in certain cases) those certifying or otherwise evaluating the system. It is impossible to examine every possible circumstance covered by an assurance case.  One approach to the problem of achieving confidence in an assurance case is to provide a parallel argument that gives reasons why the assurance case should be believed. Hawkins, Kelly, Knight and Graydon have taken this approach with their concept of the "confidence case." We have taken a different approach. Consider the following notional assurance case arguing that the system is safe (taken from the first posting in this series):   Figure 1: A Notional Assurance Case We want to be able to understand how the evidence (Ev1…Ev3) leads to confidence that the claim (C1) holds and by how much. We would also like to understand what "confidence" in the claim means, and how existing confidence can be increased. The remainder of this post will focus on a more detailed description of Baconian probability and eliminative induction. Two Forms of Induction Deciding how to use evidence as a means of evaluating belief in a hypothesis is a classic philosophical problem—one that is classically solved through the use of induction. When we talk about induction, we typically mean enumerative induction. But there is another form of induction, eliminative induction.In enumerative induction, support for a hypothesis is increased every time confirming evidence is found. Past experience forms a basis for an expectation of future behavior. For example, consider a person coming into a room and turning on a light. If the light has gone on before, the person will have some confidence that it will go on again. If it has gone on many times before, the person should have even higher confidence that it will go on the next time. Eliminative induction in contrast to enumerative induction looks for reasons to doubt the hypothesis. So our person deciding whether to believe a light will go on when he throws the switch will have to consider reasons why it might not, and then eliminate them. When the person walks into the room, he may have some initial, perhaps unfounded, confidence that the light will go on when he throws the switch. If he checks the wiring and finds that the switch is connected, he will have more confidence. If he verifies that there is power to the switch, he will have still more confidence. And, if he shakes the bulb and does not hear a rattle, he will have even more confidence that the light will go on. Eliminative Argumentation and the Confidence Map We can now begin to answer the questions posed in our notional assurance case shown in Figure 1. The evidence in the case leads to increased confidence in the claim to the extent that it removes doubts about the claim. We will achieve increased confidence in the claim C1 as we eliminate more doubts. This method is the idea behind what we call eliminative argumentation, which is based on assurance cases, eliminative induction, and defeasible reasoning.Reasoning about doubts in an argument is a use of defeasible reasoning. A defeasible argument is always subject to revision as we learn more about the claims and evidence that form the argument. There are only three kinds of doubts relevant to an assurance case: we can doubt claims, we can doubt evidence, and we can doubt the inference between claims or between two claims or between a claim and its supporting evidence. We show these doubts explicitly in a modified assurance case— a confidence map. Figure 2: The "light turns on" assurance case fragment Figure 2 is a portion of the assurance case for the claim that the light turns on when the switch is thrown. Notice that each of the three sub-claims of C1.1 is based on a doubt about why the light might not turn on when the switch is thrown. In a confidence map these doubts are expressed explicitly. Figure 3: Rebutting defeaters Figure 3 shows the beginning of a confidence map for the "light turns on" example. R2.1 … R2.3 are called rebutting defeaters. They attack the validity of claim C1.1. If any of these defeaters is true, then the claim is invalid. If all of the defeaters can be eliminated, then we have no reason to doubt the claim. If we have no knowledge about the truth or falsity of one or more of these defeaters, then there remains a doubt as to whether claim C1.1 is valid.There is an implicit inference rule in the confidence map shown above; namely, if all of the rebutting defeaters are shown to be false, then the claim is valid. It may be that there are additional, as yet unidentified, rebutting defeaters to claim C1.1. In the confidence map notation we make this explicit as shown in Figure 4. Figure 4: Adding an inference rule and attacking it via an undercutting defeater The confidence map now makes the inference explicit and adds an additional undercutting defeater, namely UC3.3, which raises a doubt about the sufficiency of the inference rule. Figure 5: Attacking the evidence via an undermining defeater Figure 5 expands one leg of the confidence map to show the evidence that is being used to eliminate defeater R2.3, namely that the bulb does not rattle when shaken. A rattle would indicate a broken filament. The figure also shows an example of an undermining defeater—one that questions the evidence by suggesting that the person shaking the bulb may be hard of hearing. It also includes a second inference rule (that if a bulb does not rattle then it is good), which is undercut by a defeater suggesting that the bulb may not be incandescent, and therefore the absence of a rattle may be meaningless. The grey circle under the undercutting defeater indicates that we have assumed that the bulb is, in fact, incandescent and will consider that defeater to be eliminated without any further analysis. Of course this does not represent a complete confidence map, even for the elimination of R2.3. For instance the bulb may not rattle when shaken because the filament did not burn out, but merely was defective. Measuring Confidence Figure 6: Counting confidence At the beginning of this post, I stated that one of our goals was to measure confidence. One way to go about this is to simply count defeaters near the leaves of the map and then count uneliminated defeaters. In Figure 6, I recap the confidence map as discussed. In this example, we have eliminated one defeater (UC4.2 as indicated by the grey circle) out of a total of five. Expressed as 1|5, this is an example of Baconian probability (named after Sir Francis Bacon, who first proposed the idea of measuring degrees of certainty). Note that the Baconian probability is not a ratio and cannot be reduced. 1|5 is read "one out of five" while 2|10 would be read "two out of ten" a very different level of doubt: in the first case there are four unresolved doubts, whereas in the second there are eight. We can increase our confidence in C1.1 by eliminating additional defeaters. For instance, if the examiner has recently undergone a hearing test, we could eliminate UM4.1. Conclusion This blog post discusses achieving confidence in a claim and introduces the idea of defeaters and confidence maps. To review, doubt in an argument can result from uncertainty about the claims, evidence, or inferences. Such uncertainty is represented by a defeater. A rebutting defeater attacks a claim by suggesting a reason why it can be false. An undermining defeater attacks the evidence and asks why the evidence may be compromised. Finally, an undercutting defeater attacks the inference rule and asks how the premise can be OK, but the conclusion uncertain. Confidence in a claim increases as the reasons for doubt are eliminated. If we have eliminated zero defeaters, then we have no reason to have confidence in the claim. If we have eliminated all of the defeaters, then we have no reason to doubt the claim. But if there are still uneliminated defeaters, then there remains some doubt about the claim. It may be hard to completely eliminate a defeater. Suppose we know little about the light bulb examiner, but we know that 1 percent of the population as a whole will have trouble hearing a rattle in a defective light bulb. Shouldn’t we be able to mostly eliminate defeater UM4.1 on that basis? That and related topics will be discussed in the next article in this series. Additional Resources To read the SEI paper Measuring Assurance Case Confidence using Baconian Probabilities, please visit http://www.sei.cmu.edu/library/assets/whitepapers/icsews13assure-id5-p-16156-preprint.pdf To read the SEI paper Eliminative Induction: A Basis for Arguing System Confidence, please visit http://www.sei.cmu.edu/library/assets/whitepapers/icse13nier-id10-p-15833-preprint.pdf To read the SEI Technical Report, Toward a Theory of Assurance Case Confidence, please visit http://www.sei.cmu.edu/reports/12tr002.pdf To read the Hawkins, Kelly, Knight, and Graydon paper A New Approach to Creating Clear Safety Arguments, please visit http://www-users.cs.york.ac.uk/~rhawkins/papers/HawkinsSSS11.pdf

By Ian GortonSenior Member of the Technical StaffSoftware Solutions Division Many types of software systems, including big data applications, lend them themselves to highly incremental and iterative development approaches. In essence, system requirements are addressed in small batches, enabling the delivery of functional releases of the system at the end of every increment, typically once a month. The advantages of this approach are many and varied. Perhaps foremost is the fact that it constantly forces the validation of requirements and designs before too much progress is made in inappropriate directions.  Ambiguity and change in requirements, as well as uncertainty in design approaches, can be rapidly explored through working software systems, not simply models and documents. Necessary modifications can be carried out efficiently and cost-effectively through refactoring before code becomes too ‘baked’ and complex to easily change. This posting, the second in a series addressing the software engineering challenges of big data, explores how the nature of building highly scalable, long-lived big data applications influences iterative and incremental design approaches. Iterative, incremental development approaches are embodied in agile development methods, such as XP and Scrum. While the details of each approach differ, the notion of evolutionary design is at the core of each. Agile software architects eschew large, planned design efforts (also known as Big Design Up Front), in lieu of just enough design to meet deliverable goals for an iteration. Design modifications and improvements occur as each iteration progresses, providing on-going course corrections to the architecture and ensuring that only features that support the current targeted functionality are developed. Martin Fowler provides an excellent description of the pros and cons of this approach. He emphasizes the importance of test-driven development and continuous integration as key practices that make evolutionary design feasible. In a similar vein, at the SEI we are developing an architecture-focused approach that can lead to more informed system design decisions that balance short-term needs with long-term quality.Evolutionary, emergent design encourages lean solutions and avoids over-engineered features and software architectures. This design approach limits time spent on tasks such as updating lengthy design documentation. The aim is to deliver, in as streamlined a manner as possible, a system that meets its requirements. There is, of course, an underlying assumption that must hold for evolutionary design to be effective: change is cheap. Changes that are fast to make can easily be accommodated within short development cycles. Not all changes are cheap, however. Cyber-physical systems, where hardware-software interfaces are dominant, offer prominent examples of systems in which hardware modifications or unanticipated deployment environments can lead to changes with long development cycles. Other types of change can be expensive in purely software systems, as well. For example, poorly documented, tightly coupled, legacy code can rarely be successfully replaced in a single iteration. Incorporating new, third-party components or subsystems can involve lengthy evaluation, prototyping, and development cycles, especially when negotiations with vendors are involved. Likewise, architectural changes—for example, moving from a master-slave to a peer-to-peer deployment architecture to improve scalability—regularly require a fundamental and widespread re-design and refactoring that must be spread judiciously over several development iterations. Evolutionary Design and Big Data Applications As we described in a previous blog post, our research focuses on addressing the challenges of building highly scalable big data systems. In these systems, the requirements for extreme scalability, performance, and availability introduce complexities that require new design approaches from the software engineering community. Big data solutions must adopt highly distributed architectures with data collections that are (1) partitioned over many nodes in clusters to enhance scalability and (2) replicated to increase availability in the face of hardware and network failures. NoSQL distributed database architectures provide many of the capabilities that make scalability feasible at acceptable costs. They also introduce inherent complexities that force applications to perform the following tasks: explicitly handle data consistency tolerate a wide range of hardware and software faults track component monitoring and performance measurement so that operators have visibility into the behavior of the deployment Due to the size of their deployment footprint, big data applications are often deployed on virtualized, cloud platforms. Clouds platforms are many and varied in their nature, but generally offer a set of services for application configuration, deployment, security, data management, monitoring, and billing for use of processor, disk, and network resources. A number of cloud platforms from service providers, such as Amazon Web Services and Heroku, as well as open-source systems, such as OpenStack and Eucalyptus, are available for deploying big data applications in the cloud. In this context of big data applications deployed on cloud platforms, it’s interesting to examine the notion of evolutionary system design in an iterative and incremental development project. Recall that evolutionary design is effective as long as change is cheap. Hence, are there elements of big data applications where change is unlikely to be a straightforward task, and that might, in turn, require major rework and perhaps even fundamental architecture changes for an application? We posit that there are two main areas in big data applications where change is likely so expensive and complex that it warrants a judicious  upfront architecture design effort. These two areas revolve around changes to data management and cloud deployment technologies: Data Management Technologies. For many years, relational database technologies dominated data management systems. With a standard data model and query language, competitive relational database technologies share many traits, which makes moving to another platform or introducing another database into an application relatively straightforward. In the last five years, NoSQL databases have emerged as foundational building blocks for big data applications. This diverse collection of NoSQL technologies eschews standardized data models and query languages. Each technology employs radically different distributed data management mechanisms to build highly scalable, available systems. With different data models, proprietary application programming interfaces (APIs) and totally different runtime characteristics, any transition from one NoSQL database to another will likely have fundamental and widespread impacts on any code base. Cloud Deployments. Cloud platforms come in many shapes and sizes. Public cloud services provide hosting infrastructures for virtualized applications and offer sophisticated software and hardware platforms that support pay-as-you-use cost models. Private cloud platforms enable organizations to create clouds behind their corporate firewalls. Again, private clouds offer sophisticated mechanisms for hosting virtualized applications on clusters managed by the development organization. Like NoSQL databases, little commonality exists between various public and private cloud offerings, making a migration across platforms a daunting proposition with pervasive implications for application architectures. In fact, a whole genre of dedicated cloud migration technologies, including Yuruware and Racemi, is emerging to address this problem. Where opportunities for new tools such as these exist, the problem they are addressing is likely not something that can be readily accommodated in an evolutionary design approach. LEAP(4BD) Our Lightweight Evaluation and Architecture Prototyping for Big Data (LEAP4BD) method reduces the risks of needing to migrate to a new database management system by ensuring a thorough evaluation of the solution space is carried out in the minimum of time and with minimum effort. LEAP(4BD) provides a systematic approach for a project to select a NoSQL database that can satisfy its requirements. This approach is amenable to iterative and incremental design approaches, because it can be phased across one or more increments to suit the project’s development tempo. A key feature of LEAP(4BD) is its NoSQL database feature evaluation criteria. This ready-made set of criteria significantly speeds up a NoSQL database evaluation and acquisition effort. To this end, we have categorized the major characteristics of data management technologies based upon the following areas: Query Language—characterizes the API and specific data manipulation features supported by a NoSQL database Data Model—categorizes core data organization principles provided by a NoSQL database Data Distribution—analyzes the software architecture and mechanisms that are used by a NoSQL database to distribute data Data Replication—determines how a NoSQL database facilitates reliable, high performance data replication Consistency—categorizes the consistency model(s) that a NoSQL database offers Scalability—captures the core architecture and mechanisms that support scaling a big data application in terms of both data and request load increases Performance—assesses mechanisms used to provide high-performance data access Availability—determines mechanisms that a NoSQL database uses to provide high availability in the face of hardware and software failures Modifiability—questions whether an application data model be easily evolved and how that evolution impacts clients Administration and Management—categorizes and describe the tools provided by a NoSQL database to support system administration, monitoring and management Within each of these categories, we have detailed evaluation criteria that can be used to differentiate big data technologies. For example, here’s an extract from the Data Model evaluation criteria: Data Model style a.    Relationalb.    Key-Valuec.    Documentd.    Columne.    Graphf.    XMLg.    Object Data item identification    a.    Key-value for each fieldb.    Objects in same store can have variable formats types storedc.    Opaque data items  that needs application interpretationd.    Fixed or variable schemae.    Embedded hierarchical data items supported (e.g. sub documents) Data Item keya.    Automatically allocatedb.    Composite keys supportedc.    Secondary indexes supportedd.    Querying on non-key metadata supported Query Styles a.    Query by keyb.    Query by partial keyc.    Query by non-key values        i.    Indexed       ii.    Non-indexedd.    Text search in data items       i.    Indexed       ii.    Non-indexed In LEAP(4BD), we first work with the project team to identify features pertinent to the system under development. These features help identify a specific set of technologies that will best support the system. From there, we weight individual features according to system requirements and evaluate each candidate technology against these features. LEAP(4BD) is supported by a knowledge base that stores the results of our evaluations and comparisons of different NoSQL databases. We have pre-populated the LEAP(4BD) knowledge base with evaluations of specific technologies (e.g., MongoDB, Cassandra, and Riak) with which we have extensive experience. Each evaluation of a new technology adds to this knowledge base, making evaluations more streamlined as the knowledge base grows. Overall, this approach provides a systematic, quantitative, and highly transparent approach that quickly provides a ranking of the various candidate technologies according to project requirements. As we have demonstrated thus far in this series, there are many facets to LEAP(4BD). The next post in this series on Big Data will explain the prototyping phase. In the meantime, we’re keen to hear from developers and architects who are evaluating big data technologies, so please feel free to share your thoughts in the comments section below. Additional Resources To listen to the podcast, An Approach to Managing the Software Engineering Challenges of Big Data, please visithttp://url.sei.cmu.edu/iq

By Rick Kazman Senior Member of the Technical Staff Software Solutions Division The process of designing and analyzing software architectures is complex. Architectural design is a minimally constrained search through a vast multi-dimensional space of possibilities. The end result is that architects are seldom confident that they have done the job optimally, or even satisfactorily. Over the past two decades, practitioners and researchers have used architectural patterns to expedite sound software design. Architectural patterns are prepackaged chunks of design that provide proven structural solutions for achieving particular software system quality attributes, such as scalability or modifiability. While use of patterns has simplified the architectural design process somewhat, key challenges remain. This blog explores these challenges and our solutions for achieving system security qualities through use of patterns. One common approach for realizing system security is through authentication and authorization. The question for the architect is "How do I design them into the software?  What do I have to build in to achieve them?" To resolve this dilemma, the developer might choose to use a pattern. By way of analogy, a building architect might decide to make a truss to support a roof and so choose one of the standard available designs for trusses. The architect is starting with a pre-packaged piece of design. A pattern in software architecture is a piece of design, just like a truss in a physical building. It is commonplace, however, for developers to implement architectural patterns incorrectly. Due to such factors as ambiguity in specifications or miscommunication during the development process, the transition from design to code often does not go smoothly. Translating patterns into code involves the risk for error that is inherent in any translation process. Moreover, there are interpretation issues regarding how any pattern should be implemented; different programmers often implement the same pattern differently. Patterns also degrade over time, due to changes made during maintenance. Programmers often insert seemingly innocuous changes to the code without realizing that they’re actually undermining the intent of the pattern. For example, let’s say that you’re designing a system where A should never call B directly, for various reasons, e.g., because you don’t want them too tightly coupled or because they’re in different security domains. To resolve this you insert C, an intermediary (sometimes called the Mediator pattern). But later a programmer comes along and looks at the code and thinks, "Well it looks to me like A can just call B," and changes the code, without understanding the design intent. The programmer can’t see the grand plan from looking at the minutiae. In this way, even good designs typically are undermined—they erode and degrade over time. For some system qualities, such as modifiability, this is bothersome. For security, it is potentially fatal. I along with my colleagues—Jungwoo Ryoo of Pennsylvania State University and Amnon Eden of the University of Essex, U.K—developed an approach to address these challenges. Our approach comprises two major components: the design guide and the Two-Tier Programming Toolkit (TTP), described below. In the design guide, we first provide natural language descriptions of approaches for achieving design intent. If your purpose is to achieve security, for example, your design intent might involve detecting, resisting, or recovering from attacks on your system. To help you realize your intent, the guide presents a hierarchy of tactics for building in security. Tactics form the building blocks of patterns. To continue a previous analogy, if a pattern is a roof truss, a tactic is a component of that truss: a chord, web, or plate. Tactics involve a more fine-grained level of design than do patterns.   Hierarchy of Security Tactics Presented in the Design Guide In the figure above, you’ll notice that Limit Exposure is categorized as a tactic for resisting attack. The Limit Exposure tactic minimizes the "attack surface" of a system; it typically is realized by having the least possible number of access points for resources, data, or services and by reducing the number of connectors that may provide unanticipated exposure. This tactic focuses on reducing the probability of and minimizing the effects of damage caused by a hostile action. The Limit Exposure tactic is associated with patterns that build in security, such as the Check Point pattern. This pattern monitors incoming messages, to determine whether they are consistent with a given security policy, and triggers countermeasures if necessary. After you have chosen the pattern and its associated tactics for achieving your design intent, these must be converted into code. As mentioned above, whenever you undertake such a translation step, you have the possibility of producing an inaccurate translation. To solve this challenge, our approach requires that you express your chosen pattern by creating (or, even better, reusing) code charts. Code charts are formal specifications that model and visually depict a program’s structure and organization.  The figure below shows the Check Point pattern as expressed through a code chart. The Check Point Pattern as Represented by a Code Chart You create the code charts through the Two-Tier Programming Toolkit, the second element of our approach. Two-tier programming integrates the representations of both design and implementation. Frequently, when a system undergoes redesign, the changes between the design and the code that implements it may not be coordinated, causing the design to deteriorate. To maintain design quality, programmers must keep these two tiers coordinated through the software’s lifecycle. Our TTP toolkit is a round-trip engineering tool; it maintains design and implementation as separate representations, while facilitating the propagation of changes between them.  It supports round-trip engineering in that it has facilities for forward engineering (planning a new design and ensuring that this design is faithfully implemented by the code) and reverse engineering (determining what structures already exist in the code, and modeling those). The TTP Toolkit was created by a team of colleagues at the University of Essex whom I joined in developing this tool. The program visualizations are reverse engineered from Java source code, and visual specification of object-oriented design is represented in a visual language (based on the formal specification language, LePUS3). When you have created the code charts via the TTP toolkit, you will need to associate each pattern with the source code that implements it. To do this you simply need to associate (or bind) each variable in the pattern to a specific programming language construct. For example, the Check Point pattern makes use of a security policy, represented as "Policy" in the code chart.  The Check Point monitors if messages are consistent with the Policy; if they are not, a counter-measure, represented as "Countermeasure" in the code chart, may be triggered. Variables and Constants of the Check Point Pattern Finally, the TTP Toolkit can check that your design is implemented correctly. The toolkit automates the verification process so that you can easily determine whether the code conforms to a tactic or pattern specification. The tool lets you recheck the design at the click of a button any time you make changes to the code, to assure conformance as the system evolves. This verification mitigates the problem of programmers’ unwittingly undermining patterns during maintenance. Successful Verification through the TTP Toolkit Round-trip engineering tools usually support reverse engineering by generating visualizations from analyzed source code. If the output of the reverse engineering tool (in this case, code charts) can feed directly into the forward engineering tool, and vice versa, then the engineering cycle can be said to be closed—hence round-trip engineering. Closing the engineering cycle can ensure that anytime a gap between design and implementation is generated, it can quickly be detected and remedied.In summary, our approach involves these steps for achieving system security: Identify your security requirements to determine your design intent. Using the design guide, find security tactics to compose security patterns, creating a library. Through the TTP toolkit, convert this library of security patterns to code charts. Use the TTP toolkit to automatically ensure conformance of implementations to design specifications. Wrapping Up and Looking Ahead The design guide provides a natural language description of the architectural approach, guiding the architect from a statement of design intents to one or more tactics, which in turn lead to one or more patterns. We have already specified a large number of security patterns in the guide and the toolkit, and are adding more.  While many patterns catalogs have emerged over the past two decades, what sets this work apart is that the toolkit adds rigor to the specification of a pattern and adds conformance checking to ensure that the design and implementation always stay synchronized.  This synchrony is essential. If developers don’t maintain this coordination, then the software’s design will certainly degrade over time, a condition referred to as "software entropy." Every designer and project manager should be aware of this risk and concerned about reducing it. It’s useful, by the way, to note that the TTP toolkit has limitations and cannot check every property. For example, it can’t tell you that A is going to happen after B, or is going to happen immediately after B, because those are temporal specifications, involving timing and ordering. The toolkit can only confirm structural properties, such as A calls B or A inherits from B. In other words, it can check static properties and relationships as opposed to dynamic properties and relationships. We hope that our approach will enable practitioners to improve their design process and the software it produces. As with the introduction of any new technology, we expect to encounter some resistance to learning a new tool and a new formalism and concern about adoption risks. We also want practitioners to realize that this technology won’t solve all of their design problems. It will solve some important ones, however. If successful, the design guide and TTP toolkit will make it easier to maintain conformance and reduce the cost of doing so. Adopting our approach should result in higher quality design and implementation, in turn resulting in more secure implementations of designs.  Please let us know your thoughts about the utility of this techniques in the comments section below. Additional Resources To read the book Software Architecture in Practice, which I co-authored with Len Bass and Paul Clements, please visit http://resources.sei.cmu.edu/library/asset-view.cfm?assetid=30264 To read the paper Using Security Patterns to Model and Analyze Security Requirements by Sascha Konrad, Betty H.C.Cheng, Laura A. Campbell, and Ronald Wassermann, please visithttp://www.cse.msu.edu/~cse870/Materials/rhas-03-CRC.pdf To read the paper  A Pattern Language for Security Models by Eduardo B. Fernandez and Rouyi Pan, please visitwww-engr.sjsu.edu/fayad/current.courses/cmpe133-fall04/docs/PLoP2001_ebfernandezandrpan0_1.pdf To read the paper In Search of Architectural Patters for Software Security by Jungwoo Ryoo, Phil Laplante, and Rick Kazman please visit http://dl.acm.org/citation.cfm?id=1591991To read the paper Security Engineering with Patterns by Markus Schumacher and Utz Roedig, please visit http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.4.1662M. To read the paper, Architectural Patterns for Enabling Application Security by Joseph Yoder and Jeffrey Baraclo, please visit  http://www.idi.ntnu.no/emner/tdt4237/2007/yoder.pdf

By Will Casey Senior Member of the Technical StaffCERT Division Code clones are implementation patterns transferred from program to program via copy mechanisms including cut-and-paste, copy-and-paste, and code-reuse.  As a software engineering practice there has been significant debate about the value of code cloning. In its most basic form, code cloning may involve a codelet (snippets of code) that undergoes various forms of evolution, such as slight modification in response to problems.  Software reuse quickens the production cycle for augmented functions and data structures. So, if a programmer copies a codelet from one file into another with slight augmentations, a new clone has been created stemming from a founder codelet.  Events like these constitute the provenance or historical record of all events affecting a codelet object. This blog posting describes exploratory research that aims to understand the evolution of source and machine code and, eventually, create a model that can recover relationships between codes, files, or executable formats where the provenance is not known. Other major events for codelets may include creation, copy, modification, and deletion of a codelet from a project.  The presence of software bugs and vulnerabilities brings a new set of questions for which the provenance of codelet is particularly relevant.  Consider the copy/paste of a codelet that contains a bug. By cloning the programmer has essentially introduced a new bug by copying an old bug. The tracing of provenance in software developments has therefore become increasingly important to tracking both bugs and vulnerabilities.  A generative model of provenance developed from observed software histories may inform researchers and developers how code evolves and mutates over time.  Taking that notion one step further, a generative model that uses  Bayesian Statistical techniques provides inference possibilities, such as identifying relations in software where the provenance is not completely known. Examples of problems where provenance models and inference may be critical include identifying attribution for malware and analyzing potential vulnerabilities or vulnerability surface in third-party software product chains. These are two increasingly important problems in today’s era of cyber warfare where we aim to both minimize vulnerability exposure and understand adversarial attack patterns.  Further provenance inference may be an important step in understanding the dynamics of cyber conflict:  If an organization deploys sophisticated cyber attacks can counter attacks (or more generally other attacks) occur by simply copying the malicious code and re-deploying it?  Codelet mobility (the ease at which an implementation can be copied and redeployed) is an important factor to understanding the strategic options of agents in cyber conflicts. Initial Research This work began about 18 months ago when CERT researchers began using automated techniques to identify relationships (i.e., common authorship) among different types of malware. In the not-too-distant past, identifying the provenance or origin of malware required reverse engineering, which was labor intensive and time consuming. I worked on that project with Aaron Shelmire, a former CERT researcher, to develop a Jaccard similarity index, which measures the amount of commonality between any two digital artifacts, source code files, or binary executables, based on the number of code clones in common. The Jaccard index identifies all common substrings in the artifacts and presents a measure of how much code is code in common. Not only is this technique a more efficient approach than reverse engineering, we have arrived at techniques that are algorithmically optimal, and shown that these methods may be used to prioritize the efforts of reverse engineers. We used the Jaccard index to systematically examine subsets of artifacts to identify malware families, which share a high degree of code similarity in binary images or epoch similarity in traces (runtime). The basic notions of similarity are sufficiently long interesting strings or clusters of shorter strings.  Because there are many generative factors giving rise to code clones (e.g. compiler artifacts, linked static libraries, copy-cut-and-past) the question arose, How can we interpret these measures in to infer shared provenance and how may we do so with statistical confidence? From a malware attribution perspective this question reduces to How can we demonstrate that a high degree of similarity between two malware families is due to a common attacker or common source code? How could an analyst be certain that the similarities were not the result of shared, online, open-source code? This simple question sparked our research question: How should we measure provenance in software? This exploratory research involves several phases: Phase 1 involves leveraging a large-scale software history to gain a better understanding of how code evolves. Phase 2 involves creating a generative model that will allow us to investigate the statistical problem of inference.  Phase 3 involves the application of the statistical model to problems of inference This research provides malware analysts greater certainty in determining shared authorship and other cloned features among two or more file A Foundation in Phylogenetics Our interest in inference models for software provenance are preceded by effective inference models in the biological problems of phylogenetics, which allowed biologists to infer the structure of evolution events and determine common ancestry by examination of contemporary biological data.  In fact, if you have received a flu vaccine then you have benefited from this science inference and prediction. We recognized similarities to phylogenetics in our own work: We were researching malware artifacts in hopes of understanding more about their historical developments from the artifacts alone (with the provenance data withheld). To address this completely, we would need to understand and model the events that modify and edit source code. To accomplish this, we needed to look at source code with a known provenance. Turning to Open-Source Code It is an opportune time to explore the history of software for any researcher interested in examining the evolution of source code. Unix maintains a 43-year history detailing the evolution of its code while Linux maintains 21-year history. We decided to examine Mozilla’s 14 year open-source history. The previous five years contain many significant developments indicated by the following statistic: 2,546 users modifying 25,000 files containing a total of 122 million lines of source code modified.  Of the 122 million lines of code (LOC), only 4.2 million LOC are unique. This statistic suggests that a vast majority of lines are redundant, many of which are copied or cloned.  If foundational events may be understood and subsequent mutations identified within a generative provenance model, it is possible that among the 4.2 LOC, there may be still even greater reductions that arise from understanding the mutation structure and co-migration patterns of source code.  While an entire list of driving factors leading to code mutation and migration may not be practical, major modification patterns can be inferred and dynamic modification patterns can be learned from data with know provenance. Addressing Size and Scale One of the major challenges in our research involves addressing the size and scale of the Mozilla source-code distribution, which in the last five years has logged 120,355 major commit events. One method to address this size and scale is through indexing techniques. For example, in January 2013 Mozilla distributed 17 million LOC (only 20 percent of which are unique) and distributed in 15 thousand files. Indexing these individual lines of code as well as their neighboring lines of code have has allowed us to identify portions of code that have migrated from one file to another during development events. While there are large numbers of file objects and LOC objects, the relation between file objects and LOC objects is very sparse. Indexing also allowed us to map relations such as co-occurrence clusters of LOC and their containing files. Using these observations we may observe statistical trends of modifications made to files over time and test our assumptions of sparse structures and evolution patterns such as the following scenario: files exhibit rapid growth (initially) with lots of code added in, and then settle down eventually with bug fixes generating the only modifications there after. By creating an index of unique lines of code, we were able to map codelets as code patches with co-migratory patterns in files. A challenging aspect of modeling provenance problems for software has been the notion of object granularity [Bunge and Ram, Liu] or determining what objects should be modeled.  Because source code objects are so closely tied with their application or function, both the file (as object) or even the line of code (as object) present drawbacks. We theorize that the most useful object to consider are functional in nature, such as a well-formed function definition in the C programming language, which may also be considered a codelet (or a set of lines-of-code). Toward testing this viewpoint our indexing techniques allow us to observe co-migration of codelets or sets of lines forming well defined C language functions. Another challenge in our research involved interpreting the summary of actions provided by developers. One way to interpret this summary is to create topic models that focus on key terms, such as bug, merge, adding code, etc.   By creating topic vectors, we enhanced the inference of why certain modifications were made. Another obstacle that we needed to address in our research involved accounting for the many mechanisms that affect the copy number of lines of code. Addressing this challenge involved understanding how to model deletion events in a distributed revision control system, these events unlike their analogous events affecting the provenance of physical objects, are harder to deal with theoretically as remote branches when merged may re-introduce lines of code that were previously deleted.  Collaboration In examining Mozilla’s source code, we have collaborated with Sudha Ram, a researcher at the University of Arizona who is an expert in data provenance. Her research along with Jun Liu has examined provenance in Wikipedia pages, iPlant Collaborative, and in the Materials Management domain. Their work on digital provenance in edit histories of Wikipedia pages has helped identify collaboration patterns that result in high quality pages. Ram and Liu have developed a model of semantics for data provenance known as W7, a technique that describes the who, what, when, where, why, which, and how, for data events. We are also working with Rhiannon Weaver and Katherine Prevost, fellow CERT researchers, who are helping us design statistical measures of ownership and topic models for summaries associated to modification events. Future Work  With our indexing approach, we have been able to establish measures of commonality between malware files. We are aiming to use these measures to infer the likelihood for several different types of code clone events: subtraction - modifications that remove lines of code. addition - modifications that introduce additional lines of code to enhance a function. mutation - modifications of code by slight in situ modifications to fix/enhance function.  moving/copying - reorganization or re-introduction of codelets in files. Our real goal, however, is to understand the mobility of code and how it mutates and flows from one source to another. For practical problems attaining this understanding will allow us to assess the likelihood of why certain codes are found in common.  For malware, if we can confirm that 20 percent of a malware group is related to a previously identified malware group, can we conclude with confidence that what other factors would lead to such a result? We plan to continue our collaboration with Sudha Ram with the intent of producing a provenance model for software code. Specifically, this model will describe the evolution mechanism for code clones. After training and evaluating the model, we will use statistical analysis to identify clone evolution patterns where inference may be applied to determine provenance among two different pieces of malware. Specific evolution patterns that admit model validation can then be applied to operational problems of malware attribution. In this next phase of our research, we are also collaborating with two researchers in the SEI’s Emerging Technology Center, Naomi Anderson, and Kate Ambrose-Sereno, who are examining data provenance models for software. We welcome your feedback on our research. Please leave a comment in the feedback section below. Additional Resources To read the book Treatise on basic philosophy: ontology I. The furniture of the world by Mario Bunge, please visit http://www.springer.com/philosophy/epistemology+and+philosophy+of+science/book/978-90-277-0785-7 To read the article, Who does what: Collaboration patterns in the wikipedia and their impact on article quality by Jun Liu and Sudha Ram, please visithttp://dl.acm.org/citation.cfm?doid=1985347.1985352 To read the article, A Semantic Foundation for Provenance Management by Jun Liu and Sudha Ram, please visithttp://link.springer.com/article/10.1007%2Fs13740-012-0002-0#page-1

By Lisa Brownsword,Senior Members of the Technical Staff Although software is increasingly important to the success of government programs, there is often little consideration given to its impact on early key program decisions. The Carnegie Mellon University Software Engineering Institute (SEI) is conducting a multi-phase research initiative aimed at answering the question: is the probability of a program’s success improved through deliberately producing a program acquisition strategy and software architecture that are mutually constrained and aligned? Moreover, can we develop a method that helps government program offices produce such alignment? This blog post, the third in a series on this multi-year research, describes our approach to determining how acquisition quality attributes can be expressed and used to facilitate alignment among the software architecture and acquisition strategy. In the first post in this series, we identified specific instances where misalignment between software architecture and acquisition strategy resulted in program delays and cost overruns—and, in some cases, program cancelation. In the second post, we outlined seven patterns of failing behavior, or "anti-patterns," that were major contributors to this misalignment. In addition, we presented a model that links key program entities and their relationships that could avoid those anti-patterns. Finally, we observed that alignment among software architecture and acquisition strategy does not occur naturally. Of particular note in this early research, we postulated that acquisition quality attributes reflecting the program’s business goals can be used to judge the effectiveness of an acquisition strategy—analogous to software quality attributes reflecting the mission goals that are used to judge the effectiveness of a software architecture. In this latest effort, a team of researchers that, in addition to myself includes Cecilia Albert, Patrick Place, and David Carney, have focused on determining how to express, elicit, and analyze acquisition quality attributes for their utility in surfacing potential incompatibilities between a software architecture and an acquisition strategy. Our Research Approach We patterned our approach to defining acquisition quality attributes on the original SEI research on software quality attributes. We began by forming a starting starter set of potential acquisition quality attributes. We compiled an unordered list of roughly 30 qualities derived from a review of government acquisition strategy guidance and discussion with acquisition professionals, colleagues, and several brainstorming sessions within our team. Examples of these initial acquisition quality attributes are criteria by which acquisition strategies are judged affordability achievability effectiveness flexibility Next, we defined an approach for expressing program-specific acquisition quality attribute scenarios using the SEI’s earlier work in software architecture where stakeholders are encouraged to create small "stories" that specify some event (the stimulus) that occurs in a particular part of the lifecycle (the environment) and a desired behavior (the response). For example, in software architecture, a quality attribute might be expressed using the following three-part scenario: stimulus - an internal component fails environment - during normal operation response - the system recognizes an internal component failure and has strategies to compensate for the fault In the acquisition domain, a similar example might be as follows: stimulus - an unexpected budget cut environment - for a multi-segment system response - the program is able to move work between major segments to speed up or slow down separate segments within the available funding Next, we elicited acquisition quality attribute scenarios from former program management office personnel and members of Independent Technical Assessment teams (informally called Red Teams) to capture 55 acquisition quality attribute scenarios covering 23 government programs. We needed these scenarios to validate that qualities related to an acquisition strategy can be expressed in a meaningful way through these scenarios. In addition, we were able to show that a tight link exists between the acquisition quality attribute scenario and some element of the of the acquisition strategy. These scenarios are detailed in a technical report, Results in Relating Quality Attributes to Acquisition Strategies. The following is an example of the scenarios we constructed: stimulus - a new need arises when we want to react quickly environment - where there are only a limited number of contractors able to do the work response - the program can satisfy the need by adding the work to an existing contract potential acquisition tactic - Award IDIQ contracts to multiple (perhaps eight or so) vendors and issue task orders in a round-robin fashion We built and validated a prototype workshop adapted from the software Quality Attribute Workshop (QAW). The major changes we made to a conventional QAW were to place more emphasis on business presentation and replace the architecture presentation with one on the program’s acquisition strategy plans, thus we termed our revised approach the Acquisition Quality Attribute Workshop (AQAW). We then prototyped our AQAW using a real program but substituting SEI staff for various program stakeholders. The prototype AQAW successfully generated twenty acquisition quality attribute scenarios. While only a single instance, the prototype successfully demonstrated that an AQAW is a plausible approach for capturing acquisition quality attribute scenarios. Finally, we analyzed all captured acquisition quality attribute scenarios for trends, implications, and potential incompatibilities. Some of our findings are described below.Different scenarios result in different acquisition strategies. For an acquisition quality attribute scenario to influence the acquisition strategy, there must be some element of the scenario that leads the program office to choose a strategy. For instance, we found a number of acquisition quality attribute scenarios relating to new technology and the issues that arise if the chosen innovative technology fails to deliver on its promises: A new technology the program office expects to use is found to be unsuitable where schedule is of prime importance; the program office switches to an alternative that is also currently under development and is evaluated to be suitable. A new technology the program office expects to use is found to be unsuitable where costs must be kept as low as possible; the program office instructs the contractor to restart but using an alternative technology. In the case of these scenarios, the stimulus is the same but the environment changes. In the first scenario, schedule is more important than cost. The second scenario reverses their relative importance. In the first scenario, an acquisition strategy starting multiple developments simultaneously with a requirement for some kind of decision between the alternatives would be appropriate. In the second scenario, a strategy starting a single development contract and continuing to use it while switching to a more feasible technology might be more appropriate. Although this is a simple example, it demonstrates that different acquisition quality attribute scenarios can lead to different acquisition strategies. This finding strengthens our contention that our use of acquisition quality attributes and acquisition quality attribute scenarios is analogous to the use of software quality attribute and software quality attribute scenarios and that we may continue to rely on methods and mechanisms developed for that purpose to assist with the creation of sound acquisition strategies. Incompatible Acquisition Scenarios. Conflicts between scenarios are not always obvious, and may be quite subtle without some analysis.  For example, organization ABC is using a large, complex legacy system deployed in multiple operational locations, where each location installed its own local variant of the system. Over time, these variants have diverged in response to differing requirements of the local users. To accommodate mission changes, it has become important to share data across these locations in a more integrated way. A new program was initiated to acquire one replacement capability that would support all of the differing needs across the multiple fielded locations. The program decided to implement an incremental approach to replacing the legacy system so they could respond to budgetary constraints and uncertainties. For the example described above, the following scenario reflects the expectation of one set of influential stakeholders who advocated the use of a commercial off-the-shelf (COTS) product that had been successfully used at one of the operational installations: stimulus - There is a desire to replace a complex component of a large legacy system with a COTS package environment - Within an established enterprise architecture with many local variations implemented that are largely different from each other response - The program runs a competition to evaluate COTS packages for an enterprise-wide solution. A second set of stakeholders, reflecting operational users, is counting on the new system to quickly address their current needs. These needs vary among the current fielded locations. During the time it takes the program to define an agreed upon set of requirements for each increment, the user representatives from the various fielded location change their requirements. This leads to a second acquisition scenario: stimulus - Requirements for the next release keep changing environment - For a program with a fixed budget that must be carefully managed response - The program accepts the new requirements Implied in these two scenarios is a third set of stakeholders: the enterprise system engineers, who are advocating the implementation of an enterprise architecture that extends across all of the local fielded implementations. This enterprise architecture could be incompatible with both of the above scenarios: each COTS product, by definition, is built to an architecture and a set of requirements that organization ABC has no control over. Moreover, the demands for local fielded implementations compete with architectural changes within a constrained budget. Unfortunately, the two scenarios are potentially incompatible with respect to designing the acquisition strategy. The first scenario describing the implementation of a common COTS product across all locations could provide sizeable value in terms of moving to one capability that is used across all fielded locations, but it may not meet what the current users described in the second scenario consider urgent needs—and both of these may conflict with the move to an enterprise architecture. Looking Ahead We were pleased to demonstrate a critical link between acquisition quality attributes and an acquisition strategy. We also identified potential recommendations for how an acquisition quality attributes could be expressed in program specific three-part scenarios. In the next phase of this work—already under way—we are working to develop an alignment method that collects and analyzes acquisition and software quality attributes in a way that enables deconfliction and prioritization. We believe that acquisition strategies and software architectures built from this consistent set of quality attributes would be aligned and mutually constraining.   Additional Resources To read the SEI technical note, Results in Relating Quality Attributes to Acquisition Strategies, please visit,http://resources.sei.cmu.edu/library/asset-view.cfm?AssetID=78312 To read the SEI technical report, Quality Attribute Workshops, please visithttp://www.sei.cmu.edu/library/abstracts/reports/03tr016.cfm To read the SEI technical report, Isolating Patterns of Failure in Department of Defense Acquisition, please visit http://resources.sei.cmu.edu/library/asset-view.cfm?assetid=53252

By Kate Ambrose SerenoTechnical AnalystSEI Emerging Technology CenterThis post was co-authored by Naomi AndersonIn 2012, the White House released its federal digital strategy. What’s noteworthy about this release is that the executive office distributed the strategy using Bootstrap, an open source software (OSS) tool developed by Twitter and made freely available to the public via the code hosting site GitHub. This is not the only evidence that we have seen of increased government interest in OSS adoption. Indeed, the 2013 report The Future of Open Source Software revealed that 34 percent of its respondents were government entities using OSS products. The Carnegie Mellon University Software Engineering Institute (SEI) has seen increased interest and adoption of OSS products across the federal government, including the Department of Defense (DoD), the intelligence community (IC), and the Department of Homeland Security. The catalyst for this increase has been innovators in government seeking creative solutions to rapidly field urgently needed technologies. While the rise of OSS adoption signals a new approach for government acquirers, it is not without risks that that must be acknowledged and addressed, particularly given current certification and accreditation (C&A) techniques. This blog post will discuss research aimed at developing adoptable, evidence-based, data-driven approaches to evaluating (open source) software. In this research, members of the technical staff in the SEI’s Emerging Technology Center (ETC) explored the availability of data associated with OSS projects and developed semi-automated mechanisms to extract the values of pre-defined attributes.  The challenges of applying data analytics to address real problems and of understanding OSS assurance align with the ETC’s mission, which is to promote government awareness and knowledge of emerging technologies and their application, as well as to shape and leverage academic and industrial research. Our research leveraged the "openness" of OSS to develop an evidence-based approach for assessing and assuring OSS. This approach, which focused on producing evidence in support of assurance claims, is based on generating artifacts and creating traceability links from assurance claims to those artifacts. Beyond a Trust-Based Approach If we think of traditional, "shrink-wrapped" software, we accept that the software is developed by and purchased from a vendor who delivers a product against specified requirements.  The software comes with installation instructions, FAQs, and access to support via hotlines and websites. Generally speaking, there is a company or some kind of legal entity that stands behind the product. With OSS development, however, multiple developers from different organizations (even independent developers) can contribute to the code base of a product, which may or may not be backed by a single legal entity. In some cases, developers include helpful information in the software repository; in other cases, users are on their own to get the software working in their environment. Specific functionality may be driven by the community of developers, or by a small core team. Current methods to assess software (OSS or otherwise) are trust-based and rely heavily on expert opinion.  For example, users may run experiments with the software in a controlled environment to determine whether or not it is safe to operate. When certifying and accrediting OSS or any software, however, the trust-based model is not valid for several reasons: In today’s environment, many organizations and entities incorporate some aspect of open-source into their software. As a result, no single company or organization represents an OSS capability. Individual expert assessments are manual and do not scale to the level required for large-scale, mission-critical projects that apply OSS. Assurance claims are based on opinion rather than on a data-driven designation of assurance. For these reasons, we wanted to develop a prototype of a tool that reaches beyond the functions of traditional static analysis tools. Our aim is to create a tool that government or industry could use to support their decision of whether to adopt an OSS package. We felt it was important to develop a tool to provide supporting evidence, rather than one that would provide a determination of whether a particular software package is "good" or "bad." In this age of sequestration and other pressures on the expense of acquiring and sustaining software-reliant systems, government agencies can realize numerous benefits from a good OSS development and adoption strategy, including cost savings and increased flexibility in the acquisition and development of systems. Foundations of Our Approach In 1998, after Netscape published the source code for Netscape Communicator, Bruce Perens and Eric S. Raymond, founded the Open Source Initiative, an organization dedicated to promoting OSS. Since that time, a large number of OSS repositories have surfaced including Github, Launchpad, Sourceforge, and Ohloh. In developing an approach, our team of researchers and software developers at the SEI wanted to create a tool that leveraged features of OSS, including the openness of code, development environment, documentation, and user community. Our aim was to design and develop an integrated, semi-automated software assessment capability that would allow an assessor to explore the evidence supporting an assurance claim. The upside of the renewed interest in OSS adoption, both in government and industry, is that a wealth of data now exists within these repositories that provide insight into development of OSS as well as the code review and code committal process. Our aim with this research was to move beyond simple bug counts and static analysis and provide richer context for those charged with assessing software systems. While no one measure or metric could provide an accurate assessment of software, we reasoned that several characteristics could provide acquirers with a more complete view of OSS assurance. During our study, we identified measurable characteristics that could be of interest, particularly if assessed in combination.  For example, we examined complexities of the coding language used, test completion, and vitality or inertia of the project.  Other characteristics that we evaluated included milestones. Our analysis included proposed release dates versus actual release dates. Meeting clearly described milestones and schedules is often an indicator of sound project management. bugs. We examined issues such as severity, discovery date versus fixed date, timing for a fix to be included in a release, percentage of bugs carried from previous release, distribution of bugs by severity, time-to-fix measures, rate at which new bugs are identified, bugs tracked in current release as well as bug aging and classification. Bug counts and defect density alone are not sufficient. If we look at sluggish time-to-fix measures, however, that may signal problems with the current release. documentation. We looked at whether there was a process to update documentation, whether the documentation was up to date or not, release notes, change log, how lines of code versus the length and completeness of the user manual correlate.  A lack of documentation is a risk to adopters and implementers of the software because the implementers are left to their own devices to get the software working in their environment, which can cause significant delays in rollout.  user base growth over time. We looked at activity levels in mailing lists (users and developers). We also considered activity levels at conferences, market penetration, and third-party support. We reasoned that evidence of increasing or decreasing activity from the user community was evidence of the strength of the product. developer involvement over time. Our evaluation spanned number of commits (making suggested changes from the user community), number of unique authors, lines of contributed code versus total lines of code, evidence of hobbyist developers, and a network diagram illustrating connections and influence of the community of developers, code committers and reviewers. We reviewed the social network of the developer community that supported particular OSS projects. Context is important. Using the data collected to help build an understanding of the development environment, developer activity, and user community commitment helps potential adopters get a better sense of the viability of the OSS project. Challenges When we first began this research, we focused on identifying data that would allow us to make valid comparisons between identifiers of quality in different software repositories. We soon realized, however, that quality attributes really are context dependent. For example, OSS acquirers may place various levels of importance on whether software is updated during daytime hours by full-time employees or during evening hours by hobbyists. Instead of placing a value judgment on these variables, we altered our approach to identify characteristics such as the ones listed above that can be used by decision makers to determine relevancy and weighting. As we progressed through the research, we also realized that OSS repositories were starting to explore ways to represent data relevant to the OSS projects in the repositories.  For example, Github maintains a graphs section that highlights data, such as code stability, trends over time, and a separate punch card section that represents the volume of code commits over the span of a week. Another example involves Ohloh, which provides a side-by-side comparison along different parameters about the OSS projects. Another challenge that we encountered surfaced after we began exploring the OSS repositories. We found that while there are many typical developer tools being used, they were all being used differently across different software projects. One example of this involved JIRA, a bug tracking software that offers users configurable fields. Another example can be found in the Apache Software Foundation project Derby, some bugs have fields for urgency, environment, issues, fix information, or bug behavior facts while others do not.  Looking Ahead All indicators point to increased adoption of OSS. In November 2013, Federal Computer Week published an article detailing the adoption of OSS across the DoD. An article on OSS and government in Information Week earlier that month stated that "Federal agencies, looking for new ways to lower their IT costs, are exploiting open-source software tools in a wider range of applications, not only to reduce software costs, but also to tighten network security, streamline operations, and reduce expenses in vetting applications and services." In the coming year, we will continue our work in data analytics and OSS assurance. We are interested in collaborating with organizations to expand selected data to analyze and find correlations among seemingly disparate dimensions and measures in software development produce evidence for specific OSS projects that are critical to mission needs test-specific assurance claims using data-analytics approach and build-in bi-directional traceability between claim and evidence build tools to accommodate large scale analysis and evidence production (multiple OSS projects along multiple dimensions) experiment with evidence production targeting tools develop and publish a comprehensive open source assurance classification system If you are interested in collaborating with us, please leave a comment below or send an email to info@sei.cmu.edu. Additional Resources For more information about the SEI Emerging Technology Center, please visithttp://www.sei.cmu.edu/about/organization/etc/ To read the article Has Open Source Officially Taken Off at DoD? by Amber Corrin, please visithttp://fcw.com/Articles/2013/11/19/DOD-open-source.aspx?Page=1 To read the article Agencies Widen Open-Source Use by Henry Kenyon, please visithttp://www.informationweek.com/agencies-widen-open-source-use--/d/d-id/899851 To read the article Army C4ISR portal uses open-source software for faster upgrades by William Welsh, please visit http://defensesystems.com/articles/2013/01/30/army-c4isr-portal-open-source-software.aspx

By Anne Connell Design Team Lead CERT Cyber Security Solutions Directorate  This blog post was co-authored by Tim Palko.  According to a report issued by the Government Accountability Office (GAO) in February 2013, the number of cybersecurity incidents reported that could impact "federal and military operations; critical infrastructure; and the confidentiality, integrity, and availability of sensitive government, private sector, and personal information" has increased by 782 percent—from 5,503 in 2006 to 48,562 in 2012. In that report, GAO also stated that while there has been incremental progress in coordinating the federal response to cyber incidents, "challenges remain in sharing information among federal agencies and key private sector entities, including critical infrastructure owners." Progress in this area was hindered by "difficulties in sharing and accessing classified information and the lack of a centralized information-sharing system," the report stated. This blog post describes a tool that members of the CERT Cyber Security Solutions (CS2) Directorate are developing to provide the various agencies and organizations that respond to cyber incidents a platform by which to share information and forge collaborations.   I have witnessed these challenges to effective collaboration first-hand. In my role, I am often called upon to observe subject matter experts who advise incident responders while they manage cyber incidents to assist in collecting evidence and presenting it to authorities working criminal cases. In this role, I have repeatedly observed incident responders, including law enforcement and subject matter experts, operating in disconnected siloes. Representatives from these agencies literally set up separate work stations.  While attackers are organized and well-coordinated in their efforts, agencies and organizations that respond to these cyber incidents operate in disconnected siloes that are in need of a shared platform for trusted collaboration. The aim of our work is to create this platform. We intend the platform to be used across a range of groups, such as computer security incident response teams (CSIRTs), incident responders, commercial companies, and law enforcement.  Our Approach: Cerebro At its core, Cerebro is a prototype that allows collaborators to identify and tag actionable information. The information is collected at a level of granularity that allows collaborators to specify the extent to which they want to share this information: at a group level, an organizational level, or with all participants of an organization.  As outlined in the IEEE paper Cerebro: A Platform for Collaborative Incident Response and Investigation, which I co-authored along with Tim Palko, our approach incorporates a six-phase model that represents the process an incident responder/assessor goes through when responding to a cyber incident:  Site assessment. The primary goal of this phase is to develop a response plan to handle the incident based on its assessment and prior experience. Activities in this phase include assembling and training the team and creating situational awareness of the systems that comprise the site or event.  Site aggregation. Activities in this phase include conducting a network assessment of a site or event and minimizing the scope and impact of the attack. The site assessor also begins site categorization and collection and uses forensic software or toolkits to obtain and extract evidence.  Site analysis. Activities in this phase include identifying lessons learned from the handling of the assessment. A meta-profile containing the reduced critical assessment of the site is imported to the data store to generate rules or steps for creating better preparedness, which may include modifying policy or process or making changes to configurations. Collaborative investigation/correlation. Activities in this phase involve the establishment of a trusted environment that enables communication about the site, which is essential when preparing for a collaborative response. This environment must provide analysts a space to make observations and correlate disparate data types (site security, network analysis, etc.). Applied machine learning is also performed on the dataset. The machine learning produces the predictive analysis that is pushed to the analyst. Policy/rule application. Activities in this phase focus on data analysis, a crucial part of the investigation process. The availability of data from multiple sites and/or events opens the possibility of cross-site analysis to establish links among events occurring at individual sites. During investigation and correlation, the collaborative mechanism runs and shares watch-lists, security events, and rule sets. Reports from individual sites and/or events are collected and sent to Cerebro. After a detailed analysis—which might involve two analysts who have the same observation or the system's autonomously identifying links happening at multiple events—Cerebro generates push notifications to alert the user of the associations. Site incident strategy. Activities in this phase occur once the incident investigation has concluded. The site administrator and site responder (both humans) take appropriate steps to mitigate any risks or bring compromised systems back online. Policy and rules developed automatically in the analysis stage are presented as a critical stage to disseminate information, but ultimately any action taken based on these notifications and rules is taken by a person.  Cerebro takes a practical approach to defining a system model that collects and analyzes data in a trusted cloud-computing platform, which allows us to store large volumes of data while simultaneously processing them to find, store, and categorize evidence of malicious attacks. We will host our tool on an extensible large-scale analysis platform for managing and analyzing data (such as logs and communications). The analysis platform provides better management and a better security mode, and is equipped with a suite of open-source tools for log and data extraction, data and evidence storage, data and log analysis, and forensics.  In a cyber attack, organizations may encounter an adversary who targets communication between administrators to disrupt the effectiveness of their response. In designing a framework to foster collaboration in the wake of a large-scale cyber attack, we envisioned role-based access control that draws upon the principle of least privilege. Cerebro comprises two main components:  a Roles and Responsibilities Model that defines the entities involved in the response and investigation, their responsibilities, and their interactions  a Process Model that defines the phases of the response and investigation process, as well as the execution of responsibilities in these phases Together, these components ensure that the response and investigation team members are able to effectively manage the required tasks. In particular, the system model integrates the technical incident response and the legal investigation and prosecution process in a multi-site collaborative manner.  Our approach builds on several well-known principles for effective collaboration. For trust establishment, we rely on an incentive-based approach in which organizations learn more about vital watch-list information and obtain access to tools and resources to respond to and recover from attacks.  Cerebro also relies on an approach involving organizational access policy; organizations providing value in the identification and response process can collectively define important pieces (IP addresses, type of attack, pattern identification) of an investigation.  For managing tasks and processes, we focus on identifying and indexing areas of interest that warrant collaboration to integrate them into an well-defined process workflow for each organization.  Challenges of Our Approach  One challenge of our approach with Cerebro involves addressing security issues so that potential users can be assured that their information will circulate within an intended audience. To address this issue, we designed a system that uses two-factor authentication: role-based and signature-based.  Ideally, our tool fosters 100 percent participation by all involved. Cerebro observes the 90-9-1 rule. This is the basic observation that in a collaborative platform, such as a wiki 90 percent of participants will "lurk" and simply observe information being posted 9 percent of participants will actively edit and produce the information being created 1 percent of participants will be involved in content validation, administration, and rule generation Our approach hypothesizes that if only 9 percent of participants are involved in information analysis, that group can act on the information and ideally retain enough so it doesn’t compromise their organization. The lurkers will hopefully retain some of the lessons learned by the methodologies employed by the real subject matter experts. The lurkers, however, cannot be relied upon to provide actionable intelligence.  Early Influences and Collaborations  As with any research effort, our work has been influenced by many other researchers including Dr. Eric Nyberg, a professor in the Language Technologies Institute in Carnegie Mellon University's School of Computer Science. Dr. Nyberg’s research in this field helped us gain a greater understanding of machine learning and rule generation.  This research also draws upon theories introduced by Dr. Carolyn Rosé who teaches an applied machine learning vlass in the Human Computer Interaction Institute. Dr. Rose’s research focuses on better understanding the social and pragmatic nature of conversatio, and using this understanding to build computational systems that can improve the efficacy of conversation between people, and between people and computers. Looking Ahead Working with our customers over the past few years, we developed prototypes of tools that contribute to the data collection, analysis, and collaboration space. Alongside the development efforts for these prototypes, we are working on a version of Cerebro that will act as the trusted platform between them.  Additional Resources To read the paper Cerebro: A Platform for Collaborative Incident Response and Investigation, by Anne Connell, Tim Palko, and Hasan Yasar, please visithttp://ieeexplore.ieee.org/stamp/stamp.jsp?arnumber=06699007 For a list of other tools and resources developed by members of the CERT Cyber Security Solutions Directorate, please visithttp://www.cert.org/forensics/ To read the February 2013 report, CYBERSECURITY: National Strategy, Roles, and Responsibilities Need to Be Better Defined and More Effectively Implemented, please visit http://www.gao.gov/assets/660/652170.pdf

By Lori FlynnMember of the Technical StaffCERT Secure Coding team Although the CERT Secure Coding team has developed secure coding rules and guidelines for Java, prior to 2013 we had not developed a set of secure coding rules that were specific to Java’s application in the Android platform. Android is an important area to focus on, given its mobile device market dominance (82 percent of worldwide market share in the third quarter of 2013) as well as the adoption of Android by the Department of Defense. This blog post, the first in a series, discusses the initial development of our Android rules and guidelines. This initial development included mapping our existing Java secure coding rules and guidelines to Android applicability and also the creation of new Android- only rules for Java secure coding. Motivation for Our Work Software programmers produce more than 100 billion new lines of code for commercially available software put into operation each year, according to a recent article published in Defense Systems. Meanwhile, programming errors happen at an estimated rate of 15 to 50 errors per 1,000 lines of code. Even with the advent of automated testing tools, the article states that "numerous studies and a substantial amount of research suggest that approximately one error per every 10,000 lines of production code still exists after testing. That would equate to 10,000,000 errors in the code produced each year." To help cope with this enormous problem, the CERT Secure Coding team has developed a set of secure coding standards providing rules and guidelines that programmers and analysis tools can use to evaluate source code for compliance to the standards, which can help developers avoid or discover errors that make code insecure. A rule is advice for secure coding which is normative: a violation of the secure coding advice always must constitute a defect in the code. If the recommendation depends on programmer intention, then it might not be possible to automatically enforce the recommendation, and it is not considered normative.) A guideline is a secure coding recommendation that is excluded from being defined as a rule because it is not possible to form a normative requirement around it. For example, if a recommendation depends on programmer intention, then it might not be possible to automatically enforce the recommendation. Even if the recommendation is always a good idea, if violation of the recommendation does not always constitute a defect in the code, the recommendation is a guideline, not a rule. There have been several recent initiatives by the DoD to incorporate the Android OS into its mobile computing strategy: The Army Mobile Handheld Computing Environment is an effort to devise an Android-based smartphone framework and suite of applications for tactical operations. The Defense Advanced Research Projects Agency (DARPA) also selected the Android platform for its transformative apps program, which aims to develop a diverse array of militarily-relevant software applications (apps) using a new development and acquisition process.  Foundations of Our Work Our project aims to create rules that can be verified and enforced, as well as to develop a means for checking code for violations of these rules. This research was part of the Source Code Analysis Laboratory (SCALe). The Mobile SCALe project is extending this existing CERT concept to create a source code analysis laboratory environment for the analysis of code for mobile computing platforms. The first platform that Mobile SCALe has focused on is Android. Our work was led by a team of researchers who, in addition to myself, included William Klieber, Dr. Dean Sutherland, and David Svoboda of the CERT Secure Coding team; Dr. Lujo Bauer and Dr. Limin Jia of Carnegie Mellon University's Department of Electrical and Computer Engineering; and Dr. Fred Long of Aberystwyth University. The Japan Computer Emergency Response Team Coordination Center (JPCERT), including Masaki Kubo, Hiroshi Kumagai, and Yozo Toda, assisted in the development of the Android secure coding rules and guidelines. Our Mobile SCALe work can be separated into coding rules and guidelines development and static analysis development phases, which will guide the development of blog posts in this series. Our work in 2013 to develop Android secure coding rules and guidelines involved three main tasks: mapping our existing trove of Java secure coding rules, analyzing their applicability to the Android environment mapping our (at the time) in-development trove of Java secure coding guidelines, analyzing their applicability to the Android environment in time for that analysis to be included as an appendix in the published hardcopy book printed in 2013 developing new Android-only rules and guidelines for Java development  Rules and Guidelines for Java Secure Coding for Android As we outlined in our recently published technical report, Mobile SCALe: Rules and Analysis for Secure Java and Android Coding, we started developing secure coding rules and guidelines for Android by focusing on a specific language, Java, and adding an Android section to the CERT Oracle Secure Coding Standard for Java wiki. Although Android apps can be written in native code, such as C or C++, our 2013 work focused only on the Java language. Hosting secure coding rules and guidelines on our secure coding wiki allows us to collaboratively create these coding standards with assistance from the development and security communities, vet it with expert opinion, and receive feedback.  Android secure coding advice also exists elsewhere, but we found it in incomplete sets and in disparate locations across the web. One technique we use to develop new coding standards is to mine our other CERT secure coding standards. Many of our rules from C/C++/Java apply to C#. A second technique we use is to mine vulnerability databases, using public databases such as the Department of Homeland Security’s National Vulnerability Database, and sometimes CERT also is provided or develops privately held information about vulnerabilities. A third standard technique we use is to mine current literature. We found useful advice for secure Java coding for Android in separate locations, such as the Android developer website, Google, and security researcher websites, as well as research papers, online articles, and security conference presentations. We use a standardized format for all coding rules and guidelines (whether they are C, C++, Java, Perl, or Android-only) on our wiki. Each begins with a summary of the rule or guideline and an explanation of security issues it addresses. Then the wiki lists an example of non-conforming code and explains why it’s a problem. The wiki also includes references where readers can go for more information. We also provide a score that encases the severity of any vulnerability and the likelihood that the vulnerability can be exploited as well as an analysis of the cost of remediation if a violation is found in the code. Consider the following excerpts from our one of our guidelines applicable to secure development of Android apps: DRD00-J. Do not store sensitive information on external storage (SD card)Android provides several options to save persistent application data, one of which is External Storage, such as /sdcard or /mnt/sdcard.Files saved to the external storage are world-readable. Consequently, they can be modified by other apps installed on the device or by the user (by enabling USB mass storage and manipulating files from a PC).The Android API Guides [Android Guides 2013] Storage Options states:Caution: External storage can become unavailable if the user mounts the external storage on a computer or removes the media, and there’s no security enforced upon files you save to the external storage. All applications can read and write files placed on the external storage and the use can remove them.[Guideline]. Developers should not store sensitive data to external storage devices because files stored externally have no guarantee of confidentiality integrity, and availability.Noncompliant Code ExampleThe following code creates a file on the external storage and saves sensitive information to the file:  Compliant Solution (Save a File on Internal Storage) The following code uses the openfileoutput () method to create "myfile" in an application data directory with permission set to MODE_PRIVATE so that other apps cannot access the file:  Likelihood/Severity. We make three numerical estimates, based on our understanding of the security issues addressed by the secure coding advice (the rule or guideline). Severity estimates the consequences if the advice is ignored. Likelihood estimates how likely it is that a flaw introduced by violating the rule or guideline could lead to an exploitable vulnerability.  Remediation cost estimates how expensive it is to remediate existing code to comply with the advice. Our analysis found that the severity of a problem such as the one illustrated above would be high, and that the likelihood would be probable. The cost to remediate such a problem would be medium, since automatic detection with manual correction is possible if sensitive data sources are identified. The above format can be found for each of the CERT secure coding rules and guidelines. Three sections of the wiki were developed in 2013: Analysis for Android applicability of CERT Oracle secure coding rules and addition of Android-specific implementation advice to many of those rules on our wikiOur rules for Java are published in the book The CERT Oracle Secure Coding Standard for Java. The book was published in hard copy in September of 2011. Development of the standard continues on a wiki accessible and contributed to by the public but maintained by the CERT Secure Coding team. A summary of the analysis status at the end of 2013 is in Table 1 below. The full table with the current applicability analysis and analysis details for each  rule can be found here: Analysis of Applicability of CERT Oracle Java Secure Coding Rules to Android.For Java rules found to require Android-specific implementation advice, a new section, Android Implementation Details, was added to that rule’s wiki page just above the bibliography section. Analysis for Android applicability of CERT secure coding guidelinesA team of researchers developed guidelines for secure coding in Java, which were initially published on the wiki site maintained by the CERT Secure Coding team. Those guidelines were published in the book Java Coding Guidelines, which was published in September. Not all of the Java guidelines that the CERT Secure Coding team had developed and published, however, could be applied to the Android OS. A summary of the analysis status at the end of 2013 is in Table 1 below. The full table with the current applicability analysis, with analysis details for each guideline, is here: Analysis of Applicability of CERT Java Secure Coding GUIDELINES to Android. DRD-labeled Android secure coding rules and guidelinesWe also created new rules and guidelines for Android secure coding. As we state on the wiki, the new rules and guidelines labeled "DRD" are applicable only to the Android platform. They do not apply to the development of Java programs for other platforms. The list of DRD rules below is current as of February 17, 2014, and can be found on our wiki: - Do not store sensitive information on external storage (SD card) - Limit the accessibility to your sensitive content provider - Do not allow WebView to access sensitive local resource through file scheme - Do not broadcast sensitive information using an implicit intent - Do not log sensitive information - Always canonicalize a URL received by a content provider - Restrict access to sensitive activities - Do not release apps that are debuggable Two of the eight Android-only rules are actually Android-specific instances of more general Java rules: one concerning the logging of sensitive data and the other the canonicalization of file path names. Four of the remaining six rules focus on the handling of sensitive data by Android apps. In particular, these rules highlight aspects of Android programming that could lead unwary programmers to release sensitive data by misusing features of the Android architecture. Looking Ahead  The next post in this series will focus on the development of two tools that analyze information flow within and between Android apps. One of these tools analyzes potential communication between apps by focusing specifically on the sending and receiving of intents, which are a core inter-app communication mechanism in Android. Tracing these intents can be challenging because there are typically multiple methods of entry into a program. The blog post will also discuss a new static analysis tool we are developing. Later this year, we will also post about our ongoing Android secure coding work: expanding the coding rules and guidelines beyond Java and further development of our newest static analysis tool. Additional Resources To view the Android wiki on the CERT Secure Coding site, please visit https://www.securecoding.cert.org/confluence/pages/viewpage.action?pageId=111509535 To read the SEI technical report, Mobile SCALe: Rules and Analysis for Secure Java and Android Coding, please visithttp://resources.sei.cmu.edu/library/asset-view.cfm?assetid=69225

By C. Aaron CoisSoftware Engineering Team Lead CERT Cyber Security Solutions Directorate This blog post is the first in a series on DevOps At Flickr, the video- and photo-sharing website, the live software platform is updated at least 10 times a day. Flickr accomplishes this through an automated testing cycle that includes comprehensive unit testing and integration testing at all levels of the software stack in a realistic staging environment. If the code passes, it is then tagged, released, built, and pushed into production. This type of lean organization, where software is delivered on a continuous basis, is exactly what the agile founders envisioned when crafting their manifesto: a nimble, stream-lined process for developing and deploying software into the hands of users while continuously integrating feedback and new requirements. A key to Flickr’s prolific deployment is DevOps, a software development concept that literally and figuratively blends development and operations staff and tools in response to the increasing need for interoperability. This blog post, the first in a series, introduces DevOps and explores its impact from an internal perspective on our own software development practices and through the lens of its impact on the software community at large. At the SEI, I oversee a software engineering team that works within CERT’s Cyber Security Solutions (CS2) Directorate. Within CS2, our engineers design and implement software solutions that solve challenging problems for federal agencies, law enforcement, defense intelligence organizations, and industry by leveraging cutting-edge academic research and emerging technologies. The manner in which teams develop software is constantly evolving. A decade ago, most software development environments were siloed, consisting of software developers in one silo and mainframe computers and a staff of IT professionals who maintained that mainframe in another silo. The arrival of virtualization marked a technological revolution in the field of software development. Before, if I needed a new server for my web application, I would have to order the server, and wait for it to ship. Then, upon arrival, I would have to rack the server, install and provision the system, and configure networking and access controls, all before I could begin my real development work.  Today, virtualization allows us to create and proliferate virtual machines almost instantly. For example, my developers simply click a button to create a virtual machine, and it appears instantly. This ability to instantaneously generate synthetic computers that run on a shared infrastructure underlies a range of modern technologies, such as Amazon’s Elastic Compute Cloud (Amazon EC2), that provide resizable compute capacity in the cloud. This new immediacy powers a lot of cool technologies, such as cloud platform OpenStack, Platform-as-a-Service (PaaS) solutions such as Heroku or Microsoft’s Windows Azure, and software development tools such as Vagrant, as well as enterprise infrastructures of most modern companies. At the same time, these technologies enable us to automate more tasks and command larger, more powerful infrastructures to increase the efficiency of our software development operations. It Works on My Machine There’s a saying often heard among young developers: "it works on my machine." This references developers, often early in their careers, who write a piece of code to fix a bug. Then, after testing the code locally on their machine, they proclaim it fit for deployment. Inevitably, when they install it on the customer’s system, the code breaks because of differences in system configuration. This problem provides a canonical example of the types of issues that DevOps can help you avoid. To mitigate this prolific problem, SEI researchers leverage Vagrant to manage the creation of a canonical environment (which is a set of virtual machines) for each software project replicated locally for each developer on the project team. These virtual machines are configured to be identical to the machines in our testing, staging, and ultimately production clouds. This setup ensures that if it works on our developer’s local machine, it will also work on the production system, whether hosted by us or in a customer’s infrastructure. Moreover, synchronicity assures developers that if it works on their machine, it will work on other developers’ machines because they are using the same environment for that project. Files that define the configuration of these project environments are small and can be checked in to source control along with software code. The ability to check configuration files into source control allows the development team to update, share, and version the project environment—along with the code itself—with the assurance of parity throughout the team. This methodology also provides a far simpler onboarding process when new developers join a project, as their environments setup is reduced to a single "create environment" command. This advanced process, unimaginable a decade ago, offers just one example of the power and precision that DevOps automation brings to software engineering. A New Approach for Developing Software Another innovation that has impacted the manner in which software is developed stresses collaboration between developers who wrote the software and the operations team (i.e., the IT group) that maintains an organization’s hardware infrastructure. The incarnation of DevOps can be traced to 2009 when a group of Belgian developers began hosting "DevOps" days during which they stressed collaboration and interaction between these two entities. Previously, developers and operations staff would work independently until their interests converged, usually with an inefficient and costly struggle to integrate their work products and efforts for the final race to deployment. DevOps emerged from the realization that infrastructure should support not only the production capability, but also the act of development. Ideally, DevOps should exist in one merged environment and set of concepts. For example, if I am writing software in a virtualized environment, I can be assured that the software I’ve developed will deploy seamlessly in that environment. Integrated DevOps assures us that the operations team remains involved throughout the software development lifecycle to ensure a smooth, efficient process through transition and deployment. Just as security concerns cannot be initially ignored and then successfully addressed at the end of a project, the same is true for successful deployment and maintenance concerns. DevOps provides an ideal solution for iterative software development environments, especially those that release software updates frequently, such as Flickr. The initial push for DevOps stemmed from the need to integrate operations to make software development more efficient and of higher quality. At the SEI, we are taking that concept and pushing forward, along with many others in the software industry, to fully-automated DevOps processes. Automated DevOps In an article published in the August 2011 edition of Cutter IT Journal, Why Enterprise Must Adopt Devops to Enable Continuous Delivery, co-authors Jez Humble and Joanne Molesky wrote that "automation of build, deployment, and testing" are key to achieving low lead times and rapid feedback. The authors write that automation also offers "configuration and steps required to recreate the correct environment for the current service are stored and maintained in a central location." Any software organization must be an early adopter of innovation to maintain a competitive edge. As a federally-funded research and development center, the SEI must maintain high standards of efficiency, security, and functionality in systems we develop. Forward-thinking approaches to process, including heavily automated DevOps techniques, allow us to systematically implement, maintain, and monitor these standards for each project we work on. Looking Ahead While this post served to introduce the concepts of virtualization and outline some DevOps practices, future posts in this series will present the following topics: a generalized model for DevOps advanced DevOps automation DevOps system integration continuous integration continuous deployment automated software deployment environment configuration. We welcome your feedback on this series, and what DevOps topics would be of interest to you. Please leave feedback in the comments section below. Additional Resources To listen to the podcast, DevOps—Transform Development and Operations for Fast, Secure Deployments, featuring Gene Kim and Julia Allen, please visithttp://url.sei.cmu.edu/js. To view the August 2011 edition of the Cutter IT Journal, which was dedicated to DevOps, please visit http://www.cutter.com/promotions/itj1108/itj1108.pdf Additional resources include the following sites: http://devops.com/   (currently being revamped) http://dev2ops.org/ http://devopscafe.org/ http://www.evolven.com/blog/devops-developments.html http://www.ibm.com/developerworks/library/d-develop-reliable-software-devops/index.html?ca=dat-

By Anne ConnellDesign Team Lead CERT Cyber Security Solutions Directorate This blog post was co-authored by Barbora Batokova and Todd Waits. The source of a recent Target security breach that allowed intruders to gain access to more than 40 million credit and debit cards of customers between Nov. 27 and Dec. 14, 2013, has been traced to a heating, ventilation, and air conditioning (HVAC) service sub-contractor in Sharpsburg, Pa., just outside of Pittsburgh, according to a Feb. 5 post on a Wall Street Journal blog. The post stated that the intruders were able to gain access to Target’s system after stealing login credentials from one of Target’s HVAC subcontractors, who had been given remote access. This breach demonstrates how any vulnerability in a critical information system can be exploited to disrupt or harm the normal operation of any commercial or industrial sector. In this blog post, we will present a tool we have developed that increases a security incident responder’s ability to assess risk and identify the appropriate incident response plan for critical information systems. We define a critical information system as a computer-controlled information system that manages the operation and essential assets of any commercial or industrial sector, including energy delivery backup generators water sewer systems airports railway public transportations oil and natural gas emergency medical services information technology (IT) systems business management systems A compromise of any of these critical systems and its assets in terms of confidentiality, integrity or availability (CIA Triad of Information Assurance) typically leads to the loss of: financial stability revenue stakeholder trust customer confidence competitive advantage key technologies property life As the Target breach illustrates, organizations face large-scale distributed attacks on a regular basis. Another example of a similar breach is the compromise of networks of the three top medical device makers, according to an article published Feb. 10 in the San Francisco Chronicle. A report prepared by the Minority Staff of the Homeland Security and Governmental Affairs Committee presented in February 2014 stated "in the past few years, we have seen significant breaches in cybersecurity which could affect critical U.S. infrastructure… Nuclear plants’ confidential cybersecurity plans have been left unprotected. Blueprints for the technology undergirding the New York Stock Exchange were exposed to hackers." As a result, organizations and law enforcement officials are increasingly interested in investing in tools that will allow them to assess the security of their information systems, and identify and address any vulnerabilities. Often times, the approach to investigating and assessing critical information systems involves interviews with individuals who own and maintain such systems and recording the details about such systems in handwritten notes that are later transcribed into a word processing program. Due to the lack of a systematic process and standardized terminology, however, this approach is not effective, nor sustainable. It prevents people from accessing historical data from past assessments and leveraging the experience of previous assessors. Moreover, it does not afford real-time assessment and prevents people’s ability to compare and identify patterns of data. Finally, effective collaboration on assessments is nearly impossible, as it is cumbersome to share and expand upon the collected information. As we outlined in a paper presented at IEEE’s Technologies for Homeland Security Conference in November 2013, The CERT Assessment Tool: Increasing a Security Incident Responder’s Ability to Assess Risk, organizations and law enforcement officials tasked with assessing and/or responding to large-scale attacks on critical information systems typically face three challenges: It is hard for organizations and security incident responders to establish adequate levels of trust. Organizations are reluctant to share information and communicate when effective response to such attacks requires them to share information including networks, diagrams, and logs. Even after establishing adequate levels of trust, it is extremely hard to manage the sheer number of tasks and processes in the assessment and response plan. This plan includes identifying the vulnerability or a threat, determining if adequate controls are in place, mitigating the vulnerability if necessary, sharing the response plan, participating in collaborative decision making, and sharing information and analysis across the team of incident responders. Site administrators who own the critical information systems are at the core of the assessment and response plan and thus must be involved during the entire duration of the assessment and/or the event. This proves challenging because site administrators may need to make modifications to a system that they depend on, so the solution involves them in the process. Foundations of Our Approach In 2010, at the request of a federal government agency, we began work on development of a customer-specific version of a tool and accompanying training component that would enable that agency to conduct critical information systems assessments. After we deployed the tool to the agency, we received positive feedback that it was an operational solution with a built-in training path for critical information systems assessment. We received additional feature requests for the tool, including scaling the methodology to other domains within the organization. This feedback led to the development of the CERT Assessment Tool prototype, which provides a framework for assessing and addressing risk in critical information systems in various domains. The prototype, which is described in our paper, proves the concept of facilitating multiple people to actively collaborate, investigate, plan, and respond to ensure the confidentiality, integrity and availability of their critical information systems and the assets they manage. Our approach, which can be tailored to any commercial or industrial domain, is based on established methodologies and principles for risk management and information assurance, including NIST’s Recommended Security Controls for Federal Information Systems and Organizations (800-53) and Risk Management Guide for Information Technology Systems (800-30). The CERT Assessment Tool’s hierarchical data model addresses the identified need to standardize the terminology and taxonomy used during assessments. Having this data hierarchy and structure enables the application to create a systemic approach to assessment. Data Model Below we have included an image of a data model, which shows the hierarchy and relational qualities that each assessment has to present and future objects in the system. Each assessment can have multiple sites. For each site there are multiple systems, and for each system there are at least six system attributes collected to describe the platform, configuration, vulnerabilities, controls, mitigations, and responsible contact. The CERT Assessment Tool incorporates machine learning and decision support systems to provide actionable information and guidance that is repeatable. The tool and its systems reduce the reliance on experts, making knowledge more accessible and available when it is needed. The application utilizes Drools, an open-source business rule management system, to enable users to create domain-specific rules. Our approach defines a concrete process of information collection and assessment, guiding the user through various stages of assessment, through the implementation of the security plan.  The approach also incorporates a role-based model that defines the entities involved in the assessment and their responsibilities. How the CERT Assessment Tools Works The prototype of the CERT Assessment Tool currently runs on any Windows computer; however, the prototype can be architected for mobile and tablet deployment as well, which will provide flexibility to users and afford a nearly real-time assessment on site. The application enables users to record the interview with a site/system administrator and tag the information about the system to create object meta data, and establish a standardized assessment taxonomy. The image above illustrates areas where the tool segments the functionality for the assessment. The left-hand side of the image lists the systems and all of the tagged information. The center section is the free-form text area to support unrestricted information collection and tagging. The right panel provides real-time guidance questions based on the user’s inputs and expert system push notifications should a system be tagged with a known vulnerability. For a proper critical information system (e.g., HVAC, energy delivery, elevators) assessment, the user is required to enter and tag the following information: System Name Characteristics - e.g. type of connection, updating/patching process, model number, etc. Contacts - e.g. any person associated with the system Vulnerabilities - any flaws or weaknesses in the system security   Controls - any safeguards or measures implemented to minimize or eliminate the identified vulnerabilities Mitigations - any risk-reducing controls recommended from the risk assessment Once the information is tagged, the system creates push notifications that help the user ask additional interview questions, identify vulnerabilities or recommend possible mitigations. One of the sources the application taps for these recommendations is the National Vulnerability Database (NVD). For example, if one of the tagged system characteristics is Windows 7, the application will search the NVS and display the common vulnerabilities for Windows 7 to provide the user with additional context for the assessment. Once enough information is tagged and the system reaches a certain threshold of available information from past assessments, it is possible to create automatic rules using the Drools engine to provide specific expert advice about system vulnerabilities, existing controls, and mitigations. If we apply this approach and methodology to the HVAC system that was compromised during the Target breach, the initial (very simplified) assessment taxonomy would look like this: System Name: HVAC Characteristics: Vendor-Updated Windows 7 Internet-Connected Contacts: Fazio Mechanical Services - Vendor Vulnerabilities: Remote Access Active Directory Credentials Controls: Password Policy Based on this information, the CERT Assessment Tool with its guidance questions and decision support system would be able to recommend or alert the user that a remote access capability with active directory credentials is a vulnerability that needs to be properly mitigated to prevent a security breach. The CERT Assessment Tool can be used for assessments of a wide variety of information systems, which may or may not be deemed critical by the organization: Industrial Control Systems (ICS) that manage any kind of industrial production, including the following: SCADA (Supervisory Control and Data Acquisition) computer information systems monitor and control industrial, infrastructure or facility-based processes in commercial buildings, airports, hospitals such as HVAC and energy distribution. DCS (Distributed Control System) information systems control a manufacturing system, process, or any kind of dynamic system in which the controller elements are not centralized in one location but are distributed throughout the system with each component sub-system controlled by one or more controllers. Examples of such systems are electrical power grids, water management systems, and traffic signals. Distributed File Systems (DFS) that allow access to files from multiple hosts sharing via a computer network, such as the NFS from Sun Microsystems and DFS from Microsoft Network Information Systems (NIS) that manage networks such as gas supply or telecommunications Looking Ahead Given the increasing complexity and frequency of attacks on critical information systems, organizations and security incident responders need a tool that will allow them to effectively collaborate on assessments and security planning. The CERT Assessment Tool allows users to take a more pro-active role in risk management and gain an enhanced situational awareness of all systems on site. The application is the first line of defense in making users aware of the security of their information systems. Additional Resources To read the paper, The CERT Assessment Tool: Increasing a Security Incident Responder’s Ability to Assess Risk, by Anne Connell and Todd Waits, please visithttp://ieeexplore.ieee.org/stamp/stamp.jsp?arnumber=06699006.

By David MundieSenior Member of the Technical StaffCSIRT Development Team Social engineering involves the manipulation of individuals to get them to unwittingly perform actions that cause harm or increase the probability of causing future harm, which we call "unintentional insider threat." This blog post highlights recent research that aims to add to the body of knowledge about the factors that lead to unintentional insider threat (UIT) and about how organizations in industry and government can protect themselves. This research is part of an ongoing body of work on social engineering and UIT conducted by the CERT Insider Threat Center at the Carnegie Mellon University Software Engineering Institute. UIT is becoming increasingly common.  For example, about a year ago, spear phishers from China infiltrated the New York Times website in hopes of gaining access to names and sources that Times reporters had used in a story. A year earlier, Google pulled more than 22 malicious Android apps from the market after they were found to be infected with malware. This year, security blogger Brian Krebs reported that "The breach at Target Corp. that exposed credit card and personal data on more than 110 million consumers appears to have begun with a malware-laced email phishing attack sent to employees at an HVAC firm that did business with the nationwide retailer, according to sources close to the investigation." The Target breach spear phishing attack is an example of social engineering and illustrates how UIT can cause harm to an organization. Foundations of Our Work Insider threat remains a major concern among computer and organizational security professionals, more than 40 per cent of whom report that their greatest concern is employees accidentally jeopardizing security through data leaks and or similar errors. This finding led to our initial research into the field of UIT and the publication of the report, Unintentional Insider Threats: A Foundational Study. In that report, which seeks to understand causes and contributing factors in UITs, we developed the following operational definition: An unintentional insider threat is (1) a current or former employee, contractor, or business partner (2) who has or had authorized access to an organization’s network system, or data and who, (3) through action or inaction without malicious intent, (4) unwittingly causes harm or substantially increases the probability of future serious harm to the confidentiality, integrity, or availability As the examples above illustrate, the impact of UIT can be devastating, even though it is typically the result of actions taken by a non-malicious insider. Our initial work in this field led us to conduct a second phase of research that took a deeper dive into social engineering, specifically the psychological aspects of social engineering exploits. While technical solutions may be useful on the edges, at its core UIT is a human problem that requires human solutions. Unfortunately, organizations are often loathe to report insider incidents out of fear that the news could damage their reputation or value. A very limited amount of information is publically available through lawsuit records. We also examined news articles, journal publications, and other sources, including blogs, to compile information and identify contributing factors to UIT and social engineering. Through our analysis, we have compiled information on 28 cases that is now housed in our UIT social engineering database. Contributing Factors in Social Engineering Vulnerability In the course of our research, we identified several factors that made individuals more susceptible to attack. Although our sample did not allow us to draw any conclusions on demographic factors, such as gender or age, we were able to identify several organizational and human factors. The organizational factors that we identified in our report are as follows: Security systems, policies, and practices. Many of the cases that we examined provided insight into organizational policies and procedures. Some cases indicated that the victims violated those policies, but most incident summaries do not provide sufficient information to determine whether those factors are involved.  Management and management systems. Many of the cases reveal that simple authentication credentials provide attackers with access to internal emails, company data, and entire computer networks. In one case that we examined, an attacker gained direct network access from a username-password combination and did not need to place malware or execute any other indirect attack to cause damage. Organizations must regularly perform extensive security audits to determine how best to improve internal controls; they cannot rely on security established during initial installation of a system. Job pressure. Certain industries, such as news services, place a premium on obtaining and distributing information as quickly as possible. Employees in these types of organizations may be more susceptible to outside influence from social engineering due to this pressure. The human factors that we identified are as follows: Attention. In at least one of the cases we examined, we identified fatigue as a contributing factor. In that case, a phishing message was received late at night, and the individual responded before taking the time to analyze the message. The attacker may have information about work hours that could be used as part of an organized attack. Knowledge and memory. Several cases that we examined indicated that even when employees have been trained, a large percentage will still respond to phishing attacks. It is therefore important that organizations offer constant refreshers or other means to maintain employee knowledge and keep it fresh in their minds.  Reasoning and judgment. In some cases, an employee’s safeguards were lowered, perhaps in response to the realistic nature of a phishing message and/or the pretext created through reverse social engineering (e.g., offers to organizations or employees to assist in preventing or addressing outside attacks, in solving bank account problems, or in supporting system operations). Stress and anxiety. In one case, the victim knew that the organization and its customers were receiving phishing emails. This knowledge may have increased his desire to accept an offer of mitigation that appeared to be legitimate, but in actuality was just another phishing attack. I would like to stress that we are not breaking new ground with this publication. Our intent was to add meaningful input to the ongoing discussion on how social engineering relates to the body of research on insider threat and what organizations, specifically federal agencies, can do to mitigate contributing factors. Social engineering is a key component of UIT in that many non-malicious insiders are susceptible to social engineering, and thus become a threat to their organizations. An example of the impact of social engineering is the "Robin Sage" case where a cyber security analyst and "white hat hacker" contacted security specialists, military personnel, staff at intelligence agencies and defense contractors through bogus accounts that had been established on social networking sites such as Facebook, Twitter, and LinkedIn. The recipients of these communications ended up exposing far more information than their organization or its business partners would have wanted released in the public domain. Other examples similar to this have been made public since the "Robin Sage" study. Best Practices for Organizations As we stated in our report, organizations face many challenges in countering UIT social engineering threats, including balancing operational goals with security goals to remain competitive. To stay ahead, or at least keep up with phishers and spear phishers, we suggest the following best practices based on our analysis: Training. Organizations must continue to develop and deploy effective training and awareness programs so that staff members are aware of social engineering scams and can identify deceptive practices and phishing cues. Training plans should also teach effective coping and incident management behaviors to respond to social engineering. Minimize stress. When employees are stressed and working fast, they tend to be more susceptible to social engineering attempts. Organizational leaders need to examine whether they are creating a stressful environment or one that fosters a natural workflow. For example, one aspect of a plan to minimize stress could involve allocating time for employees to fulfill information security compliance requirements. Encourage employees to monitor and limit information posted on networking sites. For example, LinkedIn members often post details about their career history, including past cities where they have lived and worked. Phishers and spear phishers often contact individuals based on the information posted on such sites. They advertise false jobs and ask recipients to send a writing sample, building a sense of trust. A person seeking a job or a networking opportunity should be trained to avoid posting unnecessary details on social network sites. Moreover, job seekers should not operate in a vacuum.  In particular, they should seek the input of a co-worker or friend to review an email inquiry to assess whether it appears legitimate. One technique for detecting unintended disclosure of information on social networking sites is to put a piece of false information on each social media site the individual uses.  For example, a user could list an alternate city or alternate dates of employment on separate sites, so that a social engineering attempt based on information from that site can be detected easily. If someone contacts the individual referencing the false information, the individual would know that this is a social engineering attempt, rather than a legitimate contact. A lot of the best practices listed above are similar to those that our team recommends for intentional insider threat. These include training to heighten awareness and reduce human error, management practices to reduce likelihood of human error, e-mail safeguards that include anti-phishing, and anti-malware, antivirus protection, data encryption on storage devices, password protection, wireless and Bluetooth safeguards, remote memory wipe for lost equipment, and attention to what is posted on social media sites. While not all best practices listed above have been validated in our report, they are strategies that we have found to be successful. Looking Ahead Our research on UIT to date has been sponsored by the Department of Homeland Security. In the next phase of our work, we plan to examine UIT in the context of the 14 sectors of the economy identified by the DHS. For example, we will examine if phishing attacks differ based on the sector of the economy where they are executed. One challenge that we continue to face is the lack of verifiable information regarding social engineering and UIT. It would be ideal if we could set up an information sharing system where organizations could share information about unintentional insider threats without feeling as if their security or reputation were being compromised. As we stated earlier, socially engineered attacks that result in UIT are very much a human problem. While technical solutions may be useful, further research is needed to identify and mitigate the organizational and human factors of UIT social engineering. We welcome your feedback on our work. Please leave feedback in the comments section below. If you have experienced an UIT, please let CERT know (also by leaving feedback in the comments section). We are looking to increase the number of cases in our database, and greatly appreciate any help we receive. All your information will be kept strictly confidential. Additional Resources To read the SEI technical report, Unintentional Insider Threats: Social Engineering, please visithttp://resources.sei.cmu.edu/library/asset-view.cfm?assetID=77455 To read the SEI technical report, Unintentional Insider Threats: A Foundational Study, please visithttp://resources.sei.cmu.edu/library/asset-view.cfm?assetID=58744

By Julien DelangeMember of the Technical StaffSoftware Solutions Division The Architecture Analysis and Design Language (AADL) is a modeling language that, at its core, allows designers to specify the structure of a system (components and connections) and analyze its architecture. From a security point of view, for example, we can use AADL to verify that a high-security component does not communicate with a low-security component and, thus, ensure that one type of security leak is prevented by the architecture. The ability to capture the behavior of a component allows for even better use of the model. This blog post describes a tool developed to support the AADL Behavior Annex and allow architects to import behavior from Simulink (or potentially any other notation) into an architecture model. The amount of software in safety-critical systems, such as automotive, aerospace, and medical domains—where failure could result in loss of life—continues to grow in size and complexity. As systems become more complex, the challenges of architecting them continue to grow. In Safety Critical Systems: Challenges and Directions, John C. Knight took note of this trend: "Breakdown in the interplay between software engineering and systems engineering remain a significant cause of failure. It is essential that comprehensive approaches to total system modeling be developed so that properties of entire systems can be analyzed. Such approaches must accommodate software property and provide high fidelity models of critical software characteristics." Many software projects capture the behavior of components using state machines or use-case diagrams. These descriptions support development efforts, either as a specifications document or as inputs for code generators. They focus only on one component, however, and do not take into account the execution environment such as the hardware platform, communication buses and potential concurrent use of resources or the integration with other architectural elements such as connections with other components, etc.  Below we describe how a tool developed in accordance with the AADL Behavior Annex can support these capabilities. Foundations of Our Approach Integration issues account for more than 70 percent of software defects. As behavior is a key aspect of the system, it is crucial to analyze the impact between behavior specifications and the overall architecture. Most modeling techniques focus on behavior (which involves what the system is supposed to do) for design and implementation purposes. If such notations accurately detail behavioral aspects, then they do not incorporate the execution environment (which involves how the service is provided). This omission makes it hard to analyze the interactions and dependencies between system functionality and its execution environment, which is of primary importance because one aspect can impact the other. (For example, a task may request a resource being already locked, leading to an expected delay or a timeout.   In contrast, AADL allows architects to specify the structure of the system as well as the system’s execution environment. The remainder of this blog post, therefore, describes research aimed at integrating a behavior description (what the system performs) into the execution environment defined in AADL (how the system performs the functions: deployment of system functions on processors, buses, etc.). AADL provides mechanisms to extend component specifications, including user-defined properties and annex languages. While user-defined properties are an extension mechanism to describe non-functional properties and is semantically limited, the annex language allows architects to associate third-party languages to a component to specify various aspects. Several annex languages have been proposed such as the error-model annex, which allows for the specification of errors and faults that occur within a component or that may propagate across the architecture. The behavior annex brings into the overall architecture the ability to describe the behavior of AADL components as well as their interactions. SAE International, formerly the Society of Automotive Engineers, published the AADL Behavior Annex language extension to describe component behavior in terms of a state machine that interacts with component members (i.e., modifying data when the component is in a specific state) and interfaces (i.e., sending data through a port when the component is in a state or triggering a state transition upon a specific event received on an AADL event port). The behavior annex is currently being revised and improved by SEI researchers. Revisions include adding the ability to connect behavior specifications with other annexes such as the error-model annex. By connecting component descriptions, analysis tools can then show the impact between different aspects of the model (e.g. behavior and error specification). This modification would provide architects greater insight into the impact that component behavior can have on other system attributes. For example, this approach will allow architects to visualize how a task activation or communication delay can generate an error that could be propagated through the rest of the architecture. The behavior annex is now included in the Open-Source AADL Toolset Environment (OSATE).Télécom Paris Tech, an engineering school that has been involved in AADL development for several years, played a significant role in incorporating the behavior annex into the OSATE framework. The behavior annex language is being used in OSATE extensions for research projects such as the System Architecture Virtual Integration (SAVI) program, which was undertaken by the Aerospace Vehicle Systems Institute. To read more about the use of AADL in the SAVI program, which was a recent post in our ongoing series on AADL, please click here. Adding Behavior to the Execution Platform The AADL Behavior Annex augments system architecture description for a better design and more accurate analyses. From a security perspective, for instance, component behavior specifies the communication policy so that analysis tools can check that all channels exchange data at the same classification level. For example, if a component has two communication channels—one for highly secured data and another for low-secured data—the behavior description will specify which one is used to send data, thereby avoiding an important type of security leak. Such validation is infeasible with existing tools because they do not provide the appropriate semantics to capture these aspects. The loose coupling of notation and analysis tools makes it hard to analyze a system’s behavior within its execution environment. Integrating these notations in a single model helps address these issues and provides better end-to-end system analysis. In the previous post in our series on AADL, we explored the use of AADL in several application domains such as aerospace and aviation as well as projects to specify cyber-physical systems. Current projects focus mostly on defining the system architecture. For example, these projects define functions such as temperature readings and how they are realized (using a dedicated device or sensor) and inter-connected (a shared bus between the sensor and the computer receiving and processing temperature values). Until recently, however, AADL specifications did not describe how they perform their designated functions. Our latest work, therefore, augments the existing language so that users can associate behavior specifications with each hardware and software component. This capability will integrate another part of the component description and extend potential analysis so that tools can find more issues or refine actual diagnosis of system defects. Integration with Other Behavior Specification Several methods and tools already exist to specify component behavior. For example, Simulink or SCADE provide architects the capability to define state machines that characterize how a component processes data and produces its outputs. Thus, when a user establishes the AADL behavior of a component that has already been defined with another language, the following question is raised: Is the AADL behavior consistent with the other specification? Indeed, one has to ensure that both behaviors (the AADL model and a second language) are consistent to assure model correctness and consistency. This assurance increases confidence that verification done using the AADL model are applicable to other notations, as well. For that purpose, as shown in the picture below, current research efforts aim to automatically generate AADL behavior specification from existing behavior model validate an AADL behavior specification according to a behavior model defined with   another notation These two different methods to connect behavior specifications (from AADL and other languages) will help engineers ensure correct specifications. In addition, ensuring correctness and consistency in behavior specifications will help architects avoid discrepancies between different behavior specifications, ensuring that the behavior description is actually what is specified in other models and not what the engineer wants to expose. The approach shown in the picture above would avoid any mismatch of behavior specifications, especially if the models are developed by different engineers (i.e., one who makes the AADL model and another who focuses on behavior specification using another language). Likewise, generating AADL behavior specification from existing models would encourage engineers to integrate the behavior description in their architecture model by reducing the learning curve. These new methods are being implemented as part of OSATE and are already released under an open-source license in the testing branch. Interested users can already use them by downloading our testing release. Wrapping Up and Looking Ahead Current research efforts from the SEI and its collaborators aim to refine and improve the AADL Behavior Annex to provide a better and more accurate description of components behavior (e.g., by providing the ability to associate behavior for system or process components, a feature not currently allowed by the current standard). As part of this effort, SEI researchers are also working on methods to connect the AADL behavior specification with existing behavior models used in industrial projects. Our research and development activities are yielding an improved method and a set of open-source software components that integrate behavior in the architecture, thereby extending the description of the system and providing more information that can be analyzed and processed by analysis tools. The results of our work should help architects detect more issues when integrating components and support design and analysis. In future work we plan to investigate impacts of behavior aspects over system quality attributes (such as performance, safety or security). AADL can capture all these aspects in a single notation. This capability provides necessary information to help designers reasoning about several architecture variances and choose the most appropriate according to their business goals. We welcome your feedback on this research effort. Please leave us feedback in the comments section below. Additional Resources For more information about AADL, please visitwww.aadl.info and our wiki https://wiki.sei.cmu.edu/aadl/

By Nader Mehravari Senior Member of the Technical Staff CERT Cyber Risk Management Team This blog post was co-authored by Julia Allen and Pamela Curtis. In October 2010, two packages from Yemen containing explosives were discovered on U.S.-bound cargo planes of two of the largest worldwide shipping companies, UPS and FedEx, according to reports by CNN and the Wall Street Journal. The discovery highlighted a long-standing problem—securing international cargo—and ushered in a new area of concern for such entities as the United States Postal Inspection Service (USPIS) and the Universal Postal Union (UPU), a specialized agency of the United Nations that regulates the postal services of 192 member countries. In early 2012, the UPU and several stakeholder organizations developed two standards to improve security in the transport of international mail and to improve the security of critical postal facilities. As with any new set of standards, however, a mechanism was needed to enable implementation of the standards and measure compliance to them. This blog post describes the method developed by researchers in the CERT Division at Carnegie Mellon University’s Software Engineering Institute, in conjunction with the USPIS, to identify gaps in the security of international mail processing centers and similar shipping and transportation processing facilities. Foundations of Our Approach This engagement was not our first time working with the USPIS. Our first project, in 2011, involved helping USPIS ensure that packages sent to other countries from the United States complied with U.S. export laws. As a result of this initial engagement, USPIS had a well-defined process for addressing export law compliance. In addition to export screening, we have worked with USPIS on projects that involve incident response, authentication services, physical security, aviation screening for international mail, Priority Mail Express revenue assurance, and the development of mail-specific resilience management process areas for mail induction, transportation, and revenue assurance. The projects that we’ve worked on with the USPIS draw upon the CERT Resilience Management Model (CERT-RMM). CERT-RMM is a capability-focused maturity model that combines aspects of IT operations management with operational risk and resilience management, such as information security and business continuity. In response to the discovery of explosives in the two packages from Yemen and the subsequent development of new security standards by the UPU, the USPIS asked our team, which included David White, Pamela Curtis, and Julia Allen,  to use the CERT-RMM assessment method and process to develop an assessment methodology along with a companion field instrument for the following two UPU standards: S58, Postal Security Standards—General Security Measures defines the minimum physical and process security requirements available to critical facilities within the postal network. S59, Postal Security Standards—Office of Exchange and International Airmail Security defines minimum requirements for security operations relating to the air transport of international mail. Based on the requirements and design criteria that USPIS had specified, our team was faced with the development of a methodology and an associated field instrument that had to be repeatable (i.e., generate consistent results when used by different teams in the same situation); cost effective and scalable (i.e, economical and functional for all locations); accurate (i.e., evidence-based); meaningful (i.e., results could easily be acted on by owners and operators of the assessed facilities); and transparent (i.e., publicly available and could be used for self-assessment). One challenge in developing an assessment methodology based on UPU standards was contending with the ever increasing interweaving of physical and cyber domains. Securing even a mundane physical asset, such as a parcel, involves controlling both tangible and intangible objects, such as computer systems, networks, processes, and sensors. Another challenging aspect of this work was that the UPU standards were not developed with much consideration as to how member countries would use them to assess whether mail processing facilities were in compliance. This challenge made it hard to develop the streamlined approach that USPIS was seeking to evaluate mail processing facilities that handle international mail. A Proven MethodAs depicted in the figure below, our team developed a methodology that defined three phases for conducting the assessment: Preparation: Analyze requirements, develop an assessment plan, select and prepare the assessment team, send and receive the pre-assessment questionnaire, obtain and inventory objective evidence, and prepare for the conduct of the assessment (initial site visit and logistics). Onsite: Prepare participants, conduct interviews, examine objective evidence, document objective evidence, verify objective evidence, perform characterizations and ratings, formulate and validate preliminary findings, generate the final results of the assessment, and identify improvements to the method and the standards. Reporting: Deliver assessment results to sponsors and key stakeholders, and preserve and archive assessment results.  The questionnaire-based assessment instrument developed by our team contains a series of questions. Below are some examples of the types of questions included, as well as the areas (in bold type) that they cover: Risk Management: Do you conduct an annual risk assessment of each critical facility? Physical Security: Do you have a written facility security plan? Physical Security: Are facilities constructed to prevent illegal entry? Access Control: What is the process for the control, issuance, and removal of identification badges? Human Resources: Do you have a documented personnel selection and hiring policy and process? Mail Security: How are high risk mail items identified in the mail stream? Transportation: Are routes, schedules, and planned stops assessed for security? Our methodology also defines the requirements for evidence collection for each section of the standards. The team conducting the assessment must examine documented artifacts or receive oral and written statements and affirmations confirming or supporting the implementation (or lack of implementation) of a practice. If the assessment team observes specific weaknesses in implementation, team members record them on their assessment worksheets. For example, a weakness in implementation of S58 Section 5.1.1, Risk Assessment and Facility Security Plans, might be that the facility’s security plan covers general lighting requirements, but not interior emergency lighting. Our field instrument is used by the assessment team to keep track of results of interviews and other objective evidence collected. The assessment team uses this information to rate the degree by which each requirement in the standards is implemented. The assessment team also uses our field instrument to generate a summary of the results, in the form of a colored heat map, which is presented to the facility owners and operators. A small segment of a sample heat map is shown in the figure below. The heat map was designed to be easily understood by all relevant stakeholders, ranging from security operations staff to senior decision makers. (LEGEND: FI = Fully Implemented; LI = Largely Implemented; PI = Partially Implemented; NI = Not Implemented; S = Satisfied; NS = Not Satisfied.) Through our research, we aim to transition our solutions to our stakeholders by enabling them to conduct assessments independently. In early 2012, the assessment method was piloted by USPIS staff at several international postal administrations. As a result of one of the USPIS pilot assessments, one of the national postal administrations closed an international mail dispatch facility and moved operations to a new facility with improved security controls and conformance with UPU standards. Other assessments have shown that postal administrations largely conform to UPU standards and that having the specific feedback of assessment results encourages them to make the minor improvements needed to ensure full compliance. One consistent finding of our pilot highlighted the effectiveness of the method for producing accurate assessments. The S58 standard requires a single, written security plan for critical facilities. While each pilot location that the USPIS assessed maintained a security plan, the plans at those sites did not contain all of the elements that the standard requires, including general facility design standards, perimeter barriers, perimeter windows, doors or other openings, lighting, and locking mechanisms and key controls. The methodology that we developed for the USPIS included evidence-discovery procedures such as direct artifacts, indirect artifacts, and affirmations. These evidence-discovery procedures helped the assessment team locate those elements in other documents such as maintenance plans. Other benefits that participating facilities cited in field reports and assessment results included: insight into the strengths and weaknesses of current security practices recognition of a strong security posture by the International Civil Aviation Organization, World Customs Organization, and supply chain partners that rely on postal services for moving goods guidance to prioritize security-related improvement plans feedback on the maturity of the organization’s security program enhanced identification and prioritization of security risks The method, the results, and the overall explanation of the project are detailed in a technical note titled A Proven Method for Identifying Security Gaps in International Postal and Transportation Critical Infrastructure, which I co-authored with Julia H. Allen and Pamela D. Curtis of the SEI and Gregory Crabb of the USPIS. Looking Ahead Our work with the USPIS has demonstrated that, when properly interpreted, CERT-RMM can be applied to a wide range of business objectives. For example, CERT-RMM allows for the addition of new asset types (such as mail items) and the development of new process areas (PAs) (such as mail induction and mail revenue assurance), which can be used in concert with existing CERT-RMM process areas. These new PAs reference established CERT-RMM PAs for specific types of processing including the following: identification of discrepancies in mail-specific PAs invoke the Incident Management and Control PA, the purpose of which is to identify and analyze events, detect incidents, and determine an appropriate response. identification of risks in mail-specific PAs invoke the Risk PA, the purpose of which is to identify, analyze, and mitigate risks to organizational assets that could adversely affect the operation and delivery of services. In January 2014, we  presented our work jointly with the USPIS at the 93rd Annual Meeting of the Transportation Research Board Conference. One message we delivered during that presentation was that moving mail—from the time it is accepted to the time it is delivered—is fundamentally a transportation activity, where the entities being transported are mail items. The security and resilience management techniques we developed for postal administrations are thus also applicable to other transportation modes, as well as to other safety and security standards. These transportation modes include those modes that move people, (such as metropolitan area transit systems) and the transport of goods by air, ground, and sea. Pilot organizations have shown that using our method, which is ultimately a structured and scripted assessment instrument, is an effective way to assess compliance with the UPU postal security standards. During the 25th Universal Postal Congress in Doha, Qatar, in September 2012, our method was recognized as the approach for assessing compliance with the new UPU security standards.The USPIS and other postal sector organizations continue to use the assessment method to achieve initial results and assess progress made after implementing improvements. In 2014, the method will be provided to civil aviation authorities, who will use it primarily to assess the performance of postal administrations in meeting the screening and other international airmail security standards of S59. Additional Resources To read the SEI technical report, A Proven Method for Identifying Security Gaps in International Postal and Transportation Critical Infrastructure, please visithttp://resources.sei.cmu.edu/library/asset-view.cfm?assetid=77265. For more information about the CERT Resilience Management Model (CERT-RMM), please visithttp://www.cert.org/resilience/products-services/cert-rmm/cert-rmm-model.cfm. To hear Gregory Crabb, Inspector in Charge of Revenue, Product, and Global Security at the USPIS, discuss his organization’s use of CERT-RMM, listen to the CERT podcast at http://www.cert.org/podcasts/podcast_episode.cfm?episodeid=6E2258D2-DE92-C7DF-D7D2B43BEBCEF8A0&pageid=34576.

This post is the second in a series on prioritizing malware analysis.  By Jose Andre MoralesResearcherCyber Security Solutions Division Every day, analysts at major anti-virus companies and research organizations are inundated with new malware samples. From Flame to lesser-known strains, figures indicate that the number of malware samples released each day continues to rise. In 2011, malware authors unleashed approximately 70,000 new strains per day, according to figures reported by Eugene Kaspersky. The following year, McAfee reported that 100,000 new strains of malware were unleashed each day. An article published in the October 2013 issue of IEEE Spectrum, updated that figure to approximately 150,000 new malware strains. Not enough manpower exists to manually address the sheer volume of new malware samples that arrive daily in analysts’ queues. In our work here at CERT, we felt that analysts needed an approach that would allow them to identify and focus first on the most destructive binary files. This blog post is a follow up of my earlier post entitled Prioritizing Malware Analysis. In this post, we describe the results of the research I conducted with fellow researchers at the Carnegie Mellon University (CMU) Software Engineering Institute (SEI) and CMU’s Robotics Institute highlighting our analysis that demonstrated the validity (with 98 percent accuracy) of our approach, which helps analysts distinguish between the malicious and benign nature of a binary file.  The purpose of this project was to facilitate malware analysis by placing incoming malware samples in a prioritized queue. Those samples at the top of the queue would be considered the most dangerous and given highest priority for full analysis. This prioritization provides an order to a large set of malware samples, thereby guiding analysts in deciding which malware samples to analyze first. Background When the highly dangerous Flame virus was discovered in 2012, it came to light that a sample of this malware was in the repository of a major anti-malware company for at least two years. This incident motivated this research by observing that a malware analyst may have no way of choosing which malware to analyze first from a large repository. The repository may have a mix of both malicious and benign files.  Several analysis systems currently exist including Anubis http://anubis.iseclab.org/ BitBlaze http://bitblaze.cs.berkeley.edu/  Cuckoo Sandboxwww.cuckoosandbox.org Joe Securityhttp://www.joesecurity.org/ Malheurhttp://www.mlsec.org/malheur/ Norman Sharkhttp://normanshark.com/products-solutions/products/malware-analysis-mag2/ Darkpoint https://darkpoint.us/ These systems do a good job at informing a user what a binary under analysis does when executed, but do not perform any prioritization of samples for analysis. Analysts therefore need an automated approach to analyzing these files, getting a sense if they are benign, seen before malware, not seen before malware, and malware that is highly dangerous, and having the samples prioritized based on a set of features representing highly malicious malware that require high priority for analysis. With such an automated method in place, the analyst’s job of choosing which sample to start with is greatly simplified, producing a streamlined process of deep malware analysis that focuses on the most dangerous samples first. Approach To create a prioritized queue of malware samples in an efficient manner for a large set of incoming files, we decided to run each sample through a runtime analysis system for three minutes. The use of runtime analysis helped us quickly understand how a sample interacts with the underlying operating system. These interactions create execution events from which features can be extracted and used in our prioritization. We used the CERT Malicious Code Automated Run-Time Analysis (MCARTA) Windows runtime analysis system for this research. The system was given five sets of sample inputs:  A set of 2,277 known benign samples, all consisting of WIN32 Portable Executable files A set of 65 known malicious samples, the samples were the top ranked most infectious and wide spreading during the 2008-2013 period, as ranked by Kaspersky’s yearly reports A set of 291 malware samples named in the Mandiant APT1 report A set of 1884 malware samples from the Citadel and Zeus families A set of 11K known malicious samples downloaded from virusshare.com The sets described in items 1- 4 above were used for training classification and clustering algorithms, the set in 5 was used for testing. As each malware was fed to MCARTA, the runtime analysis lasted three minutes and the results were stored in reports. We built customized scripts to extract the desired features from the reports and place them in a comma-separated value file. These files were saved for later use in the research. We added the malware names for each sample from the anti-malware engines F-Secure, Microsoft, and Symantec that were being used by the anti-malware scanner service VirusTotal. The queries to VirusTotal were automated via a script that used VirusTotal’s web-based application programming interface (API) set. The features used in this research were based on our prior experience with runtime analysis and detection of malware. Our methodology was based on our suspicion assessment approach by asking the question "Who are you?" + "What do you do?" The first question provides provenance, which, for this research, we have defined as the person or group that developed the source code of the file under analysis, and attribution of the static file image for a given process and was answered by checking for a verified digital signature. The second question is based on the execution events of a process on a system and was answered using the extracted features from our MCARTA runtime analysis framework.  The features can be grouped in two broad areas: observed and inferred. Observed features are identified directly from the captured data, whereas inferred features required analyzing the captured data to conclude if a specific event occurred or not. Our focus for feature selection was on behaviors related to infection, injury, and survivability used by malware when entering a system:  Infection is normally an essential component of malware facilitating the spreading of the malware in other files and processes. This assists the malware in achieving several goals such as delegation of nefarious deeds to other processes and detection avoidance.  Injury is used by malware to damage the target system in some way including data stealing, access denial to essential system components, and deletion of files and processes.  Survivability is the malware’s need of running undetected for as long as needed to complete its overall goal. To survive, a malware can carry out several acts, such as stopping or deleting anti-malware software and running secretly as a background process or service. One can surmise that for any given malware at least one of these three behaviors are in use and can be identified via the various techniques used to implement them during runtime. Our features represent the techniques that can be used to implement these three behaviors of malware within the universe of file system, process, and network activity of an end host computer. The features used in this research were based on the Windows 7 operating system and were as follows:      File System Observed:Open FilesFind FilesCreate FilesGet file attributesMove FilesCopy Files   Inferred:Attempts to delete selfCopy/Move self to other file system locationOpen standard windows cryptography related dll Registry Observed:Created registry keysOpen KeysSet ValueDelete ValueEnumerate Value   Inferred:Values deleted from machine/currentversion/run/ registry keyDisable antimalware from starting on reboot by deleting its value from currentversion\run registry keySet value currentversion/run to start self or copy of self at rebootSet value in a registry key created by sampleDelete values with Registry key \CurrentVersion\Internet SettingsDeleted registry keys referring to an anti-malware product Processes Observed:Created ProcessesKill ProcessesOpen Processes   Inferred:Process started from file created by sampleDynamic code injection: open process with desired access, create thread & write  Windows Service Observed:Open ServicesCreate Service System Environment Observed:SleepsEnumerate System ModulesCheck for Kernel Debugger Threads Observed:Create Threads Network Activity Observed:Create ObjectsDNS Requests   Inferred:Connection attempts excluding localhost (127.0.0.1)Maximum connection attempts to same IP addressDifferent non-localhost IP addresses used in connection attempt Digital Signature Observed:Verified Digital Signature Window GUI Observed:Create Window of size 0 Our previous research indicated the inferred features above are highly suggestive of malicious behaviors by malware; their occurrence indicates high suspicion of potential malware presence. These features were collected for the process under analysis. During the analysis, we created a malware infection tree and extracted features for each process in the tree. Evaluation Criteria Our research is evaluated based on the usability of the chosen features to differentiate between malicious and benign samples with minimal false positives and false negatives. Further evaluation is based on producing a priority queue with an ordering of samples allowing an analyst to decide which samples to analyze first. The samples that are initially analyzed should be either the most malicious in the set either based on their features or their absence of features potentially indicating an ability to carry out nefarious deeds in a benign like manner. Results After collecting all needed features, we analyzed the data with different classification and clustering techniques to determine the correctness and usefulness of the feature set for prioritizing malware samples. We labeled each sample with its name from F-secure, Microsoft, and Symantec by submitting the SHA256 hash value to virustotal.com via their API set. From the 11K samples only 8,969 samples were used to report the research results. The remaining samples had non-identifiable answers or error messages returned from VirusTotal.com and were therefore omitted. The tests performed were as follows: Test 1: Attempt to separate benign from malicious in the training sets. Test 2: Prioritize malware samples for analysis based on their execution behavior. Test 1 was meant to validate if our runtime features were capable of differentiating between known benign and known malicious samples. We selected a group of popular classification algorithms—i.e., K-Nearest Neighbor (K = 10), Random Forest, AdaBoost, and Support Vector Machine (RBF Kernel)—and evaluated them using 10-fold cross validation. K-fold cross validation splits the data into K equally sized chunks; the algorithms are trained on K-1 chunks and evaluated on the remaining chunk. This process is carried out with each of the K chunks serving as the evaluation set, and results are averaged across all K evaluations.  We used two popular metrics for our evaluation of classification accuracy: area under the precision/recall (PR) curve, which measures how well each method can distinguish between malicious and benign samples, and area under the receiver operating characteristic (ROC) curve, which measures the false positive rate required to experience a particular true positive rate. For the PR curves, each algorithm assigns a score to each sample in the data set, where a higher score indicates that the sample is believed to be more malicious, and we measure how well the method ranks true malware above benign samples.  The list of sample scores returned by an algorithm are sorted and iterated through: at each step, we use the score of the current sample as a threshold for classifying anomalies, and calculate the algorithms precision (number of correctly identified malware divided by the total number of predicted malware) and recall (number of correctly identified malware divided by the total number of true malware). For the ROC curves, the same process above is followed, however at each step, we use the score of the current sample as a threshold for classifying anomalies, and calculate the algorithms true positive rate (number of correctly identified malware divided by the total number of true malware) and the false positive rate (number of correctly identified benign samples divided by the total number of true benign samples).  According to the graphs below, we observe that these popular classification algorithms are consistently able to distinguish between benign and malicious samples, i.e., experiencing >=98 percent average area under the PR curve and >= 97 percent average area under the ROC curve. Additionally, for the particularly important case of advanced persistent threat malware, Random Forest and AdaBoost, well regarded ensemble methods, experienced more than 96 percent average area under both curves, further indicating that the runtime features we selected are extremely useful in identifying the malicious or benign nature of a sample. Test 2 was carried out using the 11K samples as the test set. K-means clustering was used to group together malware with similar execution behaviors. The results are shown in the pie chart below. As shown in the pie chart above, a total of 25 clusters were formed with three large clusters, a handful in the hundreds, and the remaining samples in clusters of 10 samples or less. Each of the clusters was a mixed batch of samples from known malware families along with several samples labeled as a variant or generic. Only cluster 2 had a majority of samples from the same family. Consisting of seven total samples, six of these were identified by F-Secure as Trojan.Crypt.HO, the seventh sample was Win32.Sality.3. Since Adaboost and Random Forest gave the best results from Test 1, we report their ranking of clusters and individual samples for prioritizing malware analysis.  The table below shows the confidence percentage of a specific cluster being malicious by Adaboost and Random Forest. The first column is the cluster number, the second is the Adaboost confidence percentage, the third column is the Random Forest confidence percentage, the fourth column is the difference in the two confidence percentages and the fifth column is the average of the two confidence percentages for each cluster. The difference in confidence percentage was minimal in most cases implying a strong agreement by both Adaboost and Random Forest on the certainty of a cluster being malicious or not. Clusters 3, 19, and 21 had 100 percent confidence by Adaboost and none had similar confidence by Random Forest with cluster 19 the closest at 93.7 percent. Clusters 18,19, and 20 ranked highest in Random Forest and had much more similar scores in Adaboost. The average of the two confidence scores is very revealing with clusters 18,19,15,16 ranking the highest. Cluster 18 was the highest average confidence percentage overall, that cluster consisted of 4 samples, 3 were Trojan.generic and one was variant.symmi.  Although Adaboost and Random Forest are both ensemble methods, the methods optimize different loss functions, and therefore can provide different vantage points on the maliciousness of a sample. Averaging confidence scores of several machine learning algorithms can provide a majority consensus of agreement of maliciousness providing a strong basis for deciding which clusters are truly the most malicious in a given data set. In practice, these results are typically based on a single algorithm’s results, but our study suggests majority consensus could be a better approach.           The table below shows the number of individual samples with their confidence percentage for Adaboost and Random Forest. From the results, Random Forest did not have 0 percent confidence of a sample being malicious, implying no false negatives were produced. Adaboost had 0 percent  confidence for 149 samples which can be equated as 1.6 percent false negative rate. Random Forest placed most samples in the 60-99 percent confidence range where Adaboost had the majority in 90 to 100 percent range. The results show a tradeoff between no false negatives but lower confidence in random forest or much higher confidence with some false negatives in Adaboost. In practice, low or no false negatives is preferred since one missed malware can be enough to comprise an entire system, its users and data.  In our research, individual malware samples were prioritized by the confidence score given by Random Forest due to no false negatives being produced. The names of the top, middle, and bottom 10 ranked individual samples from Random Forest are listed in the table below. The top 10 consist of samples from known malware families, the middle 10 contain 1 generic and 1 non-descript sample. The bottom 10 also contain 1 generic and 1 non-descript sample. Though the top 10 are considered most malicious since they were determined by Random Forest to be the most similar to our training data, the bottom 10 included samples of well-known malware families such as Expiro, PoisonIvy, Anserin and Ramnit. The presence of these samples in the bottom 10 could suggest they were either analysis aware, unable to execute correctly in our analysis environment, or able to execute quietly and did not produce the needed features for our analysis to rank them higher.  The 402 samples receiving a 100 percent confidence in Random Forest were considered the most malicious of the test set and thus received highest priority. Recall the algorithms were trained on the observed and inferred features previously discussed and thus these 402 samples are seemingly the most similar to the trained data. Conversely the samples ranked lower at the 10 to 19 percent and in the 0 to 9 percent category of Adaboost may equally deserve high priority since all analyzed samples were known malware samples. Therefore lower ranking confidence and especially 0 percent confidence can be malware samples that are either very stealthy in their execution, are aware they are under analysis and thus purposely act benign, or could have not executed as expected due to missing required resources in our analysis environment. In general, analyzing the highest and lowest ranking samples in a priority queue may be a highly effective approach to prioritizing malware samples for analysis.   Collaborations I led this research with support from the CERT Cyber Engineering Solutions Group led by technical manager Hasan Yasar and Dr. Jeff Schneider of CMU’s Robotics Institute. Edward McFowland III, a Ph.D. candidate at the Heinz College and a master’s student in the Machine Learning Department) provided machine-learning expertise.  Concluding Remarks In this research, we set out to make malware analysis easier by using machine learning to prioritize samples based on their runtime behaviors. The prioritized samples are ordered in a queue with the most malicious at the top. The analyst can simply analyze samples as they appear in the queue. Our feature set of runtime behaviors produced a precision recall of greater than 98 percent and greater than 97 percent for ROC curves in differentiating between known benign and known malicious samples. In the specific case of advanced persistent threats, our analysis produced greater than 96 percent for both curves.  These results verify that our runtime features are highly useful in determining the malicious nature of a sample. Based on these results, we prioritized our samples with highest priority going to the samples at the top and bottom of our results lists. These were considered the most malicious due to their similarity with our training set (top of list) and the potential ability to modify behavior and appear benign (bottom of list). Our results also suggest that averaging confidence scores of several machine-learning algorithms could provide a majority consensus of agreement of maliciousness of a sample. This produces a strong basis for deciding which samples are truly the most malicious in a given data set. In practice, machine-learning-based malware detection is typically based on a single algorithm’s result, but our study suggests that a majority consensus could be a better approach.           Our future work will focus on expanding our feature set as malware evolves by detecting their new techniques. We will also create real-time malware detection systems for various operating systems using our feature sets. We will continue to improve current machine-learning algorithms to more accurately differentiate between the malicious and benign nature of a given sample. We welcome your feedback on our approach and analysis of our results. Please leave feedback in the comments section below.  Additional Resources To listen to the CERT Podcast, Characterizing and Prioritizing Malicious Code with Jose Morales and Julia Allen, please click here.  To read about other malware research initiatives at the SEI, please visithttp://blog.sei.cmu.edu/archives.cfm/category/malware  

By Will Klieber Member of the Technical Staff CERT Division  This blog post was co-authored by Lori Flynn.  Although the Android Operating System continues to dominate the mobile device market (82 percent of worldwide market share in the third quarter of 2013), applications developed for Android have faced some challenging security issues. For example, applications developed for the Android platform continue to struggle with vulnerabilities, such as activity hijacking, which occurs when a malicious app receives a message (in particular, an intent) that was intended for another app but not explicitly designated for it. The attack can result in leakage of sensitive data or loss of secure control of the affected apps. Another vulnerability is exploited when sensitive information is leaked from a sensitive source to a restricted sink. This blog post is the second in a series that details our work to develop techniques and tools for analyzing code for mobile computing platforms. (A previous blog post, Secure Coding for the Android Platform, describes our team’s development of Android rules and guidelines.) Our work was led by a team of researchers who, in addition to myself, included Dr. Lori Flynn, also of the CERT Secure Coding Team; Dr. Lujo Bauer and Dr. Limin Jia of Carnegie Mellon University's Department of Electrical and Computer Engineering; and Amar Bhosale.   Improving Dataflow Analysis Our first tool, for which we recently completed building a prototype, addresses a problem often seen in information flow analysis: the leakage of sensitive information from a sensitive source to a restricted sink. "Sink" and "source" are terms common to flow analysis. We define a source as an external resource (external to the app, not necessarily external to the phone) from which data is read and a sink as an external resource to which data is written. Sometimes the flow of information can be from a highly sensitive source to a place that’s not authorized to receive the data. So, the source can be high privilege and the sink can be low privilege. Integrity concerns can also be analyzed using the concept of information flow. Sometimes untrusted data is sent to a place that’s supposed to store only high-trusted data that’s been sent by an authorized source. If data travels from a low-trust source to a high-trust sink, that’s also a problem.  For example, a smartphone user might install an Android game that leaks the user’s entire contact list to a marketing company. Of course, government agencies also have their sources of sensitive information and they don’t want them to be leaked to unauthorized parties.  We designed and implemented a novel taint flow analyzer (which we call "DidFail") that combines and augments the existing Android dataflow analyses of FlowDroid (which identifies intra-component taint flows) and Epicc (which identifies properties of intents such as its action string) to track both inter-component and intra-component dataflow in a set of Android applications. Our analysis of a given set of apps takes place in two phases: In the first phase, we determine the dataflows enabled individually by each app and the conditions under which these are possible. In the second phase, we build on the results of the first phase to enumerate the potentially dangerous data flows enabled by the whole set of applications. Our tool differs from pure FlowDroid, which analyzes flows of tainted information. FlowDroid focuses on information that flows in a single component of an app; our tool analyzes potentially tainted flows between apps and, within a single app, between multiple components. Our taint flow analyzer prototype for static analysis of sets of Android apps, DidFail (Droid Intent Data flow Analysis for Information Leakage), was completed in March 2014. Our team is continuing to do research and development with this analyzer, focusing on methods to efficiently increase precision. Challenges in Our Work  One challenge that we encountered in our work is that while we had access to the full source code for FlowDroid, we could access only the binary code for Epicc. Unfortunately, Epicc produced a list of intents, but did not specify where in the app they originated. To integrate the results of FlowDroid and Epicc, we needed to make Epicc produce this missing information. Since it wasn't feasible to modify Epicc, we instead modified the Android Application Package File (APK) by tagging each intent with a unique ID so that Epicc would print the intent ID. To do that, we used a function of Soot that enables the instrumentation or transformation of APKs. Soot transformed the binary into Jimple, which is an intermediate representation of a Java program that is Soot-specific. In the Jimple, we looked for intent-sending methods (e.g., methods in the startActivity family) and then inserted new Jimple code that added an extra field (with a unique intent ID number) into the intent. Then, once we processed the APK file, we compiled the Jimple back to Dalvik bytecode and wrote a new, transformed APK file.  Another way of explaining this process is that we wrote a piece of code that takes the original APK and adds a unique identification to each place in the code where the APK sends an intent. The use of unique IDs enabled us to match the output of Epicc with the output of FlowDroid. (We modified FlowDroid to also look for these same intent IDs and print them in its output.) Epicc prints out a list of properties added using the putExtra method. That is how we add the unique ID. It is just a dummy extra field. Analysis Tool Publication  As of April 2014, we believe that our tool is the most precise (publicly available and/or documented) taint-flow static analysis tool for Android apps. Our tool enables users and organizations to be very secure about the set of apps they allow to be installed together while also enabling them to install the greatest number of apps that abide by their security policy.  This tool is freely available to the public for download along with a small test suite of apps that demonstrates the new analytical functionality it provides. Our recently accepted paper, Android Taint Flow Analysis for App Sets, provides details about the DidFail tool and test results of using it with an Android app set. We encourage you to first read the paper, and then download the DidFail tool and the provided test apps.  The DidFail tool has limitations that we hope to address in future research. These limitations include false positives that are caused by a coarse-grained approach to detecting information flows between apps. A finer-grained analysis could reduce the incidence of false positives. Also, the DidFail tool focuses exclusively on Android intents as the method of data communication across applications. Android apps have other means of communicating including  directly querying Content Providers reading from and writing to an SD card using native code and communication channels (e.g., sockets or the Binder) implemented by the underlying Android Linux operating system  A Tool to Address Activity Hijacking  The second Android app analysis tool, described below, was developed to be part of the CERT Division’s Source Code Analysis Laboratory (SCALe) suite of tools for testing code for compliance with CERT secure coding rules. This tool was designed specifically to grow our Mobile SCALe tool set that checks against our new Android-focused secure coding rules and guidelines. This new tool is now part of the CERT Division’s compliance checker tool set used for our SCALe code conformance analyses. This tool has been developed for a limited audience and is currently not available for public distribution. Activity hijacking attacks occur when a malicious app receives a message (an intent) that was intended for another app, but not explicitly designated for it. In the Android middleware, intents are the primary means of inter-app communication and may include a designation of the recipient, an action string, and other data.  If no recipient is designated in an activity intent, then Android tries to find a suitable recipient (e.g., an app that declares in an intent filter in its manifest file that it can handle the specified action string). If there are multiple suitable apps, then Android prompts the user to select which one to use. The user can also designate the chosen recipient as the default to handle all similar intents (e.g., intents with the same action string) in the future and thwart hijacking attempts.  However, a malicious app can trick the user by using a confusing name. In addition, an inattentive user might not give much thought to the choice. Moreover, the device’s touch screen might register a click for the malicious app that the user did not intend. Android does not require confirmation of the user’s selection (which would be helpful in mitigating accidental clicks), even though such an accidental click can irreversibly leak sensitive information. An implicit intent is an intent that does not specifically designate a recipient component by its fully qualified class name, as opposed to an explicit intent, which does. Only implicit intents are vulnerable to activity hijacking. In addition to inter-app communication, intents are also used for intra-app communication between different components of a single app. The use of implicit intents for intra-app communication has proved to be a common mistake in the development of Android apps, as well as a violation of our secure coding rules. A component might intend to communicate with another component in the same application, but if that component uses an implicit intent (instead of an explicit intent), it might be vulnerable to another app intercepting its message.  Unfortunately, it is easy for a developer to mistakenly make app interfaces public when they should be private, allowing malicious apps to hijack or eavesdrop on apps that have access to sensitive information or resources. Moreover, closely related apps may have been developed to send intents to each other without explicitly designating the recipient, leaving open an avenue for activity hijacking. In the technical report that the SEI published describing our work on this tool, Mobile SCALe: Rules and Analysis for Secure Java and Android Coding, we detail the design and implementation of our tool, which was constructed using the Soot Java analysis framework. Our tool identifies the method calls that send Android intents. Where possible, our tool identifies the action string associated with the intent and the target of the intent in the case of an explicit intent.  Our activity hijacking vulnerability detection tool analyzes each app individually to  find likely violations of secure coding rules produce a list of the different types of intents the app registers receive produce a list of program sites (source code or bytecode locations) that send intents, along with the action string and target class if known Looking Ahead  Our goal with this series of blog posts is to share the progress on our Mobile SCALe project work, thereby extending the existing CERT SCALe conformance process to create a source code analysis laboratory environment for mobile computing platforms. This blog post is intended to update you on our work on the Android operating system, the first area of focus for Mobile SCALe.  Our team is currently at work on improving our tool that looks for information flows where the data source is sensitive and the sink is restricted. The research challenge we’re focusing on is to develop an analysis to determine taint flow endpoints with the following (sometimes conflicting) goals in mind: precision, soundness, and the ability to operate within resource limitations (i.e., time and memory).  We welcome feedback on our work. Please leave comments below.  Additional Resources  To download the DidFail tool, please visithttps://www.cert.org/secure-coding/tools/didfail.cfm.  Read the SEI technical report, Mobile SCALe: Rules and Analysis for Secure Java and Android Coding, for more information about the work described in this blog post. 

By Robert C. Seacord Secure Coding Technical Manager  CERT Division Software developers produce more than 100 billion lines of code for commercial systems each year. Even with automated testing tools, errors still occur at a rate of one error for every 10,000 lines of code. While many coding standards address code style issues (i.e., style guides), CERT secure coding standards focus on identifying unsafe, unreliable, and insecure coding practices, such as those that resulted in the Heartbleed vulnerability. For more than 10 years, the CERT Secure Coding Initiative at the Carnegie Mellon University Software Engineering Institutehas been working to develop guidance—most recently, The CERT C Secure Coding Standard: Second Edition—for developers and programmers through the development of coding standards by security researchers, language experts, and software developers using a wiki-based community process.  This blog post explores the importance of a well-documented and enforceable coding standard in helping programmers circumvent pitfalls and avoid vulnerabilities.  Community-Based Development of the CERT C Coding Standard The idea for the CERT C Coding Standard as a community-based development project arose at the Spring 2006 meeting of the C Standards Committee in Berlin, Germany. Experts from the community, including members of the C Standards Committee, were invited to contribute and were provided with editing privileges on the wiki. The wiki-based community development process has many advantages; most importantly, this form of collaborative development engages a broad group of experts to form a consensus opinion on the content of the rules. Members of the community can sign up for a free account on the wiki and comment on the coding standards and the individual rules. Reviewers who provide high-quality comments frequently receive extended editing privileges so that they can directly contribute to the development and evolution of the coding standard. Today, the CERT Coding Standards wiki has more than 1,500 registered contributors, and coding standards have been completed for C and Java, with additional coding standards for C++, Perl, and other languages under development. These guidelines and standards, if implemented, could have prevented vulnerabilities, such as Heartbleed.  Heartbleed Heartbleed emerged as a serious vulnerability in the popular OpenSSL cryptographic software library. This vulnerability allows an attacker to steal information that under normal conditions would be protected by Secure Socket Layer/Transport Layer Security (SSL/TLS) encryption. Despite the seriousness of the vulnerability, Heartbleed is the result of a common programming error and an apparent lack of awareness of secure coding principles. Following is the vulnerable code: int dtls1_process_heartbeat(SSL *s) {  unsigned char *p = &s->s3->rrec.data[0], *pl;  unsigned short hbtype;  unsigned int payload;  unsigned int padding = 16; /* Use minimum padding */    /* Read type and payload length first */  hbtype = *p++;  n2s(p, payload);  pl = p;    /* ... More code ... */    if (hbtype == TLS1_HB_REQUEST) {   unsigned char *buffer, *bp;   int r;     /*    * Allocate memory for the response; size is 1 byte    * message type, plus 2 bytes payload length, plus    * payload, plus padding.    */   buffer = OPENSSL_malloc(1 + 2 + payload + padding);   bp = buffer;     /* Enter response type, length, and copy payload */   *bp++ = TLS1_HB_RESPONSE;   s2n(payload, bp);   memcpy(bp, pl, payload);     /* ... More code ... */  }  /* ... More code ... */ } This code processes a "heartbeat" packet from a client or server. As specified in the Transport Layer Security (TLS) and Datagram Transport Layer Security (DTLS) Heartbeat Extension RFC 6520, when the program receives a heartbeat packet, it must echo the packet’s data back to the client. In addition to the data, the packet contains a length field that conventionally indicates the number of bytes in the packet data, but there is nothing to prevent a malicious packet from lying about its data length. The p pointer, along with payload and p1, contain data from a packet. The code allocates a buffer sufficient to contain payload bytes, with some overhead, then copies payload bytes starting at p1 into this buffer and sends it to the client. Notably absent from this code are any checks that the payload integer variable extracted from the heartbeat packet corresponds to the size of the packet data. Because the client can specify an arbitrary value of payload, an attacker can cause the server to read and return the contents of memory beyond the end of the packet data, which violates our recommendation, INT04-C, Enforce limits on integer values originating from tainted sources. The resulting call to memcpy() can then copy the contents of memory past the end of the packet data and the packet itself, potentially exposing sensitive data to the attacker. This call to memcpy() violates the secure coding rule ARR38-C, Guarantee that library functions do not form invalid pointers. A version of ARR38-C also appears in ISO/IEC TS 17961:2013, "Forming invalid pointers by library functions [libptr]." This rule would require a conforming analyzer to diagnose the Heartbleed vulnerability. The Latest CERT C Coding Standard Within two years of launching the wiki, the community had developed 89 rules and 132 recommendations for secure coding in C. At that point, a snapshot of the CERT C Coding Standard was created, and published in October 2008 as The CERT C Secure Coding Standard. CERT’s Coding Standards continue to be widely adopted by industry. Cisco Systems, Inc., announced its adoption of the CERT C Secure Coding Standard as a baseline programming standard in its product development in October 2011 at Cisco’s annual SecCon conference. Recently, Oracle has integrated all of CERT’s secure coding standards into its existing Secure Coding Standards. This adoption is the most recent step of a long collaboration: CERT and Oracle previously worked together in authoring The CERT Oracle Secure Coding Standard for Java. The CERT C Coding Standard continues to evolve. Existing guidelines are updated as new standards, such as Programming Languages—C, 3rd ed. (ISO/IEC 9899:2011), are introduced. Through large and small modifications—from changing a word in a rule title to writing new code examples—the guidelines continue to be improved by the ongoing activities of contributors. Obsolete guidelines are regularly culled from the wiki, and new rules and recommendations are added as technology and research warrant. In 2013, a second snapshot of the CERT C Coding Standard was prepared for publication. The wiki had grown in the intervening five years: it now had 98 rules and 178 recommendations. We elected to publish only the rules, not the recommendations, in the second edition of The CERT C Coding Standard, published in April 2014. The rules laid forth in the new edition will help ensure that programmers’ code fully complies with the new C11 standard; they also address earlier versions of the C Standard, including C99.  The CERT C Coding Standard itemizes coding errors that are the root causes of current software vulnerabilities in C, prioritizing them by severity, likelihood of exploitation, and remediation costs. Each rule includes examples of insecure code, as well as secure, C11-conforming, alternative implementations. If uniformly applied, these guidelines eliminate critical coding errors that lead to buffer overflows, format-string vulnerabilities, integer overflow, and other common vulnerabilities when programming in C. The CERT C Coding Standard, second edition, covers all aspects of the new C Standard, including best solutions, compliant solutions, and pertinent language and library extensions. It also offers advice on issues ranging from tools and testing to risk assessment. Of the 98 rules, 42 are new—since the first edition, 30 rules have been deprecated, 30 more have been added, and a new section, Concurrency (containing 12 rules), has also been added.  A Tool for Developers  The Source Code Analysis Laboratory (SCALe) provides a means for developers to evaluate the conformance of their code to CERT’s coding standards. CERT coding standards provide a normative set of rules against which software systems can be evaluated. Conforming software systems should demonstrate improvements in their safety, reliability, and security over nonconforming systems. SCALe analyzes a developer’s source code and provides a detailed report of findings to guide the code’s repair. After the developer has addressed these findings and the SCALe team determines that the improved source code conforms to the standard, CERT issues the developer a certificate and lists the system in a registry of conforming systems.  Conformance to CERT coding standards requires that the source code not contain any rule violations. Occasionally, a developer may claim that code that appears to violate a rule actually is secure because of an exceptional condition. For example, library code with insufficient thread protection could still be secure when run only in single-threaded programs. If an exceptional condition is claimed, the exception must correspond to a predefined exceptional condition, and the application of this exception must be documented in the source code. Conformance with the recommendations is not necessary but, in many cases, will make it easier to conform to the rules and eliminate many potential sources of defects. SCALe has also been used by the Department of Defense (DoD), which increasingly depends on networked software systems.  One result of this dependency is an increase in attacks on both military and non-military systems, as attackers look to exploit these software vulnerabilities. Our technical report on this work, Supporting the Use of CERT Secure Coding Standards in DoD Acquisitions, provides guidance to help DoD acquisition programs address software security in acquisitions. It provides background on the development of secure coding standards, sample request for proposal (RFP) language, and a mapping of the Application Security and Development STIG to the CERT C Secure Coding Standard. Since its inception, more than 20 SCALe analyses have been performed on a variety of systems from both government and industry for a variety of languages, including C, C++, Java, and Perl. Addressing Challenges and Future Work  Coding standards are an integral part of the software development lifecycle and increasingly a requirement. The National Defense Authorization Act for FY13, Section 933, states "Improvements in Assurance of Computer Software Procured by the Department of Defense," requires evidence that government software development and maintenance organizations, including contractors, conform to DoD-approved secure coding standards during software development, upgrade, and maintenance activities, including through the use of inspection and appraisals. The Application Security and Development Security Technical Implementation Guide (STIG), Section 2.1.5, "Coding Standards," requires that program managers "ensure the development team follows a set of coding standards." A number of challenges make compliance with these requirements difficult: Secure coding standards must be developed for ubiquitous languages with no existing standards and, where possible, published by an international standards body to allow easy adoption by the DoD;  The number of actual rule violations discovered in conformance testing is excessive and must be reduced to levels that can be reasonably addressed by the development team;  It must be demonstrated that the adoption of secure coding standards will not degrade system performance and result in slow, bloated code. To address these challenges in the coming year our work will focus on the following areas:  C++ Coding Standard. To address the lack of secure coding standards, we plan to complete the CERT C++ Secure Coding Standard. C++ is used extensively throughout the DoD including for major weapons systems such as the Joint Strike Fighter. Existing C++ coding standards fail to address security, subset the language, or are outdated and unprofessional.  Reduce Rule Violations by Enforcing Secure Coding Rules in Interactive Development Environments. To address the problem of excessive rule violations, we plan to collaborate with Clang developers and Japan Computer Emergency Response Team Coordination Center (JPCERT) to develop additional analyses for Clang’s static analyzer to check for violations of a prioritized list of secure coding rules. Clang is an open-source compiler that has been integrated into Apple’s XCode integrated development environment (IDE), which is the primary tool for developing software for iOS and OS X. Catching rule violations early will prevent these errors from propagating throughout the codebase and will allow developers to learn secure coding techniques while programming. New checkers will be submitted into the main trunk of Clang and integrated into XCode (as well as any other IDEs that support Clang integration), improving software security for all developers who use Clang.  We will also collaborate with Dr. Bill Pugh, professor emeritus in the University of Maryland’s Computer Science Department and FindBugs creator, to develop analysis against unchecked guidelines in The CERT Oracle Secure Coding Standard for Java and to integrate this analysis into Eclipse so that analysis results are immediately available to Android platform developers.  Demonstrate the Costs of Producing Secure Code. We are planning a research project with Igalia to evaluate the costs of producing a CERT-conforming implementation of the Chromium browser project. Igalia is a contributor to Chromium and a member of the World Wide Web Consortium (W3C). Chromium has several properties that make it a compelling demonstration: Chromium is an open-source project, released under a BSD-style license, and is the foundation for the Google Chrome browser. The performance of Chromium, which has a bug bounty program, is intensely scrutinized; developers are unlikely to accept patches that fix theoretical vulnerabilities but adversely affect performance. Chromium is used by hundreds of millions of users, and a successful case study will be widely publicized and replicated. This research will evaluate the effort required to discover and mitigate secure coding violations in the Chromium codebase. We will evaluate the performance, size, and resource consumption of the code before and after remediation and note common anti-patterns and mitigations.  In the long term, we hope to continue to develop and refine coding rules for existing secure coding standards, additional coverage for other languages and platforms, and additional analysis capabilities. We welcome your feedback on our latest and future secure coding work. Please leave feedback in the comments section below.  Additional Resources For more information about, The CERT C Coding Standard, Second Edition: 98 Rules for Developing Safe, Reliable, and Secure Systems, please visit  http://www.informit.com/store/cert-c-coding-standard-second-edition-98-rules-for-9780133805383.  To sign up for a free account on the CERT Secure Coding wiki, please visit http://www.securecoding.cert.org.  To subscribe to our Secure Coding eNewsletter, please click here. 

By Will DormannVulnerability AnalystCERT Division The Heartbleed bug, a serious vulnerability in the Open SSL crytographic software library, enables attackers to steal information that, under normal conditions, is protected by the Secure Socket Layer/Transport Layer Security (SSL/TLS) encryption used to secure the internet. Heartbleed and its aftermath left many questions in its wake:  Would the vulnerability have been detected by static analysis tools?  If the vulnerability has been in the wild for two years, why did it take so long to bring this to public knowledge now?  Who is ultimately responsible for open-source code reviews and testing?  Is there anything we can do to work around Heartbleed to provide security for banking and email web browser applications?  In late April 2014, researchers from the Carnegie Mellon University Software Engineering Institute and Codenomicon, one of the cybersecurity organizations that discovered the Heartbleed vulnerability, participated in a panel to discuss Heartbleed and strategies for preventing future vulnerabilities. During the panel discussion, we did not have enough time to address all of the questions from our audience, so we transcribed the questions and panel members wrote responses. This blog posting presents questions asked by audience members during the Heartbleed webinar and the answers developed by our researchers. (If you would like to view the entire webinar, click here.) I have been a software vulnerability analyst with the CERT Coordination Center (CERT/CC) since 2004 with a focus on web browser technologies, ActiveX, and fuzzing. In addition to myself, answers to audience questions from the Heartbleed panel discussion are provided by: Brent Kennedy is a member of the CERT Cyber Security Assurance team focusing on penetration testing operations and research. Kennedy leads an effort that partners with the Department of Homeland Security’s National Cybersecurity Assessments and Technical Services (NCATS) team to develop and execute a program that offers risk and vulnerability assessments to federal, state, and local entities. Jason McCormick has been with SEI Information Technology Services since 2004 and is currently the manager of network and infrastructure engineering. He oversees datacenter, network, storage, and virtualization services and plays a key role in information security policy, practices, and technologies for the SEI. William Nichols joined the SEI in 2006 as a senior member of the technical staff and serves as a Personal Software Process (PSP) instructor and Team Software Process (TSP) mentor coach in the Software Solutions Division at the SEI. Robert Seacord is a senior vulnerability analyst in the CERT Division at the SEI, where he leads the Secure Coding Initiative. Seacord is the author of The CERT C Secure Coding Standard (Addison-Wesley, 2014) and Secure Coding in C and C++ (Addison-Wesley, 2002) as well as co-author of two other books. He is also an adjunct professor at Carnegie Mellon University. Attendee: Did anyone in the information security industry have suspicions about the security of OpenSSL before the Heartbleed story broke in the media? Will Dormann: Both Google and Codenomicon had investigated OpenSSL and discovered the Heartbleed vulnerability before its public release. Whether OpenSSL was specifically targeted is unclear. It is likely that a number of SSL/TLS libraries were tested, and it just happened that OpenSSL behaved unexpectedly, which is due to the vulnerability. Attendee: It seems as though the attacks did not start, as far as we know, until after the vulnerability was publicly announced. Has there been any effort to create avenues to distribute patches to vulnerabilities like this prior to publicly announcing the vulnerability? Will Dormann: We do not know when the attacks started. Before Heartbleed was publicly disclosed, we did not even know what to look for in an attack. Therefore, it is possible that the vulnerability was being attacked as early as two years ago, when the vulnerability was first introduced. The CERT Coordination Center (CERT/CC) offers support in coordinating vulnerabilities among affected vendors before public release. This minimizes the rushed efforts required by software vendors to produce updates after public disclosure. In this case, the CERT/CC was not involved in the pre-disclosure coordination of the OpenSSL vulnerability. Attendee: I’m curious; they said this vulnerability has been in the wild for two years. Why did it take so long to bring this to the public knowledge now? It looks fishy. Will Dormann: With any vulnerability that is discovered, there is some delay between its introduction and its discovery. It is quite common for vulnerabilities to go unnoticed for years. Attendee: Is the code part of the browser or part of the server-side platform? If the browser, is there anything we can do to work around Heartbleed to provide security for banking and email web browser applications? Brent Kennedy: Heartbleed mostly affects server-side applications, but there are some client applications (and most likely more to come) that reportedly are affected.  For vulnerable servers, the vulnerability exists on the server itself, not the client accessing it. No major browsers implement OpenSSL, so determining if you are safe when accessing email or online banking is dependent on your provider. While most banks reported no issues, it is worthwhile to check their status. Attendee: What is involved in "fully" patching this vulnerability at each impacted company? This seems to be a gray area at present. Jason McCormick: There is a black-and-white answer for the concept of being fully patched from the perspective of eliminating the acute vulnerability. That is simply to upgrade to a version of OpenSSL that is not vulnerable to Heartbleed, or to upgrade your software system that is using OpenSSL as a component to a version that addresses the Heartbleed vulnerability. That Open SSL upgrade "fully patches" the issue. The big gray area is what to do next. There is no one-size-fits-all solution here; unfortunately, organizations will have to make decisions based on their own risk tolerances and costs. Every organization should immediately be re-issuing certificates on Internet-facing systems that were vulnerable to Heartbleed as quickly as possible. The potential compromise of private key material that could be used for decryption of captured data or the impersonation of sites makes this an important consideration.  Organizations should consider their risk/cost trade-offs for their insider risk as well. For example, a university-style situation, where you have a large, heterogeneous user base with a culture of limited controls for open access, will have a much different risk/cost analysis for changing internal certificates than a corporation with a well-controlled, well-known user population with strict internal controls. Finally, you have to conduct a risk assessment of the other contents of the server or service that was affected by Heartbleed:  Are passwords in play for a web application?  What is the risk to compromised account information?  What other information was the server/service working on such that fragments of it may have been in memory?   Only by a thoughtful analysis can each organization determine what, if any, their next steps should be. That calculation will be different for every organization. Attendee: For our personal home computers, how do we get the updates to eliminate the Heartbleed vulnerability? Jason McCormick: Unless you are a home hobbyist and are affected by Heartbleed by running a server, there is nothing that home users need to do with their computers other than always keeping their software updated. The most important step that individual users can take is changing their passwords on the services they consume such as webmail, social media, banks, etc. Most large companies have announced publicly (to some degree) whether they were affected and issued recommendations for their services.  It is always a good idea to perform a regular change of your passwords regardless, so now might be a good time to do it. Additionally, even if a service you use was not affected by Heartbleed, if you used the same password for multiple sites and services, one of them may have been compromised, which means they are all compromised.  Yes, there is value in knowing whether or not an organization has re-issued their certificates, but that it too complicated for the average home user to understand. Beyond that, there is no reasonable way to know whether an organization has re-issued their certificate. By this time, we hope most organizations have done the right thing for their users. Finally, many larger online services are offering two-factor authentication (TFA) that requires a PIN-like code in addition to a password. This is accomplished using an authenticator client on a computer or smartphone or an SMS text-message based system. As part of your login process, you would enter your username and password and on the following screen the numeric code printed on the authenticator app or from a text message. Anywhere and everywhere this service is offered, users should be taking advantage of it. The use of a TFA for logins greatly mitigates the risk of compromised and weak passwords. Attendee: Shouldn’t organizations also check their applications that act as Secure Socket Layer/Transport Layer Security clients — whether those are desktop or Web applications, developed in-house or externally — if they use outdated versions of OpenSSL? This vulnerability can also be exploited against clients, not just servers. Couldn’t the memory of those client applications also contain sensitive information that could be stolen if they connect to a malicious or compromised server? Brent Kennedy: The short answer is "yes" although the server side vulnerability carries a greater risk. Exploiting Heartbleed via a client-side application would be a multi-step attack. The attacker would have to stand up a malicious SSL/TLS server and trick the user into visiting that server using a vulnerable client. In the event that it does happen, memory would be dumped from the user’s host machine. This could contain anything that is actively being processed on the computer, not just data related to the specific client application.  Jason McCormick: It is always good practice to keep all systems updated, including clients. While it is theoretically possible to attack a pure client using Heartbleed, most attacks are not practical en masse, both because browsers do not use OpenSSL (Internet Explorer uses the Microsoft Crypto library; Firefox and Chrome use network security services (NSS)) and because attacking the clients using Heartbleed would require an initial compromise such as phishing a person to connect to a malicious site. Attendee: You mentioned that website owners might want to get new SSL certificates and revoke the old ones. But how should they mitigate against the fact that browsers and most TLS clients have broken certificate revocation checking where they soft fail when they don’t get an online certificate status protocol (OCSP) response—they accept the connection. This means a man-in-the-middle attacker who obtains a certificate through Heartbleed can impersonate the site to users possibly indefinitely even if the old certificate is revoked by also blocking OCSP responses to those users. Will Dormann: It is true that a certificate revocation may not be honored by a client application. However, that is not reason to skip the revocation in the first place. For more details, see http://news.netcraft.com/archives/2014/04/24/certificate-revocation-why-browsers-remain-affected-by-heartbleed.html. Jason McCormick: The short answer here is you can’t, at least not practically, without both increasing the industry-wide robustness of OCSP services and implementing different default behaviors in the browsers. OCSP soft-fail is a deliberate behavior choice by browser makers because hard fail would cause a serious disruption to many user experiences (which is a debate for another time).  OCSP stapling (or TLS Certificate Status Request Extension) is a great step forward here, but unfortunately it is not widely implemented yet. Thought leaders in the IT world need to be pushing concepts and technologies like OCSP stapling forward at every opportunity. Attendee: Obviously, Heartbleed exposes a number of flaws in our security infrastructure (e.g., OpenSSL is being maintained by a very small number of people). I’d like to hear some about how the panelists view the resiliency of certificate authentication when stressed by something like Heartbleed. Jason McCormick: I can’t say that I agree with the opening statement of this question, that Heartbleed exposes any fundamental flaw in the architecture of our varying security infrastructures. While the Heartbleed vulnerability is serious and pervasive, it is fundamentally a coding mistake. Heartbleed is not revealing a protocol weakness such as BEAST, CRIME, or the renegotiation attacks.  As for OpenSSL being maintained by a small number of people, that is very true, but it sounds as if plans are in the works to change that through The Linux Foundation. This is great news and hopefully will lead to faster and better evolution of the OpenSSL system. The interesting issue that persists though is how you find quality cryptographers who are also good programmers and who can do work on OpenSSL. This is a hard combination to come by and I hope the financing that is expected to flow to OpenSSL can overcome some of these challenges. This is a very unfortunate bug, and I am sure more bugs will be found in this software just as all other software has bugs. I do not think, however, that OpenSSL is fundamentally broken such that it should be abandoned or considered fundamentally flawed. Additionally, Heartbleed also has nothing to do with certificate authentication. It is an important point that the Heartbleed vulnerability itself, while damaging, is limited to a particular function of the TLS protocol used for "keepalive" checks and for path MTU discovery for DTLS connections. X.509 certificate authentication and authorization is an entirely different function within the TLS protocol specification. It is used during the handshake phase of the TLS session establishment to check identity and establish the encrypted transport session. Certificate authorities and the related constellation of technologies and protocols, some of which do have some interesting challenges, are entirely separate from TLS and OpenSSL. Attendee: A mature process through the entire software development cycle is essential to reducing this type of vulnerability. Can a mature process that will catch defects like Heartbleed dovetail with an Agile software development approach? Bill Nichols: The short answer is "yes." There is no wide agreement on the specific practices in Agile, but it is generally agreed that Agile involves delivering value to the user. Vulnerabilities deliver negative value and therefore are anti-Agile by definition.  Attendee: In my 25+ years of consulting in programming circles, MANY large and well-known corporations are merely "maintaining" code (adding Band-Aids, enhancement) and not building from scratch. That aside, developers should not be the main source of evaluating the resulting code — for adherence of standards, injection of vulnerabilities, etc. Bill Nichols: If developers are maintaining or enhancing code, they should take responsibility for any changes they introduce. On many old code bases, this is hard work indeed. Excessive change is dangerous because changes often introduce new problems. The developers should not be the last line of defense, but they should be among the first and take personal responsibility not to allow vulnerabilities to escape. The tools checking code after development are absolutely essential, but very imperfect. The only way to get clean code out is to put clean code in. Moreover, someone must review each and every find from those tools. Attendee: Given this issue has existed in practice for many years, what would the panelists suggest are lessons for various positions in in the ecosystem who all missed this?: (1) open-source code reviewers, (2) component integrators, (3) testers etc., (4) auditors? Bill Nichols: Reviewing code is hard. The following practices have been known to work:  Review only a 200 to 300 lines in a single sitting. Use a checklist of items. Review the entire section of code for a single item from your checklist before moving on. Write your own checklist for review so that you recognize the problem in the code immediately when you see it. You must inspect, not read the code. If you take less than an hour for 200 lines of code, you have probably gone too fast. Many separate studies have found that review rates of more than 200 lines per hour are not very effective. For many codes, it is likely that the inspector will, on average, find less than one issue per hour. This low rate contributes to making inspection difficult to perform because it feels slow and unrewarding. Nonetheless, that find rate is many times faster than integration or system test.  For integrators and auditors, I recommend running compilers with all checks turned on. Follow up with a low-cost static check tool such as SonarQube, then run two or more proprietary static analysis tools, for example, Coverity and CAST among others. Our experience has been that using analysis tools is better, than relying on a single product because analysis tools tend to have limited overlap in their results. Resolve each and every issue. There will be a large portion of false positives. If you cannot afford to resolve all the issues, you should recognize that 10 to 15 percent are likely to be real defects. Oddly enough, some people have found that the lower ratio of total finds (positive and negative), the lower the true positive to false positive ratio. You should definitely consider dynamic checking tools such as fuzzers. You should re-inspect any module in which you discover a defect in test. Robert Seacord: There are many lessons here. One is that developers, reviewers, and auditors should make sure they have an up-to-date and comprehensive knowledge of secure coding. The SEI provides secure coding training and numerous books have been written on the subject. Attendee: Robert, is it realistic to burden software developers with an ever-increasing set of things they need to worry about rather than building the solutions to these problems into their languages and tools? Robert Seacord: I am not sure if it is realistic, but because developers are the last line of defense between the languages and tools they are using and deploying vulnerable products, they need to shoulder the burden. We are heavily involved in language standards committee work to try to help improve the inherent security of these languages. You can look at David Keaton’s blog post to see some of the specific improvements we have made in C11. Bill Nichols: Mistake-proofing the development environment is desirable, but Robert described some barriers. Moreover, a seemingly safer environment can lead to compensating behavior (i.e., the Peltzman effect) that can undermine the improvements. Regardless of improvements to the environment, developers must hold themselves to a high standard. Our experience suggests that a large portion of the vulnerabilities result from mistakes programmers make routinely, but can be found and removed with some discipline. Developers need to learn and apply the meta-tools required to do good work. They need to understand and use sound design principles, sound coding practices, and effective review, inspection, and test. If these techniques are applied diligently, the exposure can be reduced by a factor of between 5 and 20 without adding overall cost to development. Attendee: Shouldn’t SSL_malloc()’s return be checked to be a valid pointer prior to using it as destination for memcpy? What if SSL_malloc() returned a NULL on memory exhaustion? Robert Seacord: Yes. This is almost certainly a violation of ERR33-C. Detect and handle standard library errors. Attendee: Is C++ a suitable language for writing security-critical code or are there more secure languages available that would avoid problems such as this? Robert Seacord: There are languages that are less susceptible than C and C++ to reading or writing memory outside of the bounds of an object, as occurred in Heartbleed. Whether or not these languages are appropriate for a particular system depends on a number of factors including the type of application, existing code, and the knowledge and skills of the developers. Networking applications are frequently written in C because of the need to optimize performance and because of the bit level manipulations frequently required. Attendee: Had the OpenSSL coders used something like SafeC implementation, do you think HeartBleed bug would not have occurred? Robert Seacord: Probably not, but the reason may have been because OpenSSL might not have been widely adopted if it had not been written in a language for which compilers and tools were widely available. Language is part of the issue, but there is no such thing as a secure language. Attendee: Has anyone determined if the vulnerability would have been detected by static analysis tools? Robert Seacord: The vulnerability was not detected by static analysis, either because the analysis was not performed or because many analysis tools such as Coverity would not have detected the problem because of the number of levels between inputting the tainted value and using it. Many static analysis tools (including Coverity) can now detect this vulnerability, particularly if the code is annotated so that the tool is aware that the certain macros/functions return tainted inputs.  Attendee: A majority of U.S., federal, civilian, DoD, and Intel websites have unknown anomalies that potentially are very similar to the "Open SSL" issue. The June deadline for FedRamp is quickly approaching that mandates certification prior to Authority to Operate. What actions should the U.S. Federal Government (including Intelligence) be taking to prevent similar unknown vulnerabilities and anomalies? What differentiated this particular proprietary fuzzing tool (rather than open-source that did not find the anomaly) that caused this tool to find Heartbleed? Should the U.S. CERT be advising the U.S. Federal Government to be redirecting resources to determine anomalies prior to "Authority to Operate" (ATO) using a proprietary tool to prevent hacking of their networks to ensure zero day vulnerabilities prior to ATO? Robert Seacord: In general, there is no one tool that is likely to catch all possible problems. The best solution is to use a collection of dynamic and static analysis tools (and to develop the code securely to begin with). SEI/CERT’s approach is to use the SCALe to provide conformance testing of source code against secure coding standards using a variety of static analysis tools. This is consistent with the requirements of Section 933 of the FY13 National Defense Authorization Act. Attendee: Could you recommend some static program analysis tools? Robert Seacord: As an FFRDC, we cannot endorse any tools. In general, the static analysis tools tend to have non-overlapping capabilities, so you may need to use more than one. For the Source Code Analysis Laboratory (SCALe), we use Coverity, FindBugs, Fortify, and Eclipse to analyze Java code and Coverity, Fortify, LDRA, Microsoft Visual C++, PCLint, GCC, and Compass Rose for C language systems. Attendee: What are the implications of the general use of fuzzing tools for the U.S. federal government? Should a minimum standard be established for what a fuzzing tool should accomplish to ensure no zero day vulnerabilities?  Will Dormann: CERT works with vendors to encourage them to use fuzzing tools. If they do not fuzz on their own, somebody else will, and they may discover vulnerabilities using that technique. It would be nice if there were some sort of standard for fuzzing robustness. However, enforcing a requirement can be difficult. How can one objectively measure the amount of fuzzing that an application or library has endured? There are so many variables at play that it is not as simple as saying that "Application X withstood Y number of fuzzing iterations without crashing." The vision of ensuring no zero day vulnerabilities is a worthy goal, but perhaps not ever achievable. Also consider the fact that some vulnerabilities are discovered without the use of fuzzing at all. The idea of fuzzing metrics has been batted around in the past (http://dankaminsky.com/2011/03/11/fuzzmark/); however, it appears that not much progress has been made in this area. Attendee: There’s been a lot of chatter about open source being the problem. Do the panelists think a closed-source solution would’ve fared any better? There seem to be plenty of vulnerabilities regardless of open/closed status, so shouldn’t the lesson learned be that regardless of open/closed status, we need to do a better job of coding our software securely and continuing to test it even after it’s released? Will Dormann: Open-source software is not inherently less secure or more secure than closed-source software.  Vulnerabilities can be discovered without the availability of source code. The quality of the code being written is what affects the quality of the applications and libraries that we use. Looking Ahead As the answers above demonstrate, Heartbleed is fundamentally a coding mistake and one that could have been prevented. Through open exchanges like this, we hope to prevent future vulnerabilities. We welcome your feedback in the comments section. 

By Sarah A. SheardSenior EngineerSoftware Solutions DivisionThis post is the first in a series on this topic.  The Government Accountability Office (GAO) recently reported that acquisition program costs typically run 26 percent over budget, with development costs exceeding initial estimates by 40 percent. Moreover, many programs fail to deliver capabilities when promised, experiencing a 21-month delay on average.  The report attributes the "optimistic assumptions about system requirements, technology, and design maturity [that] play a large part in these failures" to a lack of disciplined systems engineering analysis early in the program. What acquisition managers do not always realize is the importance of focusing on software engineering during the early systems engineering effort. Improving on this collaboration is difficult partly because both disciplines appear in a variety of roles and practices. This post, the first in a series, addresses the interaction between systems and software engineering by identifying the similarities and differences between the two disciplines and describing the benefits both could realize through a more collaborative approach.  Origins of Systems Engineering and Software Engineering Systems engineering is an interdisciplinary field of engineering that focuses on how to design and manage complex engineering projects over their lifecycles. System engineering textbooks first appeared around 1960, long before the advent of software engineering. However, systems engineering only emerged as a discipline (complete with journals, documented practices, and academic departments) in the 1990s, well after software engineering had been an established field in computer science. As part of the maturation of systems engineering, two capability models were created that documented its practices; by the time they had been merged and then folded into the Capability Maturity Model Integration (CMMI) model, they had become mainstream. Standards were written in the 1990s that document systems engineering, and in the 2000s they generally have been harmonized with software standards.  The discipline of software engineering traces its roots to a 1968 NATO conference when Mary Shaw named "software engineering" as an aspiration, a field that was needed but not yet developed. Over the next several decades, software engineering both became better defined and grew in new directions. In 1996 Shaw categorized software engineering eras as "Programming-any-which-way," (1955-1965), "Programming-in-the-small," (1965-1975), "Programming-in-the-large" (1975-1985), and as "Programming-in-the-world" (1985-1995). The field continues to evolve, with recent focus on topics such as mobile programming and big data.  Present Day Software engineers today have adopted many of the principles and practices of systems engineering. For example, developers do not merely react to requirements; they elicit, prioritize, and negotiate changes to them. Developers do not just build code; they architect it, plan it, and establish rules to ensure data is appropriately shared and processed. Software project managers do not just distribute development tasks among individual coders and hope they integrate a seamless final product; they ensure that developers collaborate using plans, processes, and daily standup meetings. Software engineers do not just write programs; they design modular pieces, define integration principles, build for interoperability and modifiability, and perform integration testing. Meanwhile the systems whose development programs employ systems engineers are increasingly dependent on and controlled by software. For example, the effort to write and maintain software on fighter aircraft now exceeds all other costs. Systems engineers are responsible for ensuring that every part of the system works with every other, and because software represents the intelligence of human-made systems, it is especially important to the integration of the system. Yet many chief systems engineers and program managers have had more experience in mechanical or electrical engineering than in software or software engineering.  Overlap and Uniqueness Today, systems engineering and software engineering activities overlap to a great extent. The figure below shows activities common to both disciplines in the middle (purple), activities related more to systems engineering than software on the left (red), and activities related more to software than systems on the right (blue). The term "hardware" appears only on the right, because software engineers use it to mean anything that isn’t software, but systems engineers generally do not use it. Instead, they call physical subsystems by name (e.g., power, propulsion, structures) or describe disciplines or domains (e.g., mechanical engineering, survivability, software, orbit analysis, launch sequence). It’s important to note in the figure above that systems engineers’ responsibilities tend to be broad but not deep, certainly not deep in software. Conversely, while software engineers’ responsibilities, may be broad across the software, they are deep in areas necessary to create working code. While most of the comparisons in the figure are fairly intuitive, three points require some elaboration: customer interface, non-functional requirements, and what each group does not do. Although customer interface is a systems engineering role, the software group must participate when discussing software requirements and design. For example, software engineers in Agile sprints often interface one-on-one with a customer representative to select the next development chunk, based on customer priorities.  Systems engineers are responsible for ensuring that non-functional requirements are met across the system (qualities they call "ilities" that include reliability, usability, and producibility, for example). Software engineers focus on software "quality attributes" (such as reliability, usability, and maintainability, for example.) The two concepts are clearly similar, but there are some differences; for example, software's counterpart to hardware producibility, namely creating the second and later copies, is trivial, and security and maintainability are more important. The contrast between systems and software engineers is most evident in what they do not do. The role of systems engineers is to balance concerns and perform tradeoff analyses at a high level and not get immersed in details in any area; they consistently delegate details to specialists. System engineers cannot say a task is not their job: if no one else is doing it and it needs to be done, the task defaults to systems engineers. In contrast, software engineers (and mechanical, electrical, etc. engineers) focus their attention inside a domain, mastering details and keeping up with evolving practices and technologies. Theory and Practice It is important to note that what systems and software engineers should do is not always the same as what they actually do. In fact, in our experience at the SEI, the majority of complaints each group has about the other results from a failure to meet best practices, rather than from significant differences in best practices. If the waterfall model or a Vee lifecycle model is applied as a rigid sequential process, software engineers can think systems engineering does not apply to software. If the program organizes software engineers only into integrated product teams other than the team responsible for systems engineering, integration and test, then software engineers can get frustrated with their lack of leverage. But good systems engineers also find these situations to be frustrating. Similarly, both systems and software engineers get frustrated if "agile" is used as an excuse to address only nominal-case functional requirements and not system level requirements like security and scalability.  Needed Knowledge Most of today’s systems engineers were not software engineers or computer scientists first, nor do many of them have significant background or experience with software engineering techniques, tools, and methods. What software concepts they need to know to perform systems engineering on software-reliant systems effectively is an interesting question, which I will address in a future blog post. Few universities offer a joint degree in systems and software engineering (exceptions include Cornell University and Carnegie Mellon, which offer a joint certificate in systems and software engineering).  Moreover, systems engineering programs ordinarily do not require a foundational understanding of software.  The Graduate Software Engineering (GSWE) 2009 guidelines provide recommendations for how to integrate systems engineering into software engineering education. The Systems Engineering Research Center/University Affiliated Research Center (SERC-UARC) published a Systems Engineering Body of Knowledge and Curriculum to Advance Systems Engineering (BKCASE) in 2012, which includes a graduate reference curriculum for a systems engineering master’s program, but its relationship to software is minimally discussed (in two pages).  The Road Ahead The earlier part of this blog identified the similarities and differences between the disciplines of software engineering and systems engineering.  The remainder of this blog outlines the benefits both could realize through a more collaborative approach. At this time, there are many disconnects between software and systems engineers that could be minimized through better collaboration. Software engineers could get involved in key systems architecting decisions so that the best software architecture could be easier to build. Systems engineers could reduce work that the software engineers find to be non-value added, and would receive help from software engineers in analyzing tradeoffs. In sum, a breadth+depth combination would identify innovative ideas to solve problems and exploit opportunities. To ensure that future software-reliant systems (and systems of systems) can be built effectively, a new collaborative relationship between systems engineering and software engineering must become the norm. Moreover, there is a clear and compelling need to understand which roles must be filled by which types of professionals and which actions must be performed to ensure the two disciplines work together effectively. Improving the relationship will require overcoming the negative emotions that sometimes appear. (Both groups at a talk on systems and software that I presented at INCOSE Three Rivers Chapter in July 2013 agreed that the others "are arrogant and not as smart as they think they are.") By learning to work together effectively, systems engineers and software engineers can help increase the probability of success of new and planned ultra-large-scale software-reliant systems and systems-of-systems. In particular, they must understand each other’s strengths and objectives and work together to support them.  In future posts in this blog series, I will address the questions systems engineers must ask software engineers at the earliest program phases, and vice versa. I will examine the knowledge that systems engineers should have about software and propose how to provide them with that knowledge. I will also show that typical software engineering tasks map to systems engineering tasks and fulfill much of what customers are looking for when acquiring software-reliant systems.    What do you think systems engineering should know about software engineering? What should software engineers know about systems engineering? Please leave us feedback in the comments section below. Additional Resources To read the SEI technical report, The Business Case for Systems Engineering Study: Results of the Systems Engineering Effectiveness Survey, please visithttp://resources.sei.cmu.edu/library/asset-view.cfm?assetid=34061.

By C. Aaron CoisSoftware Engineering Team Lead CERT Cyber Security Solutions Directorate  This blog post is the second in a series on DevOps To maintain a competitive edge, software organizations should be early adopters of innovation. To achieve this edge, organizations from Flickr and IBM to small tech startups are increasingly adopting an environment of deep collaboration between development and operations (DevOps) teams and technologies, which historically have been two disjointed groups responsible for information technology development. "The value of DevOps can be illustrated as an innovation and delivery lifecycle, with a continuous feedback loop to learn and respond to customer needs," Ashok Reddy writes in the technical white paper, DevOps: The IBM approach. Beyond innovation and delivery, DevOps provides a means for automating repetitive tasks within the software development lifecycle (SDLC), such as software builds, testing, and deployments, allowing them to occur more naturally and frequently throughout the SDLC. This blog post, the second in our series, presents a generalized model for automated DevOps and describes the significant potential advantages for a modern software development team.  I oversee a software engineering team that works within CERT and focuses on research and development of solutions to cybersecurity challenges. Our engineers design and implement software solutions that solve challenging problems for federal agencies, law enforcement, defense intelligence organizations, and industry by leveraging cutting-edge academic research and emerging technologies.  The environment and manner in which software engineering teams (not just my own) write code are constantly evolving. Until recently, most software engineers operated in siloed environments, with developers in one silo and an operations staff of IT professionals who maintained systems and software deployments in another. Members of the operations staff were often uninvolved in development processes, sometimes even unfamiliar with technologies used in development, yet expected to handle deployment, release testing, maintenance, and support of software applications produced by development teams.  Likewise, development teams were unable to design or develop software to be effectively released, tested, and maintained by operations teams. The problem, of course, is that they had no understanding of the needs or processes inherent to their operations teams. This divergence led to wasted time and effort, as well as heavy amounts of risk for software products and deployments. The concept of DevOps initially evolved in 2009 as an effort to remove barriers between the development and operations teams in software development. At the SEI, we are taking that concept and pushing it forward, alongside many others working in the software industry today.  As a federally-funded research and development center (FFRDC), the SEI must maintain high standards of efficiency, security, and functionality. Our team in particular develops tools and technologies to help federal agencies assess cybersecurity risks, manage secure systems, and investigate increasingly complex cyber attacks and crimes. Cybersecurity is often misunderstood or even ignored as new systems are designed and developed, falling out of view to more high profile quality requirements, such as availability or correctness of software systems. This prioritization necessarily requires addressing security after the primary design phase, or adding security controls after large portions of development have already occurred. Due to CERT’s responsibility to our sponsors and the community, security is consistently a first-tier concern, addressed as an early and fundamental requirement for any system developed by our team. In addition to being a defining factor of our software development methodology, this posture heavily influences our approach to DevOps, weaving security considerations into every facet of our software development operation. As mentioned in my introduction, forward-thinking approaches to process—including heavily automated DevOps techniques—allow us to systematically implement, maintain, and monitor status and quality standards for each of our projects. One large component of the current DevOps movement is Release Automation, which automates the build and deployment cycles of a software project. In Agile software-development scenarios, this process has two benefits:  automation of final software deployments enabling the automation of continuous incremental deployments, triggered numerous times daily as changes are made to the software system This process ensures that the entire development and management team is aware of the up-to-the-minute state of the software, including its test status and ability to be deployed to the expected runtime environment, continually throughout development. This process also enables highly confident development, and a great deal of certainty that when the time comes, the software can be successfully transitioned to an operations team for deployment and maintenance. This certainty is based on the fact that the entire test and deployment process has been automated and performed countless times already throughout the project lifecycle.  Continuous deployment means no surprises for operations staff, which translates into predictable, low-risk releases of provably high-quality software. Many players in the modern software industry share this vision. "When it comes to ensuring quality, release automation is another major asset for development teams. It automates the complex workflows required for agile software movement between development, test and production environments by removing manual interaction so humans don’t introduce avoidable errors," Ruston Vickers wrote in a March 5 blog post on Wired’s Innovation Insights blog. That same post cites a study in which more than half of respondents (52 percent) "identified the ability to simplify, standardize and execute application releases with [fewer] errors as the main benefit of release automation." In my view, release automation allows teams to perform build, deployment, and testing activities thoroughly and continually, dramatically increasing the confidence in the state of the software and all processes surrounding release and transition.  In a traditional, siloed environment, teams of developers deploy software manually, on a periodic basis (such as once per quarter, anum, or project cycle) or as necessitated by a new product release. In contrast, for a team leveraging automated DevOps, continuous integration (the process of building and testing a software project continuously, each time new changes are made) is in place and, as part of the process, the project is continuously deployed to an integration environment for testing and review. Some highly mature DevOps operations practice Continuous Deployment, an uninterrupted process that actually deploys live software to production environment.  Automating a task requires a substantial level of understanding by all involved of the processes, technologies, and complexities of deployments of the software in question. This requisite increase in understanding makes it likely that teams moving towards automated DevOps and continuous deployment capabilities also see a positive impact in the reliability, testability, security, and other quality attributes of their software. To achieve automation at this level, the team must study and deeply consider the needs of the entire project, from inception to deployment, which will result in a superior product due to increased focus on implementation details and operational realities. To achieve the level of automation described, a number of autonomous systems must be in place, such as source control, build and deployment systems, an integration environment, and other systems to channel data and communications throughout DevOps processes. As the image below illustrates, DevOps involves a number of systems operating in concert and communicating seamlessly with other systems and with humans, to detect changes, perform the necessary autonomous functions, and notify team members of status and results. These structured interactions ensure that developers, quality-assurance staff, managers, and even external stakeholders receive continuous, real-time information on the status of the project, which is hugely beneficial, especially in Agile environments. Consistent with the Agile Manifesto: more information, more often, will lead to better project outcomes. Systems of an Automated DevOps Environment The remainder of this blog post explores our generalized model for automated DevOps, identifying the systems required to support an automated DevOps process (as shown in the illustration above):  Source control. Software developers need to safely store their code and keep track of source-code history and versions. For this reason alone, source control is of critical importance. Moreover, in an automated DevOps system, the chosen version control system (VCS) (e.g., Git and Subversion etc.) becomes the system that defines when changes have been made to the software project, triggering the rest of the automated DevOps build, test, and deploy activities.  Issue tracking system. An issue tracking system allows everyone involved to track current issues, estimates, and deadlines. Communication between this system and systems for code review and source control can allow invaluable traceability between code changes and project goals.  Build system. The build system supports continuous integration by building the software, running unit and integration tests, deploying to the integration environment, and performing any other automated checks defined for new versions of the software. This system must detect changes in source-code files from the source-control system, seamlessly communicate with and push data to the integration environment, and notify team members of status at all times.  Monitoring system. Monitoring systems continuously track all autonomous systems within the DevOps environment, notifying necessary maintenance staff if a system failure occurs. This monitoring requires communication with all autonomous systems used by the DevOps process. As a manager of a software engineering team, I don’t want to spend time debugging issues in my infrastructure. I want my developers to be able to continue coding even if a server fails. If a failure occurs, I want appropriate staff notified immediately so the issue can be fixed as quickly as possible without derailing my project. Communications system. The constant exchange of information is important. Our team uses email, wikis, and a real-time chat system to enable continuous communication among all members of the project team. Moveover, communications systems are used to allow our automated systems to communicate with humans involved in the project through channels that are already part of their workflow. On my own team, I don’t have to check a dashboard to see that my latest commit broke the build; the build server will email and send me and instant message to alert me, in real time. The communications system will also alert my team lead, so everyone is in the loop and knows to fix the issue immediately so that no other developer’s work is inhibited.  Integration environment. The integration environment hosts all of the virtual machines that make up our DevOps environment. The integration environment houses machines running all of the DevOps services listed here and provides servers for continual deployment of software projects. Once a project is created, it has deployment servers, so, at any time, stakeholders can visit those servers and see the current state of working software achieved by the project team.   Code review system. To ensure software quality, every line of code must be reviewed by a seasoned developer. The practice of reviewing code also accelerates career growth and learning. Unfortunately, it is hard to find time for code review in a busy schedule. Our automated code review system therefore detects new code committed to source control and automatically assigns it to be reviewed by a senior developer. With automatic management of code reviews, we ensure that no piece of code fails to have multiple developers review it before it ends up in the final version of a software product. Documentation system. Regrettably, documentation often remains an afterthought in production software projects. To ensure that documentation is written throughout the project, when developers are most conscious of technical details, we have developed an automated system that allows developers to write documentation easily, along with source code. All our documentation is written in Markdown, a simple plain-text format that can be easily edited in an integrated development environment (IDE) and committed to source control repositories along with source code. Once the documentation files are committed, an automated system builds robust documentation artifacts in a variety of formats (such as HTML, PDF, and Microsoft Word) for consumption by project managers and stakeholders. This process operates through the build system already monitoring source code for changes and has proven comfortable for developers by not requiring them to change tools to generate comprehensive project documents.  As I see it, all the processes described above—from source control to documentation—require some system to perform them in every software project. Of course, before automated DevOps, many of these processes were performed manually, especially manual testing and deployments. Even now, generation and formatting of system documentation or management of code reviews are still manual processes in most organizations. My goal with this post was to define all tasks necessary throughout the SDLC, and to promote the automation of as many of them as possible, thus freeing up developers to do what they do best: design and write amazing code.  Looking Ahead While this post presented a generalized model for DevOps practices, future posts in this series will present the following topics: advanced DevOps automation DevOps system integration continuous integration continuous deployment automated software deployment environment configuration We welcome your feedback on this series, and what DevOps topics would be of interest to you. Please leave feedback in the comments section below. Additional Resources To listen to the podcast, DevOps—Transform Development and Operations for Fast, Secure Deployments, featuring Gene Kim and Julia Allen, please visit http://url.sei.cmu.edu/js. To view the August 2011 edition of the Cutter IT Journal, which was dedicated to DevOps, please visit http://www.cutter.com/promotions/itj1108/itj1108.pdf. Additional resources include the following sites: http://devops.com/    http://dev2ops.org/ http://devopscafe.org/ http://www.evolven.com/blog/devops-developments.html http://www.ibm.com/developerworks/library/d-develop-reliable-software-devops/index.html?ca=dat-