By Will DormannSenior Member of the Technical StaffCERT Occasionally this blog will highlight different posts from the SEI blogosphere. Today’s post by Will Dormann, a senior member of the technical staff in the SEI’s CERT Program, is from the CERT/CC (Coordination Center) blog. This post explores Dormann’s investigation into the state of signed Java applet security.

By Joseph ElmSenior Member of the Technical Staff Building a complex weapon system in today’s environment may involve many subsystems—propulsion, hydraulics, power, controls, radar, structures, navigation, computers, and communications.  Design of these systems requires the expertise of engineers in particular disciplines, including mechanical engineering, electrical engineering, software engineering, metallurgical engineering, and many others. But some activities of system development are interdisciplinary, including requirements development, trade studies, and architecture design, to name a few.  These tasks do not fit neatly into the traditional engineering disciplines, and require the attention of engineering staff with broader skills and backgrounds.  This need for breadth and experience is often met by systems engineers. Unfortunately, system engineering is often not valued among all stakeholders in the Department of Defense (DoD), and is often the first group of activities to be eliminated when a program is faced with budget constraints.  This blog post highlights recent research aimed at demonstrating the value of systems engineering to program managers in the DoD and elsewhere. In 2004, the Director for Systems Engineering in the Office of the Undersecretary for Defense for Acquisition, Technology and Logistics (OUSD(AT&L)) came to the National Defense Industrial Association (NDIA) and voiced concerns that DoD acquisition programs were not capitalizing on the value of systems engineering (SE). He knew the value of SE and knew that it could help DoD programs, but he also knew that not all DoD program managers shared his convictions. Consequently program managers were taking shortcuts and eliminating SE capabilities from their programs.  He came to NDIA seeking quantitative evidence of the value of SE. Subsequently, others have recognized this same problem.  A recent Government Accountability Office (GAO) report indicates that acquisition program costs are typically 26 percent over budget and development costs are typically 40 percent more than initial estimates. These programs routinely fail to deliver the capabilities when promised, experiencing, on average, a 21 month delay.  The report finds that "optimistic assumptions about system requirements, technology, and design maturity play a large part in these failures, and that these optimistic assumptions are largely the result of a lack of disciplined SE analysis early in the program." Despite findings such as this, many programs still fail to deploy good SE.  Why?  It may be because there is relatively little quantitative evidence of the impact and value of SE.  Everyone can see the SE costs, such as the labor applied and the time allocated in the schedule.  The benefits of SE, however, may be less identifiable.  They often manifest themselves as risks that didn’t materialize rework that didn’t need to be done customer complaints that didn’t occur, and product deficiencies that are circumvented Because these benefits are hard to quantify, however, the return from investment in SE is often unrecognized.  To get a true picture of the value of SE, we need to quantitatively measure its impact on acquisition program performance. The remainder of this blog posting describes a research effort that the SEI undertook in partnership with the NDIA and the IEEE Aerospace and Electronic Systems Society (IEEE AESS).  This effort provided quantitative evidence of the value of SE in terms of its impact on program cost, program schedule, and program technical performance - impacts that are crucially important to program managers and executives. Building on Previous Research in Systems Engineering While "software engineering" is etched in stone at the SEI’s Pittsburgh headquarters, it is sometimes hard to draw a clear line between software and the systems supported by software.  For that reason, SEI staff have often conducted research in the SE realm. For example, the Capability Maturity Model Integration Framework addresses processes that apply equally to software development and system development. Architecture development and evaluation methods developed for software are routinely adapted and applied to systems. Through the SEI’s affiliation with NDIA, my fellow researcher, Dennis Goldenson and I became involved in developing their response to the 2004 inquiry from OUSD (AT&L) mentioned earlier.  In 2005, I suggested conducting a survey of acquisition programs to gather information about their activities related to SE, and how those programs performed.  We could then identify relationships between these factors.  We conducted the survey in 2006 and published our results in 2007. Our initial research demonstrated that those programs that deployed more systems engineering performed better against measures of cost, schedule, and technical performance.  In 2010, the DoD approached NDIA and the SEI about strengthening the business case for SE by expanding the survey population to include not just the NDIA but other professional organizations including the IEEE-AESS, and the International Council on Systems Engineering (INCOSE). For this latest study, we surveyed individual programs in participating organizations to obtain answers to the following questions: What systems engineering activities do you perform on your program?  How well does your program perform? We surveyed 148 different programs.  Although most programs were supplying systems for the U.S. defense sector, we also received a few responses from organizations serving other market sectors and operating in different countries. An Even Stronger Link Between SE and Performance Our latest results, published in the SEI technical report, The Business Case for Systems Engineering Study:  Results of the Systems Engineering Effectiveness Study, identified strong links between the performance of systems engineering tasks and overall program performance. These results provide a convincing case for the value of systems engineering. This latest study collected information from participating programs along three dimensions: systems engineering deployment. We assessed SE deployment by examining both the presence and the quality of work products resulting from SE activities.  These work products were selected from those listed in the CMMI Framework by a panel of SE experts.  Based on this assessment, SE deployment for each program was categorized as either low, medium, or high. program performance. We assessed program performance as a combination of cost performance (satisfaction of budget), schedule performance, and technical performance (satisfaction of requirements).  Again, based on this assessment, program performance for each program was categorized as either low, medium, or high. program challenge.  Some programs are inherently more challenging than others due to factors such as size, duration, technology maturity, interoperability requirements, etc.  Based on the combination of these factors, program challenge was categorized as either low, medium, or high. We then looked for relationships between these metrics. We found a very strong relationship between SE deployment and program performance.  In particular, as programs deployed more SE, they delivered better performance.  For example, among those programs deploying the least SE, only 15 percent delivered the highest level of program performance.  Among those deploying the most SE, however, 56 percent delivered the highest level of program performance. As one would expect, our research showed an inverse relationship between program challenge and program performance.  But, we also learned that SE practices became even more valuable when used with these challenging programs. We already noted that the number of programs delivering high program performance increased from 15 percent to 56 percent as SE deployment increased.  For the most challenging programs, however, the number of programs delivering high program performance increased from 8 percent to 62 percent with increased SE deployment.  This result clearly shows the increasing need for SE as programs become more challenging. As mentioned above, we measured SE deployment by assessing SE-related work products for each program.  Now, we could group these artifacts into process areas such as requirements development and management program planning product architecture trade studies product integration verification validation program monitoring and control risk management configuration management integrated product teams Grouping artifacts into process areas enabled us to probe more deeply into the relationships between SE and program performance, identifying not only the overall benefit of SE but also the benefit of specific SE processes. For each program, we assessed SE deployment in each of the 11 process areas above and analyzed the relationship to program performance.  Here again, we found strong supporting relationships for all SE process areas - increased SE deployment in any of these areas contributed to better program performance. Relationships to program performance, however, were stronger in some than in others. Particularly strong relationships to program performance were found for program planning. The number of programs delivering highest performance increased from 13 percent to 50 percent as SE activities related to program planning increased. requirements development and management. The number of programs delivering highest performance increased from 21 percent to 58 percent as SE activities related to requirements development and management increased. verification. The number of programs delivering highest performance increased from 16 percent to 54 percent as SE activities related to verification increased. product architecture. The number of programs delivering highest performance increased from 16 percent to 49 percent as SE activities related to product architecture increased. As strong as these relationships were, we found that they grew even stronger for the more challenging programs. Transitioning to the Public The results of our research can be used in a number of ways by system developers, system acquirers, and academia. For example, our findings constitute a baseline of SE deployment across the industries surveyed.  System developers can use our methods to assess their own SE capabilities, compare them to this baseline, and identify their strengths and weaknesses.  They can then develop process improvement plans to improve their weaknesses and strategies to leverage their strengths.  We continue to work with defense contractors who are applying this process to improve their SE capabilities. System acquirers can also benefit from these findings.  A Program Management Office (PMO) acquiring a system needs to deploy good SE practices in planning the program, defining system requirements, developing system architectures, etc.  The PMO also needs to ensure that the system supplier deploys good SE practices.  For example, the PMO must first include in the solicitation a definition of SE activities that they expect from the supplier.  They should evaluate the supplier’s response to these expectations as a factor in the selection of the supplier.  They should also ensure that the SE expectations are included in the contract.  They should monitor the supplier’s performance throughout execution to ensure that SE expectations are being met. The academic community can also use the results from our study.  For example, several universities offering systems engineering programs at the master’s level are using this information in their curriculum and their courses to show their students the value of systems engineering and to direct some of their courses to capitalize on the knowledge that we’ve gathered here.  INCOSE is also incorporating the results of the survey into its systems engineering handbook. Additional Resources The technical report The Business Case for Systems Engineering may be downloaded at http://www.sei.cmu.edu/library/abstracts/reports/12sr009.cfm Organizations interested in the continuing effort of showing the value of systems engineering may join INCOSE’s System Engineering Effectiveness Working Group athttp://www.incose.org/practice/techactivities/wg/seewg/ Organizations interested in joining NDIA’s Software Engineering Effectiveness Committee athttp://www.ndia.org/Divisions/Divisions/SystemsEngineering/Pages/Systems_Engineering_Effectiveness_Committee.aspx

By William Anderson Senior ResearcherSoftware Solutions Division The ubiquity of mobile devices provides new opportunities to warn people of emergencies and imminent threats using location-aware technologies. The Wireless Emergency Alerts (WEA) system, formerly known as the Commercial Mobile Alert Service (CMAS), is the newest addition to the Federal Emergency Management Agency (FEMA) Integrated Public Alert and Warning System (IPAWS), which allows authorities to broadcast emergency alerts to cell phone customers with WEA-enabled devices in an area affected by a disaster or a major emergency. This blog posting describes how the Software Engineering Institute's (SEI) work on architecture, integration, network security, and project management is assisting in implementing the WEA system, so it can handle a large number of alert originators and provide an effective nationwide wireless emergency warning system. Why We Need the WEA System In the summer of 2012, the western United States experienced unprecedented wildfires costing several lives and hundreds of millions of dollars of property damage. The Lower North Fork fire in Colorado caused three fatalities and could have been worse due to massive failures in the landline-based telephone alerting systems. A local television station reported that incorrect records in a geographic database were responsible for many of the failures. The Denver Post report on the 2012 Waldo Canyon fire near Colorado Springs (the most destructive fire in Colorado history) revealed similar problems in the emergency alert system. An additional complication to the alert process lies in the fact that more than 35 percent of U.S. homes do not have a land-based phone line. Given the mobile nature of today’s society, a better solution is needed to warn people of imminent threats using geographic data. Pursuant to the Warning, Alert, and Response Network (WARN) Act, FEMA's IPAWS, and the Department of Homeland Security’s Science & Technology (DHS S&T) Directorate have partnered with the Federal Communications Commission and commercial mobile service providers to develop the WEA system to bring emergency alert messages to mobile devices configured to allow wireless emergency alerts. This effort is the latest in a series of alert and warning systems that date back to 1951 when President Harry Truman established the Control of Electromagnetic Radiation (CONELRAD) program. The WEA system will enable alert originators—including federal, state, tribal, and territorial governments as well as local emergency response authorities that provide fire protection, transit, and public utilities—with the potential to reach the more than 325 million wireless subscribers. The four largest cellular carriers—AT&T, Sprint Nextel, T-Mobile, and Verizon, which connect with 90 percent of all cell phones in the United States—have agreed to participate along with another 72 smaller carriers (see FCC Master CMAS (prior designation for WEA) Registry for reported numbers). The SEI has two major tasks in this effort: developing an integration strategy to help assimilate alert originators into the system and identifying an initial set of best practices that apply to WEA use. The SEI is just one of several organizations involved in the WEA system project. The John Hopkins Applied Physics Laboratory is constructing a large-scale computer model of the system on which to run simulation studies and an environment to test the interfaces and system. The RAND Corporation developed a mobile-penetration strategy to determine the availability and acceptability of mobile alerting services across the country. The MITRE Corporation is working on test plan development. The WEA system went online nationally in April 2012. This activation meant that the backbone infrastructure is up and running, providing alerting authorities with the appropriate alert originating software to send messages to WEA system-enabled handsets. The hard task of expanding the number of participating originators and mobile carriers can now commence.   When an emergency responder or alerting agency, such as the National Weather Service (which went online June 28, 2012), or the National Center for Missing and Exploited Children (NCMEC) (which issued its first AMBER alert through WEA on December 18, 2012) becomes aware of an event that the public should know about, it can use the WEA system to initiate a geographically targeted alert message and identify smart phone users in the geographic area who should receive the message. The WEA system portion of the Common Alerting Protocol (CAP) specifies a text message of up to 90 characters detailing the topic of the alert, the area affected, the alert expiration, the prescribed action, and the alert sender. Most modern phones (e.g., 3G, 4G, or LTE) are already equipped with the technology to receive WEA messages. Cell phone users are automatically subscribed to the service; they do not have to opt in to receive alerts. They may opt out of all but the presidential alerts. The WEA system sends out three types of messages presidential alerts AMBER (America’s Missing: Broadcast Emergency Response) alerts imminent threats to safety or life The WEA system also uses Federal Information Processing Standard (FIPS) codes, which are alphanumeric codes to target cell phones in geographic areas at the county level. The WEA system operates using cell broadcast, a one-to-many technology rather than point-to-point (phone number addressed) text messages. The WEA system messages are transmitted on a separate channel from the voice and data communication links. As a result, authorities sending out emergency alerts can do so even when voice and data channels are saturated. Leveraging Our Architecture Analysis Expertise Along with Lawrence Jones, project lead for the SEI’s work on the WEA effort, and our colleagues in the Software Solutions and CERT divisions at the SEI, we have been leveraging our architecture and system-of-systems analysis techniques to help ensure the smooth national deployment and operation of the WEA system. Specifically, we are using the Mission Thread Workshop (MTW) approach that has been used by the Department of Defense in large systems-of-systems analysis to describe the step-by-step sequences required to execute a given mission. We are applying the MTW approach to the WEA system to develop mission thread descriptions of typical integration challenges. To analyze the WEA system as a socio-technical system-of-systems, we developed mission threads to describe operational scenarios that the emergency response community might encounter that would benefit from the WEA system alerting capabilities. Table top walk-throughs of the mission threads with emergency management experts reveal underlying quality attributes (QAs) that the system must have to perform properly. These underlying QAs can include normal system QAs, such as resilience, reliability, modifiability, securability, and sustainability, as well as QAs that are particularly relevant to the WEA system, such as accuracy, urgency, timeliness, and relevance. Addressing Security Concerns A key quality attribute of interest for the WEA system is security. Like other cyber-enabled services, the WEA system is subject to cyber threats that may prevent its use or damage the credibility of the service it provides. Attackers may attempt to delay, destroy, or modify alerts, or even insert false alerts, actions that may pose a significant risk to the public. Accordingly, a team of experts from the SEI's CERT division is examining the security aspects of the WEA system using the MTW approach. By reviewing the step-by-step sequence documented in mission threads, security experts are identifying threats and potential vulnerabilities that may make the WEA system susceptible to cyber-attacks. To address these concerns, the SEI is developing a cybersecurity risk management strategy for the WEA system that alert originators can use to identify, prioritize, develop, and implement cybersecurity risk mitigation plans.   Best Practices Work Although WEA adoption is still in the early stages, the SEI is leading a best practices task that supports the DHS effort. This work includes a lessons learned report on the New York City WEA demonstration conducted in December 2011 creation and analysis of a WEA trust model that incorporates inputs from the emergency management organization community and a sample of the public. This study identifies key factors involved in trust in WEA messages to increase understanding of the effectiveness of these alerts selected example best practices Addressing Challenges of Scale A key challenge to deploying and operating the WEA system effectively is dealing with issues of scale. It’s estimated there are between 10,000 and 40,000 alert originators who may initiate the WEA system messages including designated federal, state, territorial, tribal, and local authorities (the WEA system alert authorization is a state-driven organizing model that is expected to mimic the hierarchical structure of emergency and disaster response networks). The SEI is developing integration strategies to help those originators to bring the WEA system capability to their constituents. Many of these originators will rely on vendors to provide solutions. The vendors build the systems that help emergency response operators prepare alerts for dissemination. The human-machine interfaces vary but generally support common features, such as a text-based interface, checklist prompts, predefined templates for message support, and map-based interfaces for geographic targeting assistance. Vendors also ensure that the operators have the needed message control fields, such as What is the severity? What is the urgency? When should the message be taken down? Originators face many scale-related challenges as they attempt to implement the WEA system: Many alert originators employ different types of software and have acquired that software from multiple vendors or, even, custom built it themselves. The state of readiness to integrate with the WEA system varies from originator to originator. The use of protocols varies from one originator to another. The WEA system standard is the Common Alerting Protocol which, among other things, has a 90-character message length limit (for the WEA system), which is even shorter than a Tweet! Originators feel that this limit impedes their ability to aptly describe a situation to users, especially for AMBER alerts. The WEA system is often integrated or supplemented with other existing emergency alert system channels, including television, radio, and social media. Related DHS Science and Technology Directorate research projects involve improving the granularity of the geo-location capability (some carriers are already committed to a finer granularity than is required) and addressing the challenges associated with citizen reaction/non-reaction to the alerts. In many cases alerting authorities want to narrow their message to a geographic area smaller than the county level. They are also concerned that those who receive messages react promptly and correctly. What’s Ahead The SEI team working on the WEA system project is consolidating the results of the data collection phase of our project interviewing originators and the vendor community that supports the originators. Questions include What kind of systems do the originators use? How do the originators adopt new systems? Do they have system documentation? What problems have they encountered using the systems? The results of our work will be delivered to DHS this summer with public release anticipated sometime in the fall. We would welcome your feedback on our work. Please share your insights in the comments section below. Additional Resources For more information about the Mission Thread Workshop, please visithttp://www.sei.cmu.edu/architecture/tools/establish/missionthread.cfm For more information about the the Wireless Emergency Alerts (WEA) Research, Development, Testing, and Evaluation (RDT&E) program, formerly known as the Commercial Mobile Alert Service (CMAS) RDT&E program, please visit  http://www.fema.gov/wireless-emergency-alerts

By Peter Feiler Senior Member of the Technical StaffSoftware Solutions Division Aircraft and other safety-critical systems increasingly rely on software to provide their functionality. The exponential growth of software in safety-critical systems has pushed the cost for building aircraft to the limit of affordability. Given this increase, the current practice of build-then-test is no longer feasible. This blog posting describes recent work at the SEI to improve the quality of software-reliant systems through an approach known as the Reliability Validation and Improvement Framework that will lead to early defect discovery and incremental end-to-end validation. A Fresh Look at Engineering Reliable Systems Studies by the National Institute of Standards and Technology and the National Aeronautics and Space Administration show that 70 percent of software defects are introduced during the requirements and architecture design phases. Moreover, 80 percent of those defects are not discovered until system integration test or later in the development cycle. In their paper "Software Reduction Top 10 List" researchers Barry Boehm and Victor Basili wrote that "finding and fixing a software problem is 100 times more expensive than finding and fixing it during the requirements and design phase." Aircraft industry and International Council on Systems Engineering data shows a rework cost multiplier of 300 to 1,000. Much of this defect leakage and the resulting cost penalty are due to incomplete and ambiguous requirements and mismatched assumptions in the interaction between the components of embedded software system architectures.  To address these problems, I have developed—together with fellow SEI researchers John Goodenough, Arie Gurfinkel, Charles Weinstock, and Lutz Wrage—the Reliability Validation and Improvement Framework, which takes a fresh look at engineering reliable systems. Reliability engineering has its roots in engineering physical systems. The focus of reliability engineering has been on using historical data to predict mechanical failures in physical parts, assuming that design errors have little impact due to slowly evolving designs. Defects in software systems, however, are design errors for which reliability predictions based on historical data have been a challenge. At the same time, embedded software has become the key to system integration and thus the driver for system reliability. We needed a new perspective on reliability because it is unrealistic to assume that software has zero defects. Our work on developing a framework began when we were approached by the U.S. Army Aviation and Mission Research Development and Engineering Center (AMRDEC) Aviation Engineering Directorate (AED), the agency responsible for signing off on the flight worthiness of Army rotorcraft. AMRDEC wanted a more reliable approach than "testing until time and budgets are exhausted" to qualify increasingly software-reliant, safety-critical systems. Four Pillars for Improving the Quality of Safety-Critical Software-Reliant Systems We needed a new approach for making software-reliant systems more reliable, one that allows us to discover problems as early as possible in the development life cycle. For safety-critical systems these are not only defects in functional design but also problems meeting operational quality attributes, such as performance, timing, safety, reliability, and security. Our approach needed to identify not only defects before a system is built, but also issues that are hard to test for. In addition, the approach needed to ensure that unavoidable failures are addressed through fault management and that unhandled faults are identified, providing resilience to counter unplanned usage and unexpected conditions. The Reliability Validation and Improvement Framework that we developed incorporates four engineering technologies to address the challenges outlined above: formalization of mission and safety-critical requirements at the system and software level. Requirements on a system—the first pillar of our framework—are typically determined by business needs and operational use scenarios. They are often developed by system engineers and may evolve over time.  As Boehm points out in his 2006 paper, "Some Future Trends and Implications for Systems and Software Engineering Processes," there is a gap in translating, especially non-functional system requirements, into requirements for embedded software. Individual software units are then developed without this contextual knowledge, making assumptions about the physical system, computer hardware platform, and the interaction with other software tasks, each of which can affect functional and non-functional system properties. Our framework focuses on capturing the "shalls" of a system by specifying mission capabilities under normal conditions (mission requirements such as functionality, behavior, and performance), as well as the "shall nots," by specifying how the system is expected to perform when things go wrong (dependability requirements, such as safety, reliability, and security). Requirements are associated with the system in the context of assumptions about the operational environment and decomposed into requirements of different components of the system. This approach allows the validation and verification of requirements and assumptions in the correct context. We have identified an excellent resource, the Requirements Engineering Management Handbook developed by a formal methods group at Rockwell Collins for the Federal Aviation Administration (FAA). The handbook defines an 11-step process for capturing requirements in a more systematic and formal way using an explicit architecture model as context.  This approach allows for completeness and consistency checking of requirement specifications.  architecture-centric, model-based engineering. The aircraft industry has recognized that software-reliant system development must take an architecture-centric, model-based, analytical approach to address the limitations of conventional build-then-test practices. The industry has embraced virtual system integration to achieve validation through static analysis of integrated architecture and detailed design models. This approach leads to early discovery of software-induced problems, such as timing related loss of data, control instability due to workload sensitive latency jitter, and loss of physical redundancy due to load balancing of partitions in a networked systems. This second pillar in our framework uses analyzable architecture models combined with detailed design models and implementations to evolve and validate a system incrementally. The OMG SysML architecture modeling notation is gaining popularity in the system engineering community. For embedded software systems, SysML is complemented by SAE International’s  Architectural Analysis and Design Language (AADL), a notation I authored that provides a set of concepts with well-defined semantics for representing embedded software system architectures, which include the software task and communication architecture, its deployment on a networked computer platform architecture, and its interaction with the physical system. These semantics lead to precise specification of execution and interaction behavior and timing. A standardized error model extension to AADL supports identification of safety hazards, fault impact analysis, and specification of fault management strategies to help meet reliability and availability requirements. In this architecture-centric virtual integration approach, the annotated architecture model drives analysis of different functional and non-functional properties by generating analytical models and thereby consistently propagating changes into each analysis, as well as to implementations generated from validated models. The virtual integration approach uses a multi-notation model repository that utilizes standardized model interchange formats and maintains consistency across models, while allowing suppliers and system integrators to utilize their own tool chains. static analysis of functional and non-functional system properties. Static analysis is any technique that formally proves system properties from system models prior to deployment and execution. Static analysis complements simulation and traditional testing to validate and verify a system. This third pillar of our framework focuses on incrementally validating and verifying the virtually integrated system before the actual production software is written. Formal analysis frameworks, such as model checking for verifying functional behavior, and rate monotonic schedulability analysis for assuring timing requirements, are scalable solutions that have been successfully applied to avionics and space systems. Properties defined in the SAE AADL standard and annotations using the standardized Behavior, Error Model, and ARINC653 Partitioned Architecture extensions to AADL support different functional and non-functional analyses from the same model. system and software assurance. Safety cases using a goal-structured notation have been used extensively outside the United States to assure safety in nuclear reactors, railroad signaling systems, avionics systems, and other critical systems.  A best practice of this fourth pillar of our framework involves the development of evidence in parallel with the system design throughout the development life cycle. This evidence ranges from requirements and design review results and predictive analysis results of virtually integrated systems, to test results to provide justified confidence. This approach records claims about the system, assumptions made in the process, and evidence required to satisfy the claims. The SEI has been involved in evolving this approach into assurance cases for different system concerns. Assurance cases provide a systematic way of establishing confidence in the qualification of a system and its software by recording and tracking the evidence and arguments (as well as context and assumptions) that the claims of meeting system requirements are satisfied by the system design and implementation, and making the argument that the evidence is sufficient to provide justified confidence in the qualification. Recent developments indicate that assurance cases are being adopted within the United States. For example, the U.S. Food and Drug Administration is exploring the inclusion of assurance case reports into regulatory submissions for infusion pumps. An International Standards Organization for assurance cases is currently under development. Overview of the Reliability Validation and Improvement Framework In the figure above, the four components combine to transform the traditional software development model of a heavily document-driven practice into an analytical practice based on architecture-centric models. As we write in our SEI technical report, the centerpiece of this framework is an architecture-centric model repository supporting multiple modeling notations and a combination of architecture and detailed design models together with source code implementations. The ability to virtually integrate and analyze the models is key to improving reliability by discovering problems early in the life cycle and reducing defect rework at a higher cost in later phases. Since incomplete, ambiguous, and inconsistent requirements contribute 35 percent of system-level defects, it is valuable to formalize requirements to a level that can be validated and verified by static analysis tools. Formalization of requirements establishes a level of confidence by assuring consistency of the specifications and their decomposition into subsystem requirements. The requirements are decomposed in the context of an architecture specification and refined into concrete, formalized specifications for which evidence can be provided through verification and validation activities. This method is reflected in the Requirements Engineering Management Handbook process and supported by the draft Requirement Definition and Analysis Standard extension that is part of the AADL standard suite and applicable to other architecture notations. The aerospace industry, which already has several well-entrenched safety practices, has embraced System Architecture Virtual Integration (SAVI), which a multi-year initiative conducted under the auspices of the Aerospace Vehicle Systems Institute (AVSI). SAVI aims to improve current practice and overcome the software cost explosion in aircraft, which currently makes up 65 percent to 80 percent of the total system cost with rework accounting for more than half of that. SAVI has chosen SAE AADL as a key technology and performed several proof-of-concept demonstrations of the virtual integration concept to achieve early discovery of problems. SAVI’s current work focuses on demonstrating model-based, end-to-end validation and verification of safety and reliability, as well as defining model repository requirements to facilitate commercial tool vendor adoption.  SAVI also has an ongoing effort to assess return on investment and necessary changes to acquisition and qualification processes. The application of static analysis to requirements, architecture specifications, detailed designs, and implementations leads to an end-to-end validation and verification approach. The research community in the United States and Europe has embraced AADL models as a platform for integrating formal analysis frameworks and transitioning them quickly to industrial settings. An assurance case interchange format is being standardized and evidence tracking in the context of formalized requirements is supported through the Requirement Definition and Analysis extension to AADL. As illustrated in the figure above, our revised software development model is represented by two parallel activities: The first activity reflects the development process: build the system. This development process focuses on the creation of design artifacts ranging from requirement specification and architecture design, detailed design, and code development through integration, target, and deployment build. The second activity reflects the qualification process: build the assurance case for the system. This qualification process comprises the traditional unit test, integration test, system test, and acceptance test phases.  It extends into the early phases of development through the concept of an architecture-centric virtual system integration lab to support validation and verification throughout the life cycle. Conclusion At the SEI we continue to provide technical leadership to the SAE AADL standards committee, participate in the SAVI initiative, codify elements of the four pillars described in this post into various methods (such as the Virtual Upgrade Validation and assurance cases). We also work with organizations such as the U.S. Army, NASA, FDA, and various industrial partners to apply different aspects of these four pillars in actual programs and projects. We welcome your feedback on our work and are looking for organizations interested in applying this approach. If you are interested, please leave a comment below or send an email to info@sei.cmu.edu. Additional Resources To read the full SEI technical report, The Reliability Validation and Improvement Framework, please visit http://www.sei.cmu.edu/library/abstracts/reports/12sr013.cfm To read the white paper describing this work, Four Pillars for Improving the Quality of Safety-Critical Software-Reliant Systems, please visithttp://www.sei.cmu.edu/library/abstracts/whitepapers/FourPillarsSWReliability.cfm

By Soumya Simanta Senior Member of the Technical Staff Software Solutions Division Warfighters in a tactical environment face many constraints on computational resources, such as the computing power, memory, bandwidth, and battery power. They often have to make rapid decisions in hostile environments. Many warfighters can access situational awareness data feeds on their smartphones to make critical decisions. To access these feeds, however, warfighters must contend with an overwhelming amount of information from multiple, fragmented data sources that cannot be easily combined on a small smartphone screen. The same resource constraints apply to emergency responders involved in search-and-rescue missions, who often must coordinate their efforts with multiple responders. This posting describes our efforts to create the Edge Mission-Oriented Tactical App Generator (eMontage), a software prototype that allows warfighters and first responders to rapidly integrate geotagged situational awareness data from multiple remote data sources. An Abundance of Siloed Data While the internet has made data more accessible than ever, the information is often siloed in different systems. To complicate matters, it’s not easy for warfighters to share and tailor data in the field. Typically, data filters are predefined at development time and are often not flexible enough to meet the range of unancticipated requiremnts that warfighters and first responders may have in the field. The modifications required to support newer capabilities is often done offsite and, therefore, takes longer to reach the hands of users.  These delays are problematic for warfighters, who need the ability to configure data sources as quickly as possible. Currently, warfighters need multiple systems to view different types of data including historical information, such as the occupants of a certain address real-time information including geo-location data that reports the location of team members (often referred to as "blue force tracking") alerts including information (such as severe weather alerts) from federal agencies Using a system that we are prototyping here at the SEI, a warfighter would be able to mash up data from these sources, as well as public data sources (such as Google Places, Twitter, or FourSquare) and/or private Department of Defense (DoD) data sources. More Control to the End User Our research at the SEI is focused on giving more control to the end users, in this case warfighters and emergency responders. In particular, we provide them the ability to define data filters at runtime without writing code and mash up data from different sources, so that they can access the information that they need, when they need it, on their device. To enable this capability, I am working with a group of SEI researchers, including Gene Cahill, on the development of eMontage, a Java-based client/server application that allows users to create data filters on Android smartphones. We built eMontage using commercial off-the-shelf components, as well as public and private data sources. eMontage allows users to mash geotagged data from private and public information sources and combine relevant data on one application and screen. The eMontage architecture allows for the addition of both public and private data sources in a relatively short amount of time.  In this context, "short" translates to a few hours to a week, depending on the data format and application programming interface (API) support provided by the remote data source or web service. We are collaborating with Dr. Brad Myers and his student, Kerry Chang of Carnegie Mellon University’s Human Computer Interaction Institute. Dr. Myers has collaborated with us on previous initiatives in this arena.  They are working to help us define an appropriate interface for soldiers to use the handheld software in the challenging situations they face.  We are also collaborating on a parallel effort that deals with an interface for rapid entry of unstructured information into a smartphone. In the implemented prototype, known as Listpad, the user enters unstructured text as he or she would on a notepad, but the information is saved in a structured format. Through contextual clues, Listpad autocompletes data entry and also allows for automatic data-type detection. Testing Our Prototypes and Approach In 2012, we tested our prototypes at Camp Roberts in the United States Special Operation Command Naval Postgraduate School (NPS) Field Experimentation Cooperative Tactical Network Testbed. Our effort was aimed at determining whether warfighters could successfully use our applications determining the learning curve involved soliciting feedback on the user interface gathering additional use cases and features identifying additional data sources and formats Warfighters testing the prototype provided us with positive feedback, as well as a list of suggested features for future prototypes. In February 2013, we demonstrated the capabilities of eMontage at the Camp Roberts’ NPS's Joint Interagency Field Exploration (JIFX). This event explored the potential of emerging capabilities to address challenges faced by warfighters and first responders. Our demonstration pulled geotagged data for Hurricane Sandy from the Wireless Emergency Alert System, Twitter, Flickr, and Foursquare. We created mashups of the data that blended eyewitness information, photographs, and other information about affected areas in realtime during the hurricane. At that event, we received vital feedback from first responders, including the Federal Emergency Management Agency and the Department of Homeland Security. Future Work Our initial aim with eMontage was to develop a capability that would allow warfighters and emergency responders on the tactical edge to easily and quickly integrate tactical information sources on their smartphones.  We are working on improving the performance and security of eMontage by adding a caching mechanism and standard security mechanisms. We are also leveraging the eMontage architecture to design a prototype that will analyze streaming information (such as a Twitter stream or user location stream) at the edge in a project called Edge Analytics. Dr. Myers and Ms. Chang are also working to partially automate the "adding a new data source" to the eMontage workflow. We also plan to explore the integration of these research projects by running the eMontage server on a cloudlet and in conjunction with group context-aware mobile applications to guide the data filtering process. We welcome your feedback on our work. Please leave feedback in the comments section below. Additional Resources For more information about the SEI’s work in pervasive mobile computing, please visithttp://www.sei.cmu.edu/mobilecomputing/ To download a pdf describing other work by SEI researchers in advanced mobile systems, please visithttp://www.sei.cmu.edu/library/assets/brochures/Infosheet_RTSS_AMS_Jan13.pdf To view the presentation, eMontage: An Architecture for Rapid Integration of Situational Awareness Data at the Edge, which I presented with Ed Morris and Gene Cahill at SATURN 2013, please visithttp://www.sei.cmu.edu/library/assets/presentations/simanta-saturn2013.pdf? To view the presentation, Listpad: Creating Customized Structured Data on Mobile Devices,  by Kerry S. Chang, Brad A. Myers, Gene M. Cahill, Soumya Simanta, Edwin Morris, and Grace Lewis, please visithttp://www.cs.cmu.edu/~shihpinc/pdf/Listpad_workshop.pdf

By Kevin FallChief Technology Officer I recently joined the Carnegie Mellon Software Engineering Institute (SEI) as deputy director and chief technology officer (CTO). My goal in this new role is to help the SEI advance computer science, software engineering, cybersecurity, and related disciplines to help ensure that the acquisition, development, and operation of software-dependent systems have lower cost, higher quality, and better security. I have spent the past two decades conducting a range of research and development activities, and I have served on various Department of Defense (DoD) advisory boards. In this blog posting, I’d like to talk a little bit about my background and outline the priorities I’m pursuing at the SEI. In subsequent blog postings, I’ll describe the SEI technical strategy in more detail. My Introduction to Software I became interested in programming and reverse engineering (analyzing copy protection schemes) while in high school in southern California, where I learned to program in 6502 assembly language and BASIC on an Apple ][. This sparked my interest in software, especially operating systems. My first summer job was writing software, primarily a code de-obfuscation system, for a small training company in Manhattan Beach, California. During my undergraduate studies in computer science at the University of California, Berkeley (UC Berkeley), I spent two summers as an intern at TRW (now Northrop Grumman) in Redondo Beach where I was introduced to Berkeley UNIX (the Berkeley Software Distribution or BSD) and the C programming language. Ultimately, I worked on BSD UNIX itself, where I developed TCP/IP software targeted for deployment on the Cray X/MP and integrated Massachusetts Institute of Technology’s (MIT) Kerberos Authentication system (now the basis for authentication in Windows environments) into the BSD authentication framework. I also authored the BSD version of the cat program, perhaps the most widely used basic program in the system. While I was working at Berkeley integrating Kerberos into BSD, Robert Morris’ worm hit the Internet. Using some of the reverse-engineering skills I had acquired earlier, I helped in de-obfuscating the binary code comprising the worm. Coincidentally, this incident would be the primary motivator for the Defense Advanced Research Projects Agency (DARPA) to sponsor the creation of the SEI’s CERT Division. During the summer, after I left UC Berkeley, I joined MIT’s Project Athena doing further work on Kerberos prior to joining the PhD program in computer science and engineering at UC San Diego (UCSD). My research at UCSD focused on operating system support for multimedia. While in graduate school at UCSD, I consulted for the San Diego Supercomputer Center (then run by General Atomics) and for the Center for Communications Research (one of the Institute for Defense Analysis (IDA's) federally funded research and development centers (FFRDCs). Early Professional Career After graduating with a PhD from UCSD, I was hired to work with the Network Research Group at Lawrence Berkeley National Laboratory, which produced the tcpdump and pcap tools, numerous other TCP/IP improvements, and the NS2 network simulator, for which I served as chief architect. In this role, I added a novel emulation capability, enabling the intermixing of real-world network traffic with simulated traffic. In addition, I worked with Sally Floyd to produce a number of detailed analyses of TCP’s congestion control algorithms. My involvement with internet technologies in the late 1990’s afforded me the opportunity to work in a startup company, NetBoost, that built hardware and software for arbitrary manipulation of network traffic. The primary use of NetBoost’s technology was for building a hardware-accelerated intrusion detection system. NetBoost was acquired by Intel Corporation in 1999. After this acquisition, I worked in a product group at Intel in the area of network processors (CPUs designed to manipulate networks traffic), then moved on to a research division. Intel had established a set of independent labs ("lablets") at the University of Washington, UC Berkeley, Carnegie Mellon University (CMU), and the University of Cambridge (England). I joined the UC Berkeley lab in 2001 and worked on various aspects of networking, including communications in especially challenging environments, for which I became an IEEE Fellow in 2009. In 2011, Intel changed its university-engagement-strategy, and these labs were re-structured as smaller "Intel Science and Technology Centers" (ISTCs), which are still present at various locations, including CMU. I left Intel shortly thereafter, completing the textbook TCP/IP Illustrated Volume 1 (second edition), which was published in November 2011. I joined Qualcomm in 2011, where I worked mostly on adaptive video streaming until the end of 2012. My Service On DoD Advisory Boards While I was working at Intel and Qualcomm, I also spent time on a number of DoD advisory boards. I was a member of the Defense Science Study Group (DSSG), a DARPA-sponsored program that connects academics and academically-leaning industry members with the DoD and other government agencies. I later served as a consultant for the Defense Science Board, where I represented the potential effects of cyber attack and defense for the purpose of understanding its capabilities relative to other types of attacks and defenses, including nuclear, directed energy, chemical, conventional explosives. In 2008, I began my four-year term as a member of the Air Force Scientific Advisory Board (AF SAB), and, consequently, a special government employee. While there, I met Professor Doug Schmidt, then deputy director and CTO of the SEI. We worked together on a recently-released study that assessed the technologies and policies associated with assuring cyber situational awareness for the U.S. Air Force. Doug and my immediate predecessor, Professor Bill Scherlis, have provided me with a great deal of advice and context about the research and transition activities at the SEI, for which I am immensely grateful. Progress Thus Far at the SEI Since joining the SEI in January of this year, I have been busy with a range of projects, including reviews of proposals and subsequent selection of our FY 2014 research projects. The selected research projects include a method for quantifying uncertainty in early lifecycle cost analysis (QUELCE), techniques for automated discovery of vulnerabilities in x86 binaries, and a multi-faceted project involving several parts of CMU’s School of Computer Science that focus on technologies needed by soldiers and first responders operating at the tactical edge (EETS). You’ll be able to read more about many of these projects in upcoming postings on the SEI Blog and forthcoming web sites. We recently presented the strategic direction and results of our proposal process to the SEI’s Technical Advisory Group (TAG) and Joint Advisory Committee Executive Group (JAC-EG) that report to our DoD government sponsors at the Assistant Secretary of Defense for Research and Engineering (ASD(R&E)). Likewise, we also recently presented this material to the SEI’s Board of Visitors (BoV), which reports to the provost at CMU. Thanks to hard work by many members of the SEI, these reviews have gone exceptionally well. Indeed, members of the TAG, JAC-EG and BoV have offered many constructive suggestions and connected the SEI with fruitful collaboration opportunities. Strategic Direction My background and experience as a product developer, researcher, and an adviser to the DoD have taught me several things. First, the best research often comes from working on real problems with real users. Second, working with high-quality colleagues breeds high-quality work, so growing our connections with the best researchers and practitioners will help bring out the best in our technical staff. Third, rigid stovepipes lead to duplication, inefficiency, and sometimes conflict. Based on these observations, I have assembled the following goals for myself, the office of the CTO, and other groups at the SEI over the coming months. We intend to Develop an even higher quality and cohesive research program. Within the scope of visibility and responsibility for oversight that the CTO’s office has at the SEI, I want to ensure research projects are of high quality. In particular, each funded project at the SEI must have a clearly stated motivation and intellectual merit, along with a grounded understanding of the parties that can benefit from the work and the estimated timeframe involved. As deputy director for research, I also want to ensure the SEI’s research program is cohesive, and that the SEI’s portfolio of projects explores and defines important areas that best leverage our talents and capabilities, while also creating useful techniques, methods, and tools for our government, industry, and academic stakeholders. Increase collaboration with CMU and other academic researchers. The SEI is one of only two, DoD-sponsored FFRDCs operated by a university. As such, we have a special opportunity and obligation to participate with academia to ensure that our sponsors have access to the most innovative and game-changing technology and ideas available. We are positioned to advance this goal in a way unavailable in the past now that our DoD ASD(R&E) sponsor has enabled us to designate work as "fundamental research," which enables it to be published in a timely manner. I am committed to effectively leveraging the expertise and collaboration opportunities available to the SEI as part of CMU. We are also seeking to expand our engagement with other academic and service lab researchers. Enhance accessibility to SEI’s work. From an operational perspective, I also want to make it easier for sponsors, partners, and software professionals to interact and collaborate with the SEI. In addition to the fundamental research designation mentioned above, we are reorganizing to make us more structurally efficient, to facilitate collaboration across the SEI’s divisions, and to foster feedback loops among various groups of researchers and practitioners. This reorganization includes an internal and external "knowledge management" effort to ensure members of the SEI community can easily determine what type of work we are doing and have done in the past. In addition, we are mining our own expertise and best-known methods derived from research insights and from experience gained through working with our customers. My goal in writing this blog posting was to introduce SEI blog readers to my background and provide some insights into what I see as the shape of things to come at the SEI. I am excited about the opportunities in front of us and the structural changes we are undertaking. I believe we will continue to build upon our legacy of success in software research and transition with help from you. I’ve been working with members of the SEI’s Tech Council to craft a strategic research plan that I will describe in later blog postings. Please don’t hesitate to contact the office with ideas, thoughts, constructive suggestions, or other comments. You may also send an email to info@sei.cmu.edu, or leave feedback in the comments section below. Thanks! Additional Resources For more information on SEI research activities, please visithttp://www.sei.cmu.edu/research. To view my staff page and the office of the SEI’s chief technology officer, please visithttp://www.sei.cmu.edu/about/people/profile.cfm?id=fall_16588

By James Cebula Senior Member of the Technical StaffCERT Division Risk inherent in any military, government, or industry network system cannot be completely eliminated, but it can be reduced by implementing certain network controls. These controls include administrative, management, technical, or legal methods.  Decisions about what controls to implement often rely on computed-risk models that mathematically calculate the amount of risk inherent in a given network configuration. These computed-risk models, however, may not calculate risk levels that human decision makers actually perceive. These decision makers include the network team (e.g., those people who work to implement the controls to ensure the network is secure), the information assurance (IA) officer, and authorizing officials. For example, these models may be missing immeasurable risk factors (e.g., adversarial sophistication or the value to an adversary of data stored on the network) that decision makers perceive as high risk.  This blog post describes the problem of how network security professionals perceive risk in more depth, as well as the SEI’s research efforts to study what risk factors influence perceptions of network risk and how risk perceptions are formulated.  Within the Department of Defense (DoD) and .gov civilian government space, organizations must go through a rigorous process of control implementation, control testing and validation, and documentation of results before bringing an information system or network into production. In this process, controls selected for a particular network are configured and tested by an organization’s network team, led by an information assurance (IA) officer, to reduce the amount of inherent network risk. Guidelines such as the International Organization for Standardization’s (ISO) 27000 series and the National Institute of Standards and Technology’s (NIST) 800 series exist for this rigorous process; however, these guidelines are generic and not tailored to an organization.  After control configuration and testing is complete, a so-called authorizing official signs-off on the network on behalf of the organization. This official indicates that prior to bringing the network into production, the level of residual risk is acceptable. Risk cannot be eliminated, but it needs to be managed at a level that allows the organization to carry out its mission.  The implementation of certain controls comes with tradeoffs (e.g., network performance decrements, network usability constraints, etc.). The ratio of risks to benefits accrued from a particular implementation of controls must be tolerable to the organization’s users, management, IA officer, and authorizing official. While the adverse consequences of a particular control configuration may be known during the process of assessing risks and benefits, the risk factors that the IA officer and authorizing official perceive are not known. Risk factors are pieces of information that are considered when formulating a perception of risk.  For example, an IA officer may believe the network risk is minimal based on risk factors such as minimal adversarial activity a fully staffed network team a network architecture without wireless connections Complications can arise, however, when the IA officer and authorizing official form different network risk perceptions. Circumstances can be equally dangerous when they agree. For example, an IA officer and authorizing official may agree that a particular network risk level is minimal, but the two parties may in fact be considering different risk factors. Consequently, communications between both parties about risk-mitigating strategies can be hampered. Risk perception and its underlying risk factors must be studied to improve network security.  One challenge to the study of risk perception is determining who has the ground truth on network risk levels.  For example, the network team and the IA officer may consider different risk factors than the authorizing official because they all often have different experiences and expertise. These risk factors may be contingent upon a person’s job role (e.g., the firewall expert may perceive risk factors related to protecting a network from intrusions that other team members may not perceive) or the context of the organization (e.g., the organization has no firewall expert).  Conversely, the authorizing official may consider additional risk factors that can be applied to any organization.  This potential for inconsistent perceptions of network risk led us to consider the following two questions: Who has the ground truth on the network risk level at any given moment? What risk factors are most important to perceptions of network risk? Our research is not intended to identify the ground truth, but rather to document the variance of risk factors considered when formulating a risk perception. The SEI research team includes Jennifer Cowley, Sam Merrell, Andrew Boyd, David Mundie, Dewanne Phillips, Harry Levinson, James Smith, Jonathan Spring, Kevin Partridge, and myself.  Through a mixed methods research approach, we aim to assess the factors that individuals use when formulating a perception of network risk whether different perceptions of network risk occur whether those differences are due to different network contexts, personal experiences, and different job-related expertise This research seeks to extend the field of risk perception research into cybersecurity.  There have been several studies examining computed risk in cybersecurity, but little research exists on perceived risk in cybersecurity-related disciplines.  The SEI has studied computed-risk modeling, which associates risk with the likelihood that an outcome will occur and the impact it will have on those affected. Computed risk modeling is frequently recommended as a means to conduct network risk assessments because it is based on measurable factors such as the degree of implementation of controls. In contrast, perceived risk is what individuals identify, understand, and view as the current network risk, independent of computed risk. Risk perceptions may be based on a variety of factors much broader than the factors used in computed risk models. These factors may include prior success or failure experiences with controls currently implemented, knowledge of the adversarial sophistication level, and organizational constraints. Computed risk levels may be one risk-perceptive factor among many considered when formulating network risk perceptions.  Our Research Approach Computer scientists and practitioners have recently developed an interest in studying the human element of network security. Our project is one of the first human-subjects studies undertaken at the SEI.  Prior to the initiation of studies, the SEI conducted a moderated focus group of subject matter experts (SMEs, e.g., former auditing officials and individuals with prior industry experience with the network accrediting process) to identify all possible factors that may impact network risk perceptions.  Examples of these factors include the presence or absence of a known adversary on the network the system of network controls that were or were not implemented workforce shortages current adversarial activity that is known by the operator Note: Internal factors, such as personality and mood, may be equally important  when researching network risk perceptions. To scope the work appropriately, however, we did not include them in this research.   Our mixed methods research approach involves four human-subject studies, each with a unique mission and research method. Table 1 summarizes our study purposes and methodology. The completed results will be available later this year. The four small studies sampled a wide variety of network professionals such as individuals who implement and test network controls, IA officers, and authorizing officials. Job-related demographics were also collected. Table 1.  Overview of the parameters of all four studies The factors generated in the SME focus group were tested in all four studies to assess the importance of each factor to perception of network risk. Because there is no unifying definition of risk, we asked participants to define risk in their own words, so we can analyze the data with respect to their definitions. In addition, we asked participants in studies 2-4 to use the NIST guidelines (NIST Special Publication 800-30) to indicate the level of network risk as low, medium, or high. In this research effort, context is operationally defined as description of an organization’s network at a particular point in time. We hope to identify factors that are consistently important for low-, medium-, and high-risk perceptions. Future Work Earlier studies have found that risk perceptions help predict behavior.  It is therefore essential that the IA officer, network team, and authorizing official have similar network risk perceptions to motivate effective risk mitigating behavior.  Conversations about network risk factors provide a more comprehensive picture of network risk than the NIST rating of low, medium, or high mentioned above. Our long-term goal is to develop a metric of perceived network risk, so that we can study the relationship between perceived risk and the performance of humans and teams. After we better understand how people perceive risk, we can enhance existing training curricula with strategies to mitigate the effects of perceived network risk on human and team performance. Additional Resources For more information about research in the SEI’s CERT Division, please visitwww.cert.org. For more information about the SEI’s research in risk, please visitwww.sei.cmu.edu/risk.

By Grace LewisTechnical Lead, Edge-Enabled Tactical Systems ResearchSEI Software Solutions Division Soldiers and emergency workers who carry smartphones in the battlefield, or into  disaster recovery sites (such as Boston following the marathon bombing earlier this year) often encounter environments characterized by high mobility, rapidly-changing mission requirements, limited computing resources, high levels of stress, and limited network connectivity. At the SEI, we refer to these situations as "edge environments." Along with my colleagues in the SEI’s Advanced Mobile Systems Initiative, my research aims to increase the computing power of mobile devices in edge environments where resources are scarce. One area of my work has focused on leveraging cloud computing so users can extend the capabilities of their mobile devices by offloading expensive computations to more powerful computing resources in a cloud. Some drawbacks to offloading computation to the cloud in resource-constrained environments remain, however, including latency (which can be exacerbated by the distance between mobile devices and clouds) and limited internet access (which makes traditional cloud computing unfeasible). This blog post is the latest in a series that describes research aimed at exploring the applicability of application virtualization as a strategy for cyber-foraging in resource-constrained environments. Cloudlet-Based Cyber Foraging Cyber-foraging is a technique for dynamically augmenting the computing resources of resource-limited mobile devices by opportunistically exploiting a fixed computing infrastructure nearby. This technique allows mobile users to offload computationally-expensive processing (such as face recognition, language translation, speech and image recognition) from a mobile device onto more powerful servers, thereby preserving device battery power and enabling more powerful cloud-based computing. These capabilities are valuable for soldiers or emergency workers who often operate in situations where these resource-intensive applications must be deployed reliably and quickly. Our research focuses on cloudlet-based cyber foraging. Cloudlets, a concept created by Mahadev Satyanarayanan "Satya" of Carnegie Mellon University’s School of Computer Science, are discoverable, generic, stateless servers located in single-hop proximity of mobile devices. Cloudlets can operate in disconnected mode, which means that communication with the central core is only needed for provisioning. They are also virtual-machine (VM) based, which means that they promote flexibility and mobility, a perfect match for edge environments. My previous work in this field focused on cloudlets provisioned using VM synthesis, which was described in the SEI technical note Cloud Computing at the Tactical Edge. When we first began using cloudlets for edge applications, we realized we’d need to make changes to the provisioning process them so they’d functional efficiently and effectively in resource-constrained environments. Satya first proposed using virtual machine (VM) synthesis to provision cloudlets. During VM synthesis, a file called an "application overlay" is transferred during runtime from the mobile device to the cloudlet. This application overlay is built offline and corresponds to the binary difference between a base VM and that VM after the server portion of an application is installed. After the VM overlay has been transferred, it is applied to the base VM. The result is a complete VM that is running the server portion of the application that is executed from a client running on a mobile device. As you can imagine, VM synthesis involves large application overlay files, which can be costly to transfer in terms of battery and network bandwidth consumption in mobile and edge environments. As an alternative, we started looking at application virtualization as a possible solution to this problem. Application Virtualization Application virtualization uses a similar approach to operating system (OS) virtualization, which tricks the software into interacting with a virtual rather than the actual environment. While OS virtualization emulates the hardware for a guest OS, application virtualization emulates OS functionality for an application. To accomplish this emulation, a runtime component intercepts all system calls from an application and redirects these to resources inside the virtualized application. As we described in our technical report, Application Virtualization as a Strategy for Cyber Foraging in Resource-Constrained Environments, the combination of cloudlets and application virtualization allowed us to support the following design goals: Simplicity. Cloudlets should be set up quickly and conveniently. Preparing an application for deployment on a cloudlet should not involve extensive manual overhead, and cloudlet discovery should not require action from the user. Similarly, offloading the application to the cloudlet must be intuitive and simple.  Generality. Packaged applications should be loosely coupled to the operating system so that they can run on multiple cloudlets. This loose coupling enables regular updates and upgrades to the operating system that runs virtual applications without breaking functionality. Applications that are not too deeply integrated into the operating system or too specific to special hardware should be eligible for offloading to the cloudlet. Speed. The time from when the user selects an application to when it is ready for execution—which includes code offload time—should be reasonably small. The user must be able to track the deployment progress via messages from the cloudlet. Our approach to application virtualization with cloudlets accomplishes these goals because it does not require any code modifications and provides a high degree of application portability. Application virtualization tools generate smaller files since they package only those dependencies that are necessary for portability. Our experiments (reported in our technical note) showed that a virtualized application is approximately five times smaller than its application overlay equivalent. Smaller file sizes also lead to smaller application-ready times, defined as the time between the start of the transfer of the virtualized application to the cloudlet and the time that the application is ready for execution on the cloudlet Implementation The two main components of our implementation are the mobile device and the cloudlet host, as shown in the figure below.  The cloudlet host serves as a machine that lends its resources to the mobile device. In our implementation, all devices are connected to the same subnet within a wireless network. The cloudlet host runs a hypervisor to host multiple VMs, which provide a selection of various operating systems and versions. The two main architectural components of our implementation are The mobile device. We developed on Android 4.1, which supports multicast and is required for the discovery mechanism. All the elements of the cyber-foraging-ready application are stored on the mobile device including the application client, the application metadata, and the application package that contains the application server. The cloudlet host. This component comprises a multicast-supporting machine that runs the VM hypervisor. At the SEI, we used two tools to create and execute the virtualized applications for both Linux and Windows systems. Code, Data, and Environment (CDE) is an application virtualization tool for Linux developed by Philip J. Guo and Dawson Engler. CDE allows for virtualizing applications by monitoring their execution. Through the ptrace system call, the supervising CDE program finds files that have been accessed during execution and packages them. The resulting package also contains the environment settings and the CDE runtime environment, which executes the application. Cameyo, an application virtualizer for Windows, packages the application and its dependencies into one single, executable file. Unlike CDE, which monitors execution, Cameyo monitors the installation process and offers two mechanisms for accomplishing the virtualization.  The first mechanism involves taking a snapshot of the system, installing the application, taking another snapshot, and then computing the dependencies and modified registry keys in the difference between the two snapshots. Instead of snapshots, the second mechanism simulates the installation process, tracking all of the installer’s actions. This simulated installation does not have any permanent effect on the system. Benefits and Drawbacks of Our Approach Application virtualization has an advantage over VM synthesis in terms of its performance while deployed. We relied on two metrics to evaluate our approach: deployment time and energy consumption. These two metrics are related because the greater the amount of data transmitted, the greater amounts of battery power required. One benefit of application virtualization is significantly smaller file sizes, the result of packaging just the application and not calculating a binary difference between virtual machine images. Another benefit of application virtualization is the loose coupling between an application and its required cloudlet environment. We did realize some drawbacks of our approach. Application virtualization is not suitable for every type of environment. For example, device drivers interact with the hardware directly and cannot be virtualized. It is also hard to virtualize software that interacts with OS internals, such as antivirus programs. Application virtualization also requires careful dependency management to ensure that an application operates correctly on a cloudlet. If all dependencies are not included in the virtualized application then it will not work. Moreover, missing dependencies must be added manually, which is tedious and error-prone.  Future Work in Advanced Mobile Systems Our Advanced Mobile Systems Initiative is building a portfolio of capabilities and options for use by soldiers and emergency workers who operate in various environments with different quality attribute requirements. Cyber-foraging is one of three areas that we are investigating and our body of work on support for resource-constrained environments is growing. The other two areas of work in the SEI’s Advanced Mobile Systems Initiative are group-context-aware mobile applications in which groups of mobile devices share contextual information so that teams and their mobile devices can make better decisions regarding resource consumption These types of applications also give users a better understanding of environmental factors and the activities being performed by the group based on information gathered by sensors on the mobile device or connected to the mobile device. user-configured situational awareness mashups that allow soldiers and emergency workers to filter and combine geo-located data such as DoD situational awareness SA feeds and public data sources into a single map-based user interface. In developing this cyber-foraging strategy, I collaborated with Dominik Messenger, a student from the Karlsruhe Institute of Technology (KIT) in Germany. He came to CMU last year via the InterACT exchange program. His advisor at KIT contacted me and said that he was very interested in our research. This research and the resulting technical report were part of his diploma thesis. We look forward to your feedback on our work in the comments section below. Additional Resources To view the SEI technical note, Application Virtualiation as a Strategy for Cyber Foraging in Resource-Constrained Environments, please visithttp://www.sei.cmu.edu/library/abstracts/reports/13tn007.cfm. To read the SEI technical note, Cloud Computing at the Tactical Edge, please visithttp://www.sei.cmu.edu/library/abstracts/reports/12tn015.cfm. For more information on the SEI’s research in the field of pervasive mobile computing, please visit http://www.sei.cmu.edu/mobilecomputing/research/index.cfm To read the paper, The Case for VM-Based Cloudlets in Mobile Computing, please visit http://www.cs.cmu.edu/~satya/docdir/satya-ieeepvc-cloudlets-2009.pdf.

By Julien Delange, Senior Member of the Technical StaffSoftware Solutions Division When life- and safety-critical systems fail, the results can be dire, including loss of property and life. These types of systems are increasingly prevalent, and can be found in the altitude and control systems of a satellite, the software-reliant systems of a car (such as its cruise control and GPS), or a medical device. When developing such systems, software and systems architects must balance the need for stability and safety with stakeholder demands and time-to-market constraints. The Architectural Analysis & Design Language (AADL) helps software and system architects address the challenges of designing life- and safety-critical systems by providing a modeling notation that employs textual and graphic representations. This blog posting, part of an ongoing series on AADL, describes how AADL is being used in medical devices and highlights the experiences of a practitioner whose research aims to address problems with medical infusion pumps. Although AADL was initially applied in the avionics and aerospace domains, it is now being used in other domains where software failures have significant impact. The design of life- and safety-critical systems in the medical domain, where an error or software fault may have catastrophic consequences including loss of human life, has been the topic of recent media reports, as well as increased scrutiny from the FDA as described in this story in The New York Times.  The size and complexity of medical-related software continues to grow and the more complex the software, the more likely it is to contain bugs or errors. A classic example was the Therac-25 linear accelerator, which malfunctioned due to software problems and gave overdoses of radiation to patients. Medical devices must be designed and validated carefully to shield users from software defects.  For that purpose, engineers must adhere to a strict development process that requires analyzing and validating their architecture prior to implementation efforts. Such a process would ensure requirements enforcement and reduce potential re-engineering efforts. The SEI’s Work on AADL The SEI has been one of the lead developers of AADL and has participated on the AADL standardization committee since its inception. SEI researchers also developed the Open Source AADL Tool Environment (OSATE), which is the reference implementation for supporting AADL within the Eclipse environment. Other researchers and developers have used AADL as a foundation to design and analyze life- and/or-safety-critical systems. Likewise, several projects have used AADL to design software architecture and analyze, validate, or improve various quality attributes, such as reliability, latency/performance or security. While AADL was initially conceived for use in avionics or aerospace systems, it can be used to design systems in other domains that place a premium on life- and/or safety-critical behavior. An apt example is the domain of medical devices, which have expensive and time-consuming certification and accreditation processes to validate and verify that they are free of software bugs. Thus, the AADL design and validation methods and tools that have been developed for avionics and aerospace can be applied and reused between the various domains. One Practitioner’s Experience Oleg Sokolsky, a research associate professor with the department of computer and information science at the University of Pennsylvania, first became involved in AADL in 2004 after applying for and receiving a Small Business Innovative Research grant. Sokolsky described his introduction to AADL: The sponsor was looking for modeling languages and analysis tools for complex distributed systems. We had some tools that would be applicable, but we had to make them work in a wider context.  A connection to the architecture level was needed. I recently heard a workshop talk on AADL and it was still fresh in my head. I have a colleague who had a startup, so he led the tool development effort. We began by applying schedulability analysis techniques to AADL models, using AADL semantics. It was a good fit for the tools we had in house. Next, Sokolsky’s team extended the analysis for the same semantic representation and built what he believes is the first simulator for AADL models. Applying AADL in the Medical Domain After their initial application, Sokolsky said his team applied AADL to modeling whatever systems they were developing at the time, including medical devices. Sokolsky’s team started using AADL in their research to address software problems as part of the U.S. Food and Drug Administration’s (FDA) Infusion Pump Improvement Initiative, which seeks to make infusion pumps safer and more reliable. Sokolsky described some of the architecture problems with the infusion pump that his team was trying to address: In several cases, there wasn’t enough sanity checking on inputs. Software should be designed in such a way that it should reject wrong inputs. Unfortunately, several pump models weren’t designed that way.It was very easy for a nurse to enter a wrong value. Over-infusion and under-infusion are both serious hazards. Specifically, his team used AADL to describe the infusion pump’s software architecture, the platform-independent controller (which receive inputs and reacts to outputs), and the platform-dependent software, such as drivers for sensors and actuators between them. Sokolsky described the ways in which his team used architecture and modeling techniques to address some of the problems with the infusion pump: In the reference architecture, there should be a module that assesses quality of inputs and rejects the wrong ones. Also, timing information on the components in the architecture can help us evaluate latency of data flows to determine whether the pump will stop the motor fast enough in case of an alarm. Over the years, Sokolsky’s AADL-based research has focused on developing platform-independent code generation techniques, specifically for model-based development of infusion pump software. We had a very detailed model of the safe controller for the infusion pump, and we were looking for ways to generate code for it automatically. What we realized is that existing code generation tools are producing platform-independent code that had to then be manually connected to the platform. We were looking for ways to minimize the amount of new code that had to be written manually. That’s where AADL came in. It allowed us to describe the various platform dependencies that existed in our systems and characterize, for each kind of dependency, what code needs to be generated for a particular platform. Sokolsky said his team continues to use AADL because they understand the semantics of the language and have long-standing experience with it. One of the challenges they face, however, is keeping up with all of the changes that are occurring within the AADL community, which operates its own wiki site: Right now there are so many things going on in this AADL ecosphere. We have a hard time picking things that are mature enough that we can actually use. Looking Ahead Sokolsky described various AADL-based design artifacts that his team has developed for the infusion pump.  The artifacts include hazard analysis documents, requirement specifications, models of pump controllers, verification results, generated code, and a preliminary safety case: The goal is to provide a set of artifacts for the community and the FDA, so, on the one hand, researchers have a way to apply their formal methods to these artifacts. The FDA can look at the safety argument that we are constructing. It gives feedback to the community. Sokolsky said that the manufacturer’s intellectual property restrictions do not impede the dissemination of documents, models, and code developed by his research team.. Such artifacts can serve as case studies for the wider research community, enabling comparison of modeling and analysis tools from different research groups. These artifacts could also potentially be used by the FDA to showcase good development practices to manufacturers and develop future guidance documents for industry. Future Work SEI researchers are working to extend AADL so that it can describe system faults and analyze the safety of systems. These capabilities will enable AADL users to augment their software architectures with fault description and specify error source and propagation across hardware and software components. This enhanced notation would then be used to improve system validation and detect faults that are potentially not spotted during the initial development process and that may lead to errors. This work consists of two parts: enhancing the core of the technology with a sub-language to describe architecture safety concerns developing new tools to process this additional information within the model, validate system safety, and assist engineers in the production of documents for validating the system For the purpose of improving system validation and detecting faults, a new language has been proposed by the SEI and is currently under review by the AADL standardization committee. The language would be published as an official standard annex later by the Society of Automotive Engineers (SAE). In addition, the SEI is developing new functions within OSATE to analyze system safety and produce documents required by safety evaluation standards, such as SAE ARP4761. Looking ahead, Sokolsky said that his team will work with the SEI to use the error-model annex and specify an error model for the pump to reason about its reliability. AADL is fairly versatile and, if used to its full capacity, it gives a boost to whatever other development techniques that you are using on top of it. Our next blog post will discuss the use of AADL in the System Architecture Virtual Integration(SAVI) project. SAVI is part of the collaborative and applied research performed by an aerospace industry research cooperative. Our post will describe the use of AADL within this project to improve software safety and reliability. Additional Resources To view the AADL Wiki, please visit https://wiki.sei.cmu.edu/aadl/index.php/Main_Page For more information about AADL, please visit http://www.aadl.info For more information about the Generic Infusion Pump research at the University of Pennsylvania, please visit http://rtg.cis.upenn.edu/gip.php3

By Robert Ferguson Software Solutions Division Software sustainment involves coordinating the processes, procedures, people, information, and databases required to support, maintain, and operate software-reliant aspects of DoD systems. The 2011 book Examination of the U.S. Air Force’s Aircraft Sustainment Needs in the Future and its Strategy to Meet Those Needs states The Air Force is concerned that the resources needed to sustain its legacy aircraft may increase to the point where they could consume the resources needed to modernize the Air Force. With millions of lines of code riding on aircraft and automobiles, the cost of software sustainment is increasing rapidly. Several studies show that the cost of sustainment is already as much as 70 percent of the total cost for the life of the software. All the armed services face similar challenges, including deciding how to improve the efficiency and productivity of sustainment organizations and how much should be invested in these improvements. This blog post describes an SEI research initiative aimed at developing an economic model to help anticipate costs and postpone the potential tipping point when sustaining current products is less attractive than replacing legacy systems. Balancing Stakeholders and Resources The software sustainment problem is particularly complex for the Department of Defense (DoD) because funding decisions involve an understanding of tensions between three different perspectives: operational need (warfighter view) management of the portfolio (materiel view) capability and capacity of the sustaining organization (process, skills, tools and people) Our research is motivated by the need to help the DoD make decisions about allocating resources between sustainment work (supporting the warfighter) and improving the performance of sustainment organizations in ways that optimize long-term value to the armed services. Performance improvements are needed in many situations, including new test kits to support deployment of new radar or electronic warfare capabilities software analysis tools to analyze existing code to help developers learn a legacy system and accelerate their understanding of supported products tools supporting automated software testing to accelerate the testing process and assure test coverage of supported products training for engineers when upgrades to an existing system employ new processors, operating systems, or programming languages This SEI research initiative is developing an economic model to support decisions about allocating investments in various performance improvement alternatives. The model will analyze various factors, such as demand for sustainment, capacity of an organic workforce to perform sustainment activities, as well as timing of funding in terms of its impact on long-term costs and readiness of aircraft fleets. I, along with fellow researchers Sarah Sheard, Andrew Moore, William Nichols, and Mike Phillips, have spent the last year exploring several issues related to software sustainment. Part of our work included developing a systems dynamics model to study how funding decisions and the timing of implementation of changes in sustainment organizations affect the performance of both sustainers and warfighters. We theorized that small differences in funds and timing can have large impacts on performance. This type of system dynamics model uses stocks and flows to represent sustainment performance over time. Foundations of Our Approach Jay Forrester, a professor at Massachusetts Institute of Technology, founded system dynamics in the 1950s. Systems dynamics modeling studies the changes in many interrelated variables. Since its inception, system dynamics has been used as a modeling approach in the study of economics and organizations.  With many factors changing simultaneously, simpler economic models (such as return-on-investment and net present value) are insufficient. The interaction of the many inputs to sustainment work can cause emergent effects, such as a sudden and dramatic change in ability to meet demand. Through modeling and analysis research, we are looking for the minimum amount of data that can be used to forecast a sudden and dramatic change (which is also known as a "tipping point"). Forecasting the tipping point gives decision makers time to take action before a problem becomes intractable. We seek to answer the following questions through our research: Does the model actually show the expected tipping point behavior either in the performance of the sustainment organization or the demand for system improvement? What data provides an indication of growth in demand for sustainment on a particular acquisition program? If we can identify the needed data, will it be possible to collect it and do the necessary analysis? Do the models provide actionable information to decision makers in a timely manner? For example, does reallocating some funding from the sustainment work to the development of the workforce (test kits, process changes, technology training) help reduce the cost of sustainment? What measure of warfighter readiness correlates to the predictive factors in the model? Through this approach, we aim to help DoD acquisition programs better plan their financial investments to ensure long-term software sustainment and deliver the best value for taxpayer dollars. While the construction of a model that exhibits the expected changes in the performance of both mission and sustainment is a necessary step, much of our research will focus on sources of data. Real data collected from real programs will be needed to calibrate the model for making real decisions, including potential data collection points within the sustainment processes, such as demand for sustainment work, rate of hiring and attrition, rate of delivery of, sustainment work, and availability of skilled staff potential opportunities to measure warfighter readiness or system use such as fleet availability, successful mission performance, and error rate in operation standards for applying data collection across different kinds of products such as airplanes, satellites, and communications systems The Systems Dynamics Model The basic goal of a simulation model is first to represent the normal behavior of a system and then stimulate it with a new input to see how the responses change.  The model that we have developed represents the behavior of the different players in the sustainment process, including the warfighter, the technical capability of the sustainment organization, and the capacity of the sustainment organization to deliver the work.  We are testing the system response to various scenarios, such as Threat Change. An external change (such as a new threat to the warfighter) results in a request to update the system capability. This request means the sustaining organization will have to perform both product and process changes; the development process and testing may need to change as well. The changes often require some funding to re-equip the facility and re-train the workforce. Our systems dynamics model helps decision makers analyze the effect if funding for this improvement is delayed. Support Technology Change. The sustainment organization decides to improve its own throughput and adopts new processes to "do more with less." Typically the change is also in response to new quality goals. In this case our model helps codify the effect on sustainment capability and capacity and therefore on operational performance. Workforce Changes. Sequestration effectively decreases the staff available to sustainment organizations by 10 percent to 20 percent. How does this decrease affect a sustaining organization’s ability to meet its sustainment demand? Does it affect aspects of the warfighter mission as well? Our current model of sustainment consists of five basic process loops: Mission Demand. This loop represents system deployment and use in theater. With successful application, demand for additional missions and increased capability causes an increase in sustainment support required. When missions are less successful or systems cannot be deployed in time, demand for the system decreases and funding support wanes. Sustainment Work. The process of sustainment typically takes a system off-line as it is enhanced, repaired, and redeployed. Some of the requests for sustainment include requests for additional capability. Limits to Growth. This loop shows how the capacity and capability of a sustainment organization limit the rate of completion of sustainment work. As these limits begin to extend the time required to redeploy, the long-term effect may be a reduction in demand or a switch to using an alternate platform. Work Bigger. A sustainment organization may attempt to meet sustainment demand by employing overtime or extra contract employees. Either of these approaches may work for a short time or a small additional cost, but they stress the organization and quickly reach limits of effectiveness. The organization can hire staff, but it must also allow time for training and acculturation of new hires to meet performance objectives. Work Smarter. The sustainment organization invests in new capabilities (skills, tools, and processes) and possibly additional resources (people and facilities) to improve capacity for sustaining work. Each scenario outlined above entails several decisions and stimulates response curves from the model. The response curves help decision makers forecast how deferring decisions or reallocating resources affect both warfighters and sustainment organizations. We will consider our systems dynamic model "good" if decision makers believe they are able to make faster decisions and if the data from the model makes it easier to get sponsor support for the decisions. Initial Findings and Future Work A recent study by the U.S. Army provided us with information confirming the importance of understanding the cycles of demand for sustainment work. From that study, we identified three patterns of release in the software sustainment lifecycle: The basic fix pattern mainly focuses on low-level defects and addresses software breaks, which occur frequently (at least once a year). The block release pattern mainly focuses on enhancements to address users who have discovered new opportunities to use the system but need some additional functionality to employ it efficiently. The resulting block releases can include additional features for which there are plans and budgets. These sustainment efforts are not patches or quick fixes; they typically occur every year to two years. The modernization pattern addresses a significant change in technology or in the pattern of use, such as adding electronic warfare capabilities to an aircraft or accessing an additional external system that hasn’t been accessed previously. These sustainment efforts occur about every four to seven years. This third type of release is also often three to four times more expensive than the block release. The modernization release is often a response to some technology change. The timing (at least four years) has an interesting correlation. Moore’s Law suggests the cost-performance of processors doubles every 18 months. A modernization release every four years skips about two processor generations.  A modernization release every seven years skips about six generations. Modernization releases require the sustaining organization to develop new practices and tools and to retrain some existing personnel. Funding for these internal changes is often hard to justify and may be delayed. These delays, in turn, can cause the sustaining organization to fall so far behind the technology curve that a new acquisition contract is often required. If a contractor does win the development contract, the organic sustainment organizational capability may decline while the bulk of funds go to the contractor. Other organizations have studied various aspects of the sustainment problem. The Army examined costs of software sustainment and the work breakdown structure of sustainment as it is currently performed. Jack McGarry, author of the book Practical Software Measurement: Objective Information for Decision Makers, charted for the U.S. Army the tremendous variability in the demand for sustainment work, with the cost of the largest releases up to 10 times that of the smallest releases. The frequency of the largest releases appears to track to large-scale technology change, occurring about every four to seven years. In that period of time, the speed of processors increases by nearly 10 times, and the number of transistors on a chip also increases by 10 times. This finding suggests that technology change is an important variable in our systems dynamic model. The research we’ve conducted thus far has revealed that our system dynamics model exhibits the expected and observed behavior of product sustainment. The model, however, has not yet been calibrated to apply to a real situation. We are actively seeking an opportunity to study a real sustainment program and to collaborate on calibrating the model. We believe that data collection will not be hard to implement and the data potentially can be used for other purposes by sustainers. If you are interested in collaborating with SEI researchers on this initiative, please send an email to info@sei.cmu.edu, or leave contact information in the comments section below. We welcome your feedback on our research and look forward to hearing if you’d like to help us achieve our project goals. Please leave feedback in the comments section below. Additional Resources To read An Examination of the U.S. Air Force’s Aircraft Sustainment Needs in the Future and its Strategy to Meet Those Needs by the National Academies Press, please visithttp://www.nap.edu/catalog.php?record_id=13177 To read about software sustainment practices for the DoD, (especially chapter 16) please visit http://www.stsc.hill.af.mil/resources/tech_docs/gsam4.html To read The Economics of Software Maintenance in the Twenty-first Century, please visithttp://www.compaid.com/caiinternet/ezine/capersjones-maintenance.pdf To read the SEI technical report, Sustaining Software-Intensive Systems, please visit http://www.sei.cmu.edu/library/abstracts/reports/06tn007.cfm To see a presentation on Sustaining Software Intensive Systems — A Conundrum please visit http://www.dtic.mil/ndia/2005systems/wednesday/lapham.pdf To read an excerpt of the book Business Dynamics: Systems Thinking and Modeling for a Complex World by John Sterns, please visithttp://web.boun.edu.tr/ali.saysel/Esc578/Sterman%2013.pdf To view the proceedings of the 2012 PSMSC User Group, please visit http://www.psmsc.com/UG2012/Workshops/w4-%20files.pdf To read the Air Force Scientific Advisory Board report, Sustaining Aging Aircraft, please visit http://oai.dtic.mil/oai/oai?verb=getRecord&metadataPrefix=html&identifier=ADA562696

By Cory Cohen Senior Member of the Technical StaffCERT Division In 2012, Symantec blocked more than 5.5 billion malware attacks (an 81 percent increase over 2010) and reported a 41 percent increase in new variants of malware, according to January 2013 Computer World article. To prevent detection and delay analysis, malware authors often obfuscate their malicious programs with anti-analysis measures.  Obfuscated binary code prevents analysts from developing timely, actionable insights by increasing code complexity and reducing the effectiveness of existing tools. This blog post describes research we are conducting at the SEI to improve manual and automated analysis of common code obfuscation techniques used in malware. Obfuscation Example and Impact on Malware Analysis When shown the following example of obfuscated assembly code, an experienced malware analyst at CERT took 550 seconds (more than 9 minutes) to determine its basic functionality. 43E401: push 9387D2AF 43E406: mov  edx, F25C92BA 43E40B: pop  eax 43E40C: xor  edx, F21F75B2 43E412: and  eax, 37D28E56 43E418: jmp  43E501 ... 43E501: push ecx 43E502: mov  ecx, eax 43E504: push edx 43E505: jnz  43E601 43E506: jmp  43E603 ... 43E601: sub  eax, 13828202 43E607: ret ... 43E708: pop edx 43E709: mov ecx, [eax+edx] While this example was created explicitly for this demonstration, it shows several techniques commonly encountered in malware and represents a realistic scenario. In particular, the net effect of this code is equivalent to a single instruction that is easily understood: 43E401: mov ecx, [ecx+4] In the Department of Defense (DoD) and other large analysis environments, several hundred-fold increases in the effort required to analyze malware raises computer and network defense costs and delays the necessary insights from being generated within reasonable time and budget constraints. Our Approach to Deobfuscation Deobfuscation is a method for reverse-engineering obfuscated code. Our approach to the deobfuscation problem applies multiple semantic transformations to an obfuscated program to produce a new program that is functionally equivalent to the obfuscated program but easier to analyze. Some example obfuscation techniques and rough descriptions of the transformations include Complex hexadecimal arithmetic. Malware authors frequently add extraneous addition, subtraction, and bitwise logical operations to hide important program addresses and constants. When possible, the instructions performing these operations should be replaced with immediate values in simpler instructions. Stack pointer abuses. Malware authors will often have many unnecessary PUSH and POP instructions to make the code harder to reason about. When used in conjunction with CALL and RET instructions, control flow can be obscured as well. Such instructions should be removed to minimize stack changes. Control flow obfuscation. Unnecessary jumps in intra-procedural flow control make it hard for a human analyst to follow overall program flow control. Unconditional and always-true conditional JMP instructions should be removed. Dead stores. Instructions that write to registers and memory that are never subsequently read do not contribute meaningfully to the net effect of the function. Instructions performing dead stores can be detected by definition and usage analysis and subsequently removed. These types of transformations may be run in multiple passes and in varying orders to make it easier to defeat obfuscations that involve the application of techniques in sequence. A modular approach to the transformation should improve code maintenance and simplify the addition of new techniques. Prototype Deobfuscation Tool Early in this research effort, we used the ROSE compiler infrastructure to build a functioning prototype of a general deobfuscation tool that was specifically targeted at removing dead code. ROSE provides facilities for disassembly, instruction emulation, and control flow graph representation. We have applied ROSE in our prior research on Using Machine Learning to Detect Malware Similarity and Semantic Comparison of Malware Functions. In our current effort, we expanded upon the core ROSE capabilities by adding definition and usage analysis, dead code elimination, and other deobfuscation-specific techniques. We also implemented a rudimentary method for generating new executables as an output format by building on ROSE’s assembly features. This prototype was tested against several members of the Ramnit family, using a transformation to eliminate dead code. Applying Our Prototype Tool to String Deobfuscation While working on the prototype tool, our team was presented with an operational need to combat a specific type of string obfuscation. String obfuscation is used by various malware families to prevent important strings from being easily recovered from the executable. This technique involves moving immediate bytes into a local stack variable to construct a normal C-style null terminated string. By emulating instructions and collecting memory writes to the stack, we were able to extract the deobfuscated strings from the code. This transformation allowed us to automatically process many more files than would have been possible using a manual approach. The deobfuscated strings assisted in the development of a malware family report. We also produced a catalog of common obfuscation techniques, with specific reference to the addresses of several files demonstrating each of the techniques.  At this point in our research, we had demonstrated a working prototype and operational relevance, but needed to implement more transformations to have a complete and generally applicable tool. The primary question we still faced was "which transformations should be implemented next?" We decided to conduct an obfuscation technique prevalence study to gain some basic facts about the prevalence and distribution of the techniques in our catalog. Analyzing the Prevalence and Distribution of Obfuscation Techniques We chose to implement six tests for our initial study. Each test looked for the occurrence of a common obfuscated code pattern, such as a dead store, an opaque predicate, or an obfuscated control flow. We counted how often these patterns appeared in several datasets, containing a total of 150,000 malware files. The number of detected obfuscations was fairly low, with only about 12 percent of functions having a detected pattern, and we detected only a single test in most of those functions. This distribution suggested we might be encountering an occasional single false positive in many functions, while truly obfuscated functions were routinely positive for many detections in several tests. Adjusted to account for these probable false positives, the percentage of functions in which we detected obfuscated code declined to about 1 percent. Using the adjusted criteria, only about 23 percent of files contain an obfuscated function. Some of the likely false positives filtered by the revised criteria appear to have been caused by incorrect disassembly. In several cases we examined manually, we found that arbitrary bytes had generated nonsense instructions that legitimately contained the obfuscated code patterns that we were searching for. It is hard to build firm program analysis conclusions on unreliable disassembly foundations, so we’ve started a new research effort to develop improved disassembly methods and objective metrics for the assessment of disassembly correctness. Collaborations Charles Hines from the CERT Division and Wesley Jin, a Ph.D. candidate in CMU’s Department of Electrical and Computer Engineering, have been actively involved in this research effort. Sagar Chaki and Arie Gurfinkel, both senior members of the technical staff in the Software Solutions Division have contributed significantly as well. Additional Resources To read more about this research and the results of other exploratory research projects, download the SEI technical report, Results of SEI Line-Funded Exploratory New Starts Projects: FY 2012, athttp://www.sei.cmu.edu/library/abstracts/reports/13tr004.cfm

By David MundieSenior Member of the Technical StaffCERT Division Researchers on the CERT Division’s insider threat team have presented several of the 26 patterns identified by analyzing our insider threat database, which is based on examinations of more than 700 insider threat cases and interviews with the United States Secret Service, victims’ organizations, and convicted felons. Through our analysis, we identified more than 100 categories of weaknesses in systems, processes, people, or technologies that allowed insider threats to occur. One aspect of our research focuses on identifying enterprise architecture patterns that organizations can use to protect their systems from malicious insider threat. Now that we’ve developed 26 patterns, our next priority is to assemble these patterns into a pattern language that organizations can use to bolster their resources and make them more resilient against insider threats. This blog post is the third installment in a series that describes our research to create and validate an insider threat mitigation pattern language to help organizations balance the cost of security controls with the risk of insider compromise. Developing an Enterprise Architecture Pattern Language Our aim in developing an insider threat pattern language is to equip enterprise engineers with the tools necessary to make an organization resilient against insider threat. The patterns in our language capture solutions to recurring patterns in insider threat. For example, the Thirty-Day Window pattern, discussed by my colleague Andrew Moore in his blog Effectiveness of a Pattern for Preventing Theft by Insiders, is based on the observation that a large percentage of malicious exfiltration by insiders happens within 30 days of termination. So, organizations can improve their detection of such violations by focusing on a very narrow window of time. Once we identified and documented the original 26 patterns, we wanted to go beyond simply publishing them as a flat, two-dimensional collection. Instead, we wanted to show the relationships among the patterns by arranging them in an organic hierarchy (i.e., a pattern language). This approach follows the example of Christopher Alexander, the father of the patterns movement in the building architecture community, and his work on patterns and pattern languages. Unfortunately, insider threat poses challenges to a hierarchical approach to organizing patterns because insider threats permeate the organization. Addressing and protecting enterprise organizations from insider threat involves an enterprise-wide approach involving several different strategies. This diversity leads to multiple potentially conflicting classification systems. A classification that makes sense for human resource systems might not make sense from an incident response perspective. Likewise, some third classification might be required for the information technology staff, and so forth. Trying on Classification Systems We explored several categorizations before ultimately deciding on a multi-dimensional approach. Each of the systems that we explored provided useful perspectives on our patterns by situating them within a specific domain, whether information security, enterprise architectures, incident management, resiliency, or organizational structure. Information security pattern languages. We initially assumed that our insider threat patterns would fit well within existing information security pattern languages. As a guide, we relied on the book Security Patterns by Schumacher et al., which maps information security pattern languages to the categories in the Zachman Framework (see below). Our insider threat patterns were most compatible with the "Enterprise Security and Risk Management" patterns because the crux of the insider threat problem is that, for employees to do their jobs efficiently, they must be trusted by their employers and operate in an environment largely unfettered by authentication controls, access controls, and firewalls. As a result, our patterns are a bit of a mismatch for the identification and authentication patterns, the operating system access control patterns, and the firewall architecture patterns, which are largely focused on implementing those fetters. We soon realized that this mismatch meant that despite a commonality of purpose, the Schumacher landscape would not be ideal as a single taxonomy for our pattern language. The Zachman Framework. We then mapped our patterns to the Zachman Framework, which is an enterprise architecture framework that provides a formal and highly structured view of an enterprise. The Zachman Framework provides a diagram for organizing architectural artifacts on the basis of the intended recipient and the issue being addressed. The diagram helped us quickly realize that our patterns were at the enterprise security strategy and policy levels and not at the mechanisms and implementation levels. This was an important insight, but it did mean that our patterns were clustered in one small area of the framework, making it of limited utility as the single way of classifying our patterns. Business units. We also considered organizing our patterns by business units, such as human resources, legal, etc. Business units provide a very useful organization system for the patterns. Business units are less useful as a general-purpose classification system, however, because they obscure the fact that our patterns frequently cross business-unit boundaries. Lifecycle phase. Another approach we explored involved using the insider threat lifecycle to break the patterns into prevention, detection, and response patterns. This approach made sense in terms of risk management and incident response, but proved less satisfactory in terms of the organization-wide enterprise focus of insider threat. CERT Resilience Management Model. Finally, we considered the CERT Resilience Management Model (CERT-RMM), a broad-based model of the organizational process areas needed for resilience. Unsurprisingly, this was a better fit for insider threat patterns in the sense that its process areas divided our patterns into logical groups, but even so we did not want to give up the insights provided by the other schemas. A Multi-Dimensional Organizational Structure After exploring the categorizations listed above, we realized that it would make sense to move away from rigid, top-down, linear hierarchical systems. No one system would serve all users and all use cases equally well, so a multi-dimensional classification system was called for.One problem with multi-dimensional constructs is that the human brain struggles to conceive of ideas beyond three dimensions. Instead, we looked to a library classification technique known as faceted classification. Recently, faceted classification has become widely used again in search engines on commerce websites. When shopping on Amazon’s site, for example, users can narrow their searches by classifications or facets, such as price, color, or manufacturer. This approach made sense in this age of near-ubiquitous computing where users have easy access to higher dimensional structures. The specific implementation of faceted classification that we used is the facet map, which we downloaded from Facetmap software. We realized two benefits to organizing the pattern language as a drill-down facet map Users can specify the exact aspects of the patterns that are applicable to their circumstances. For example, a user can narrow a search to only address insider threat patterns within human resources. That same search can also be narrowed to include misuse of information technology infrastructure. The faceted classification’s formal description of the pattern language organization makes it easier for users to generate alternative representations and categories. The facet map model allowed us to organize our patterns into a map that categorizes each of the 26 patterns in a five-dimensional space defined by the classifications described above.  Figure 2 below shows the Facetmap interface to this hyperspace. Current and Future Work Our current work focuses on pattern composition because we feel that is a crucially important issue in transitioning patterns to the operational community. Instead of trying to impose a single composition method on all end users, we have created a pattern language to help usersselect from a number of different composition methods, depending on the situation. For example, the way a small software startup would want to integrate an insider threat pattern into its organization will be different from the way it might be integrated into a military organization or a multinational corporation. To assist in validating the composition operations, we are exploring the idea of using simple ontologies to capture the essential components of a pattern and its relationships. For example, the 30-day Window Pattern is essentially a relationship among the human resources and information technology staff and the employee. To allow users to easily guarantee the completeness of their pattern composition, we are testing the use of a formal ontology expressed in the Web Ontology Language (OWL). For more information on OWL and the more general use of ontologies in information security, please see CERT’s Security and Ontology webpage. The integration of our pattern language and its multi-dimensional interface, combined with the pattern composition pattern language and our ontology-driven validation methodology, will be a significant step in the evolution of insider threat mitigation techniques. We welcome your feedback on our work. Please send us an email at info@sei.cmu.edu or leave feedback in the comments section below. Additional Resources Cybersecurity experts from the CERT Insider Threat Center will present a free virtual event on current research aimed at establishing best practices to mitigate insider threats. Managing the Insider Threat: What Every Organization Should Know will take place Thursday, Aug. 8, from 9 a.m. to 5 p.m. EDT. To register, please visithttp://www.webcaster4.com/Webcast/Page/139/1742 To learn more about OWL and the use of ontologies in security, please visithttp://www.cert.org/csirts/security-and-ontology.html To read the SEI technical report, A Pattern for Increased Monitoring for Intellectual Property Theft by Departing Insiders, please visitwww.sei.cmu.edu/reports/12tr008.pdf To read the SEI technical note, Insider Threat Control: Using Centralized Logging to Detect Data Exfiltration Near Insider Termination, please visit www.cert.org/archive/pdf/11tn024.pdf To read the CERT Insider Threat blog, please visit www.cert.org/blogs/insider_threat/

Charles B. WeinstockSenior Member of the Technical StaffSoftware Solutions Division From the braking system in your automobile to the software that controls the aircraft that you fly in, safety-critical systems are ubiquitous. Showing that such systems meet their safety requirements has become a critical area of work for software and systems engineers. "We live in a world in which our safety depends on software-intensive systems," editors of IEEE Software wrote in the magazine’s May/June issue. "Organizations everywhere are struggling to find cost-effective methods to deal with the enormous increase in size and complexity of these systems, while simultaneously respecting the need to ensure their safety." The Carnegie Mellon Software Engineering Institute (SEI) is addressing this issue with a significant research program into assurance cases. Our sponsors are regularly faced with assuring that complex software-based systems meet certain kinds of requirements such as safety, security, and reliability. In this post, the first in a series on assurance cases and confidence, I will introduce the concept of assurance cases and show how they can be used to argue that a safety requirement (or other requirement such as security) has been met. Making a Sound Evaluation A system is deemed "safety-critical" when its failure could result in loss of life or catastrophic damage. Such systems are expensive to develop, build, and deploy. Due to the severe consequences of failure, most safety-critical systems are also subject to evaluation by some external authority (such as the U.S. Food and Drug Administration (FDA) for medical devices in the United States or the Ministry of Defence for flight control systems in the United Kingdom). To enable the evaluator to make a sound evaluation, developers must provide information about the system and its development, including requirements, documentation of design decisions, hazard analyses, development process, and verification and validation results. Developers submit this information (which constitutes an "argument") that they believe will convince evaluators to approve the system. Industry has long relied on process-based arguments to help evaluators conduct their evaluations, but as the National Research Council of the National Academy of Science reported in 2007: For a system to be regarded as dependable, concrete evidence must be present that substantiates the dependability claim. This evidence will take the form of a dependability case arguing that the required properties follow from the combination of the properties of the system itself (that is, the implementation) and the environmental assumptions. … In addition, the case will inevitably involve appeals to the process by which the software was developed.   Although called a "dependability case" in the NRC report, most researchers, evaluators, and users have since coalesced around the term "assurance case." When developing an assurance case for safety, the case is usually called a "safety case." As defined by the U.K. Ministry of Defence: A safety case is a structured argument, supported by a body of evidence that provides a compelling, comprehensible and valid case that a system is safe for a given application in a given environment. An assurance case is somewhat similar in form to a legal case. In a legal case, there are two basic elements: evidence, including witnesses, fingerprints, DNA, etc. an argument given by the attorneys as to why the jury should believe that the evidence supports (or does not support) the claim that the defendant is guilty (or not guilty) If an attorney argues that a defendant is guilty, but provides no evidence to support that argument, the jury would certainly have reasonable doubts about the guilt of the defendant. Conversely, if an attorney presents evidence, but no supporting argument explaining its relevance, the jury would have difficulty deciding how the evidence relates to the defendant. The goal-structured assurance case is similar. There is evidence (such as test results) that a property of interest (such as safety) holds. Without an argument as to why the test results support the claim of safety, however, an interested party could have difficulty understanding its relevance or sufficiency. With only a detailed argument of system safety but no test results, it would again be hard to establish the system’s safety. A goal-structured assurance case therefore specifies a claim regarding a property of interest, evidence that supports that claim, and a detailed argument explaining how the evidence supports the claim. An assurance case differs from a legal case in one significant respect. In a legal case there are advocates for both sides (the defendant is guilty; the defendant is not guilty). Typically, an assurance case is only used to show one point of view—the system is safe (and not the system is unsafe.) Figure 1: A notional safety case Figure 1 above shows a notional safety case in Goal Structuring Notation (GSN). The case argues that the system is safe because both of hazards A and B have been eliminated. The elimination of those hazards is supported by evidence Ev1, Ev2, and Ev3. The evidence might be test results, analysis, modeling, simulation, etc. Support for the use of assurance cases in the United States is nascent but increasing: The FDA has taken steps towards requiring their use when a device manufacturer submits a medical device for approval. NASA suggested the use of assurance cases as a part of development of the Constellation system. The Aerospace Corporation used assurance cases to help with a GPS satellite ground station replenishment program. In Europe and the United Kingdom, the use of assurance cases is more common. They are required in systems as diverse as flight control systems, nuclear reactor shutdown systems, and railroad signaling systems. Whether required or not, creating an assurance case is not just a pro forma exercise. Assurance cases provide many benefits. For instance, the act of creating an assurance case helps stakeholders establish a shared understanding of issues that contribute to safety. Once complete, an assurance case provides a reviewable artifact documenting how safety is achieved in the system. The assurance case is also useful as an artifact that a reviewer or regulator can use to understand and make judgments about the system. The SEI has an active research program in assurance cases, and in particular how to know that an assurance case adequately justifies its claim about the subject system, which is actually a classic philosophical problem: determining the basis for belief in a hypothesis when it is impossible to examine every possible circumstance covered by the hypothesis Our exploration has led us to examine theories from philosophy, law, rhetoric, mathematics, and artificial intelligence—as well as the vibrant assurance case community—as sources of relevant material to develop a theory of argumentation. The theory we are developing uses Baconian probability and eliminative induction to help eliminate sources of reduced confidence in a claim. Subsequent posts in this series will discuss our research into what we are calling argumentation theory. The theory will help make it easier to construct, modify, and evaluate assurance cases that are believable and will also make it possible to determine how much justified confidence one should have in it. In the next post we will begin to discuss these subjects. In the meantime we welcome your comments on the topic of assurance cases below. Additional Resources To read the SEI technical report Toward a Theory of Assurance Case Confidence, please visit http://www.sei.cmu.edu/reports/12tr002.pdf To read the SEI technical note Towards an Assurance Case Practice for Medical Devices, please visit http://www.sei.cmu.edu/reports/09tn018.pdf To read the report Software for Dependable Systems: Sufficient Evidence? please visit http://www.nap.edu/catalog/11923.html To read the dissertation Arguing Safety—A Systematic Approach to Safety Case Management by Timothy Patrick Kelly, please visit http://www-users.cs.york.ac.uk/~tpk/tpkthesis.pdf To read the report Ministry of Defence, Defence Standard—0056: Safety Management Requirements for Defence Systems—Part 2, please visit  https://www.dstan.mod.uk/standards/defstans/00/056/02000400.pdf

By Mike McLendon, Associate DirectorSoftware Solutions Division Software is the principal, enabling means for delivering system and warfighter performance across a spectrum of Department of Defense (DoD) capabilities. These capabilities span the spectrum of mission-essential business systems to mission-critical command, control, communications, computers, intelligence, surveillance, and reconnaissance (C4ISR) systems to complex weapon systems. Many of these systems now operate interdependently in a complex net-centric and cyber environment. The pace of technological change continues to evolve along with the almost total system reliance on software. This blog posting examines the various challenges that the DoD faces in implementing software assurance and suggests strategies for an enterprise-wide approach. Over the past decade, the DoD has been increasingly challenged to develop, implement, and continually evolve comprehensive enterprise software assurance  policies, guidance, and infrastructure capabilities to anticipate the impact of technological change and the dominance of software. As a result, there is significant uncertainty about DoD’s institutional software assurance capability to achieve the level of confidence that software functions as intended and remains free of vulnerabilities across legacy systems and systems being acquired. Although the DoD has taken various initiatives to examine software assurance issues and craft comprehensive assurance strategies, this uncertainty and its impact on warfighter performance persist as a major problem. Congressional Concerns about Software Assurance Congress has become increasingly concerned about the DoD’s progress in implementing its software assurance strategies and capabilities. For example, the National Defense Authorization Act (NDAA) for FY 2011 (Section 932) mandated that the Secretary of Defense develop and implement a strategy for assuring the security of software and software-based applications for all covered systems by no later than October 1, 2011. The FY 2012 NDAA was the first law to provide strong policy guidance to secure both new and legacy software from attack throughout the software development lifecycle. More recently, Section 933 of the FY 2013 NDAA directs the DoD to implement a baseline lifecycle software assurance policy. This policy requires the use of appropriate, automated vulnerability analysis tools in computer software code.  These tools are intended for using during development, operational testing, and operations and sustainment phases, through retirement for specific types of systems. The Scope of the DoD Software Assurance Challenge The DoD faces several challenges in creating and implementing an institutional software assurance capability for systems in acquisition, as well as legacy systems. Such a capability must be multidimensional  (including policy, guidance, process, practice, tool, and workforce concerns) to encompass reliability, security, robustness, safety, and other quality-related attributes critical to achieving the level of confidence that software functions as intended and is free of vulnerabilities. This software assurance capability must be broad in scope and rigorous in its discipline, but with enough flexibility and adaptability to address the challenges discussed below.  Software assurance policies and capabilities should span a spectrum of system and use-case challenges. These policies and capabilities need to account for the fact that the DoD maintains a diverse and complex systems portfolio including business and enterprise network information systems; modeling and simulation; automated tools for design, test, and manufacturing; complex C4ISR; and autonomous, semi-autonomous, and manned systems. This diverse and complex systems portfolio must operate in a net-centric cyber environment where each system acts as an information node in one or more networks. Although DoD systems operate in this environment, they are overwhelmingly acquired as individual systems, rather than being managed in a systems-of-systems or portfolio context for acquisition and sustainment. More than 100 systems alone are classified as major defense acquisition programs. The software assurance use case challenge also spans legacy systems that may be in operation and sustainment for multiple decades. During their lifetime, these systems undergo continuous evolution to meet warfighter performance needs and to continue to operate in the net-centric environment.  Another issue that must be considered in formulating software assurance capabilities and policies is the limited visibility at the DoD level of the size and demographics of the total software inventory. As a result, there is no ongoing analysis of this ever changing inventory to inform software acquisition and sustainment enterprise decisions as well as assurance policy, program, and investment decisions. Further complicating matters is the fact that the software supply chain is not well integrated vertically in terms of prime-to-subcontractor flow-down of software assurance requirements, independent verification and validation (IV&V), and test and evaluation to enable consistent visibility, traceability, and integrated testing. The myriad of assurance type acquisition requirements creates a confusion of stove-piped policies that sustainment and acquisition program managers must navigate, understand, and make actionable. For example, government and industry acquisition program managers face a variety of policies and requirements for information assurance, software assurance, trusted systems, program protection planning, anti-tamper, and the like. The acquisition community is therefore often confused about what these and other  various terms mean, why they are important, what needs to be done and how, and, finally, how to evaluate achievement of the desired outcome. Another challenge is that the DoD’s current assurance policies are the result of incremental changes over many years, rather than originating from a coherent and integrated strategy that recognizes the rapid pace of technology and software change. As a result, there is a need for an overarching, integrated assurance framework (ideally evidence-based) that can be continually adapted to address emerging needs and that communicates more effectively to officials who plan and execute acquisition and sustainment programs. Finally, the DoD’s sustainment and acquisition communities are decentralized geographically and organizationally with limited, on-going visibility at the DoD enterprise level of the state of the software assurance infrastructure (including tools, practices, and workforce) to inform policy and resource decisions.  Towards a Strategic DoD Software Assurance Approach The DoD must deal with a critical strategy issue in addressing the software assurance challenge. DoD’s approach relies on a decentralized approach to software assurance. In this approach, each program (acquisition and legacy) addresses software assurance (and overall system assurance) within the acquisition and contract strategy for that specific program. This approach can enable the proliferation of an ever-increasing variety of approaches to software assurance for individual systems that must operate in a systems-of-systems environment.  In 2010, the DoD initiated the Program Protection Plan (PPP) requirement for acquisition programs to consolidate in one document a number of existing assurance-type policy requirements. This policy is certainly a step in the right direction to elevate the criticality of assurance in all phases of the system lifecycle.  However, the policy allows each program to satisfy assurance requirements in different ways and relies on idiosyncratic (and potentially conflicting) software assurance tools and practices of multiple contractors. The persistence of concerns and their consequences drives the need to consider the merits of creating and organizing the DoD software assurance policy and infrastructure around a standard infrastructure. This infrastructure should include policies, contract requirements, practices, tools, and a workforce that is continually refreshed. This infrastructure should also be continually evaluated for its value to achieve consistency across programs, within the DoD’s software sustainment organizations, and the SoS network environment. This shift to an enterprise software assurance approach should include creating a meta-assurance framework that synthesizes myriad legacy assurance models into a more technologically current model. This model should effectively communicate to acquisition and sustainment managers what must be done; how to implement it; and then how to evaluate outcomes.  An effective enterprise software assurance approach should also include the means to continually and comprehensively assess and improve the state of the art of the DoD’s software assurance infrastructure to develop a baseline of past, current, the gaps, and then identify the impacts of emerging technology.  Moreover, the DoD needs enterprise strategy, plan, and performance measures for the inventory and continual analysis of the software portfolio to inform corporate decisions about software assurance policies, programs, and investments. In addition to these policy frameworks and assessments measures, the DoD also needs to enhance workforce competencies for software assurance within the context of a broad- based enterprise strategy to improve software acquisition management policy, practices, and competencies.  Since these workforce issues cannot entirely be addressed by applying existing best practices, a deliberate and executable DoD research and development investment strategy is also needed to advance capabilities in software assurance and vulnerability detection to address infrastructure gaps and future needs. This investment strategy should consider the following technical areas: Architecture and composition principles that enable separate evaluation of individual components with the possibility of combining results to achieve aggregate assurance judgments. These principles are motivated by the reality of modern software supply chains, which are rich and diverse in sourcing and geography. Modeling and analytics to support the diversity of quality attributes significant to DoD and infrastructural systems including modeling, simulation, and design techniques that support critical security attributes. Exploration of development approaches that incorporate creation of evidence in support of assurance claims into the process of development. This evidence-based assurance can harmonize incentives to create designs and implementations that can more readily support evaluation. Evaluation and other techniques to support the use of more opaque components in systems, including binary components and potentially dangerous components from unknown sources. Summary Software assurance is a complex domain that is one element of a larger mission assurance framework needed by the DoD to encompass reliability, security, robustness, safety, and other quality-related attributes. Although software assurance nests with the DoD’s overall software strategy, its related policies, plans, and infrastructure have not kept pace with the impacts of advancing technology and reliance on software to achieve warfighter performance. To address critical software assurance challenges, the DoD must shift to an enterprise led and managed infrastructure. Additional Resources To download a PDF of the report Critical Code: Software Producibility for Defense, please go towww.nap.edu/catalog.php?record_id=12979.  To read the SEI report Making the Business Case for Software ASsurance, please go to http://www.sei.cmu.edu/library/abstracts/reports/09sr001.cfm. Acquisition and sustainment workforce competencies for software assurance are a critical element in creating a robust infrastructure.For an overview of the software assurance competency domain, please go to http://www.sei.cmu.edu/library/abstracts/reports/13tn004.cfm.

By Douglas C. SchmidtPrincipal Researcher Department of Defense (DoD) program managers and associated acquisition professionals are increasingly called upon to steward the development of complex, software-reliant combat systems. In today’s environment of expanded threats and constrained resources (e.g., sequestration), their focus is on minimizing the cost and schedule of combat-system acquisition, while simultaneously ensuring interoperability and innovation. A promising approach for meeting these challenging goals is Open Systems Architecture (OSA), which combines (1) technical practices designed to reduce the cycle time needed to acquire new systems and insert new technology into legacy systems and (2) business models for creating a more competitive marketplace and a more effective strategy for managing intellectual property rights in DoD acquisition programs. This blog posting expands upon our earlier coverage of how acquisition professionals and system integrators can apply OSA practices to decompose large monolithic business and technical designs into manageable, capability-oriented frameworks that can integrate innovation more rapidly and lower total ownership costs. Collapsing Stove-Pipes with Technical Reference Frameworks DoD programs have struggled for decades to move away from stove-piped solutions towards common operating platform environments (COPEs). An earlier post in this series described the drawbacks of stove-piped solutions, which lock the DoD into a small number of system integrators, each devising proprietary point solutions that are expensive to develop and sustain over the lifecycle. Although stove-piped solutions have been problematic (and unsustainable) for years, the budget cuts occurring under sequestration have motivated the DoD to reinvigorate its focus on identifying alternative means to drive down costs, create more affordable acquisition choices, and improve acquisition program performance. The recently released DoD Better Buying Power 2.0 policy initiative, which superseded Better Buying Power 1.0, is one such alternative. Critical to the success of the Better Buying Power initiatives is the focus on the OSA paradigm, which includes a continually evolving integrated business and technical program management strategy that employs modular design, publishes consensus-based standards for key interfaces, embraces transparency, invigorates enterprise innovation practices, and shares risk via systematic reuse of software/hardware capabilities and assets. OSA technical practices—which are the focus of this blog posting—foster competition and innovation through published open interfaces, protocols, and data models; open standards; full-design disclosure; modular components; and intentionally-defined system structures and behaviors. Likewise, OSA business models—which will be the focus of later blog postings in this series—use open competition to expand the pool of defense contractors that can participate in DoD acquisition program contracts, thereby lowering costs and invigorating innovation.   The SEI is helping the DoD craft its OSA strategy and produce an implementation plan aimed at helping program managers deliver better capability to warfighters, given the fiscal constraints of sequestration.  A working group has been established to help the DoD move away from stove-piped software development models to COPEs that embody OSA practices. As part of this effort, I am serving as co-lead of a task area on "published open interfaces and standards" that is seeking to avoid vendor lock-in, encourage competition, and spur innovation by defining a limited number of technical reference frameworks. These frameworks are integrated sets of competition-driven, modular components that provide reusable combat system architectures for families of related warfighting systems. Key tenets of technical reference frameworks include improving the quality and affordability of DoD combat systems via the use of modular, loosely coupled, and explicitly-articulated architectures that provide many shared capabilities to warfighter applications Examples of these shared capabilities include data- and net-centricity, utility computing, situational awareness, and location awareness. fully disclosing requirements, architecture, and design specifications and development work products to program performers The goal of full disclosure is to ensure that competitors and small businesses have sufficient information to implement drop-in replacements for key system modules. enabling systematic reuse of software and software artifacts, such as objects, components, services, data models, and associated metadata and deployment plans In a well-honed systematic reuse process, each new project or product leverages time-proven architectures, designs and implementations, only adding new code that's specific to a particular application or service. mandating common components based on published open interfaces and standards supported by a range of open-source (access to source code) and closed-source (no access to source code) providers Open interfaces and standards are essential to spur competition and avoid being locked in/by a specific technology and/or vendor. achieving interoperability between hardware and/or software applications and services via common protocols and data models These common protocols and data models simplify data interchange and exchange between components from different suppliers or components implemented using different technologies. amortizing the effort needed to create conformance and regression test suites that help to automate the continuous verification, validation, optimization, and assurance of functional requirements and non-functional requirements These automated test suites ensure that developers need not wait until final system integration to perform critical functional and quality-of-service evaluations, nor must they expend signifcant effort (re)creating these tests manually. Limiting the number of these technical reference frameworks will increase interopera-bility and reuse opportunities, leading to cost savings across the enterprise and across the lifecycle.   The challenge, of course, is to define guidance that helps program managers and associated acquisition professionals systematically apply technical reference framework tenets to address technical, management, and business perspectives in a balanced manner. This blog posting—together with others we have planned—describes how advances in DoD combat system architectures lay the groundwork for success with technical reference frameworks.  The Architectural Evolution of DoD Combat Systems The technical reference framework approach described above has been applied with varying degrees of maturity and success to DoD combat systems over the past several decades. I, along with fellow SEI researcher Don Firesmith and Adam Porter from the University of Maryland’s Department of Computer Science, have been documenting the evolution of DoD combat systems with respect to their adoption of systematic reuse and the OSA paradigm described above, as shown in the following diagram.    (To view a larger image of this graphic, click on the image.) This diagram shows eight distinct stages along the evolutionary continuum of DoD combat systems. The ad hoc architectures in the columns on the left are highly stove-piped and exhibit little or no shared capabilities among various warfighter capabilities, such as communications, radars, launchers, etc. The increasingly advanced architectures on the right are intentionally designed to share many capabilities at different levels of abstraction in combat systems, including infrastructure capabilities, such as internetworking protocols, operating systems, and various layers of middleware services, such as identity managers, event disseminators, resource allocators, and deployment planners   common data and domain capabilities, such as trackers, interoperable data models, and mission planners involving battle management, control, and interaction with sensors and weapons in C4ISR systems external interfaces over the global information grid (GIG) to external weapon systems, as well as sources and users of information In practice, production combat systems vary in terms of their progression along the continuum shown in the figure above.  This visualization simply provides a birds-eye view of the design space of DoD combat systems with respect to architectural evolution. What’s Ahead This blog posting is the latest in an ongoing series that describes how the SEI is helping acquisition professionals and system integrators apply OSA principles and practices to acquire and sustain innovative warfighting capabilities at lower cost and higher quality.  Upcoming postings in this series will describe the other stages of DoD combat system architectural evolution shown in the diagram above. Subsequent posts will then explore a research effort to help one Navy program obtain accurate estimates of the cost savings and return on investment for both development and lifecycle of several product lines that were built using a common technical reference framework. Additional Resources To read the SEI technical report, A Framework for Evaluating Common Operating Environments: Piloting, Lessons Learned, and Opportunities, by Cecilia Albert and  Steve Rosemergy, please visit http://www.sei.cmu.edu/library/abstracts/reports/10sr025.cfm To read the SEI technical note, Isolating Patterns of Failure in Department of Defense Acquisition, by Lisa Brownsword, Cecilia Albert, David Carney, Patrick Place, Charles (Bud) Hammons, and John Hudak, please visithttp://www.sei.cmu.edu/library/abstracts/reports/13tn014.cfm To read the SEI technical report, The Business Case for Systems Engineering Study: Results of the Systems Engineering Effectiveness Survey, by Joseph Elm and Dennis Goldenson, please visithttp://www.sei.cmu.edu/library/abstracts/reports/12sr009.cfm To read the SEI technical report, Quantifying Uncertainty in Early Lifecycle Cost Estimation (QUELCE), by Robert Ferguson, Dennis Goldenson, James McCurley, Robert Stoddard, David Zubrow, and Debra Anderson, please visit http://www.sei.cmu.edu/library/abstracts/reports/11tr026.cfm To read the handbook, QUASAR: A Method for the Quality Assessment of Software-Intensive System Architectures, by Donald Firesmith, please visit http://www.sei.cmu.edu/library/abstracts/reports/06hb001.cfm To read the SEI technical report, Lessons Learned from a Large, Multi-Segment, Software-Intensive System, by John Foreman and Mary Ann Lapham, please visit http://www.sei.cmu.edu/library/abstracts/reports/09tn013.cfm To read the SEI technical report, Resource Allocation in Dynamic Environments, by Jeffrey Hansen, Scott Hissam, B. Craig Meyers, Gabriel Moreno, Daniel Plakosh, Joe Seibel, and Lutz Wrage, please visit http://www.sei.cmu.edu/library/abstracts/reports/12tr011.cfm To read the SEI technical report, Incremental Development in Large-Scale Systems: Finding the Programmatic IEDs, by Charles (Bud) Hammons, please visit http://www.sei.cmu.edu/library/abstracts/reports/09tn015.cfm

By Eric WernerChief ArchitectSEI Emerging Technology Center The power and speed of computers have increased exponentially in recent years. Recently, however, modern computer architectures are moving away from single-core and multi-core (homogenous) central processing units (CPUs) to many-core (heterogeneous) CPUs. This blog post describes research I’ve undertaken with my colleagues at the Carnegie Mellon Software Engineering Institute (SEI)—including colleagues Jonathan Chu and Scott McMillan of the Emerging Technology Center (ETC) as well as Alex Nicoll, a researcher with the SEI’s CERT Division—to create a software library that can exploit the heterogeneous parallel computers of the future and allow developers to create systems that are more efficient in terms of computation and power consumption.  As we look at computing trends in data centers available to developers, the move towards many-core heterogeneous CPUs shows no sign of abating. The majority of computers (such as smartphones and other mobile devices) contain heterogeneous hardware with multi- and many-core chips. Many existing software libraries, frameworks, and patterns, however, were not developed for large-memory, many-core, heterogeneous computing environments. Since software developers often aren’t accustomed or trained to write software for many-core architectures, new hardware architectures aren’t being used to their potential. Complicating matters even further is the fact that many common software libraries for these environments are not designed for ease of use, but rather for efficient and optimal computing. Unfortunately, software developers haven’t received the training necessary in parallel programming algorithms to best leverage the capabilities of these new architectures. Foundations in Moore’s Law Our research approach traces its foundations back to Moore’s Law. Many software and systems engineers abbreviate Moore’s Law as stating that over the history of hardware, the processor speed on integrated circuits doubles every two years. Moore’s Law actually states that the transistor density on microchips doubles every 18 months. In recent years, however, CPU manufacturers have focused less on clock speed and more on multi-core and special-purpose cores with deeper memory architectures. "The performance of individual computer processors increased on the order of 10,000 times over the last two decades of the 20th century without substantial increases in cost and power consumption," noted Samuel Fuller and Lynn Millet in The Future of Computing Performance. Fuller and Millet also advocate, "Future growth in computing performance will have to come from parallelism. Most software developers today think and program by using a sequential programming model to create software for single general-purpose microprocessors." From Gaming to High-Performance Computing: Graph Analytics for Everyday Users While heterogeneous, multi-core architectures were once largely seen in gaming systems, the high-performance computing (HPC) community has also migrated to heterogeneous, multi-core architectures. These architectures allow for high-level computations, as well as three-dimensional physics simulations. While still a specialty field, the architectures witnessed in HPC systems will soon be widely available to the everyday user. In June of this year, the International Supercomputing Conference released the Top500 supercomputer list. According to an article on the Top500 list posted on AnandTech, "the number of hybrid systems is actually down—from 62 on the last list to 54 now—but on the other hand the continued improvement and increasing flexibility of GPUs and other co-processors, not to mention the fact that now even Intel is involved in this field, means that there’s more effort than ever going into developing these hybrid systems." One phase of our research involves using HPC architectures to simulate future computer architectures and develop software libraries, best practices, and patterns that can be used by a broad community of software developers. Initially, we limited our focus to graph analytics, which are algorithms that operate on graphs, which do not have locality of reference, making it hard to parallelize operations on them. Graph analytics are widely used in government, commerce, and science and can highlight relationships that might be obscured by data. One example of a system that can be represented as a graph is a social network, where the individual components are people and the connections they form represent social relationships. For a reference platform, we are relying on the Graph500, an international benchmark started in 2010 that rates how fast HPC systems, test, traverse, and navigate a graph.  Graph 500 is a benchmark similar to the Top500 referenced above.  The Graph 500 is specifically designed to test graph algorithms because they’re fundamentally different from easily parallelizable algorithms.  We’re starting with the algorithms defined by the Graph 500, and we’re using that framework as a starting point. Identifying and Validating Design Patterns With the understanding that the development of patterns is primarily a bottom-up endeavor, we initially focused on reviewing patterns developed for homogenous, HPC patterns. These patterns will be culled from those developed by ETC researchers, as well as our collaborators in government, academia, and industry. Validation is a critical process in this phase and we are using two, independent technical validation mechanisms: We are beta-testing the software library with software engineers and have them use the library to generate novel graph analytical code for advanced computing architectures We are soliciting feedback from real-world users and plan to present the patterns that we reviewed at relevant upcoming technical conferences. The next phase of our work will focus on culling the homogenous HPC patterns that we audited to develop a library of templates and patterns that software developers, architects, and technology planners will use to effectively access and exploit future computing architectures. As stated previously, a greater utilization of resources means faster computation and possibly more efficient use of resources. A Collaborative Approach In this early phase of our work, we have been collaborating with researchers at Indiana University’s Extreme Scale Computing Lab, which developed the Parallel Boost Graph Library. In particular we are working with Andrew Lumsdaine who serves on the Graph 500 Executive Committee and is considered a world leader in graph analytics. Addressing the Challenges Ahead We recognize that even if we achieve all of our milestones, our research will not yield a silver bullet. Programmers need niche skills to address some of the problems of multiple architectures. Our approach focuses on reducing the time that developers  need to spend solving the problem of programming for heterogeneous architectures rather than fighting the computer with the hardware as the problem. This challenge problem aligns with Emerging Technologies Center (ETC)’s mission, which is to promote government awareness and knowledge of emerging technologies and their application and to shaping and leverage academic and industrial research. We hope our research will enable programmers in government and industry to use the library of templates and design patterns that we develop to produce effective software for future computing systems. Our next step involves releasing this library to our stakeholders in the DoD, and to other users, via an open-source platform to enable them to effectively access and exploit future computing architectures. While our initial phase of research focused on graph analytics in graphics processing units (GPUs), we will also investigate other hardware platforms in the future, including field-programmable gate arrays (FPGAs).  We plan to develop a library that separates the concerns of graph analysis from the details of the underlying hardware architecture. Future work will focus on graphs, but add new hardware architectures to the mix such as FPGA and potentially distributed platforms. If you are interested in collaborating with us on this research, please leave a comment below or send an email to info@sei.cmu.edu. Additional Resources For more information about the Emerging Technologies Center, please visithttp://www.sei.cmu.edu/about/organization/etc/index.cfm

By Will Casey Senior Member of the Technical StaffCERT Division Exclusively technical approaches toward attaining cyber security have created pressures for malware attackers to evolve technical sophistication and harden attacks with increased precision, including socially engineered malware and distributed denial of service (DDoS) attacks. A general and simple design for achieving cybersecurity remains elusive and addressing the problem of malware has become such a monumental task that technological, economic, and social forces must join together to address this problem. At the Carnegie Mellon University Software Engineering Institute’s CERT Division, we are working to address this problem through a joint collaboration with researchers at the Courant Institute of Mathematical Sciences at New York University led by Dr. Bud Mishra. This blog post describes this research, which aims to understand and seek complex patterns in malicious use cases within the context of security systems and develop an incentives-based measurement system that would evaluate software and ensure a level of resilience to attack. In March of this year, an attacker issued a DDoS attack that was so massive, it slowed internet speeds around the globe. Known as Spamhaus/Cyberbunker, this attack clogged servers with dummy internet traffic at a rate of about 300 gigabits per second. By comparison, DDoS attacks against banks typically register only 50 gigabits per second, according to a recent article in Business Week. The Spamhaus attack came 13 years after the publication of best practices on preventing DDoS attacks, and it was not an isolated event. The latest figures indicate that cyberattacks continue to rise. Research from the security firm Symantec indicates that in 2012 targeted cyber attacks increased by 42 percent. How is this possible?  In part, existing technologies facilitate the role of attacker over the role of defender, since in this hide-and-seek game, the tricks to hide an attack are many, whereas the techniques to seek them are meager and resource intensive.  In the SEI CERT Division our work aims at going beyond simply detecting strategic and deceptive actions of an attacker, by reversing the very incentives that ultimately make more transparent the choices made in hide-and-seek dynamics. Attackers have incentives to find weaknesses software which facilitate system compromise. We envision the possibility that these dynamics can be reversed through an altered incentive structure, credible deterrence/threats, and powerful measurement systems.  For example, we may incentivize an emerging group to acquire and deploy particular expertise to evaluate software and guarantee their validity, albeit empirically using techniques from machine learning. This combination of techniques including expert systems, model checking and machine learning can ensure increased level of resilience without loss of transparency.  Moreover, game theory provides a means to evaluate the dynamics of incentives and to understand the impacts of new technologies and use cases. Deterring Malicious Use in Systems Existing proposals for deterring malware attacks rely on the isolation of an elite network with enhanced security protocols, which undermines the utility of networking and does little to deter incentives for maliciousness.  Instead, this strategy concentrates digital assets in one place, putting all eggs in one highly vulnerable basket. Such proposals, while costly and risky, underscore the importance of introducing alternative ideas into the discussion of common information assurance goals. For example, since computer networks gather users with a variety of different interests and intents, we may wish to incentivize computer users to take steps that will compel them to reassure other users that they have not been maliciously compromised. To obtain this assurance we may leverage the work of technical and security experts, which involves sophisticated software vulnerably probing techniques (such as fuzz testing) and trust mechanisms (such as trusted hardware modules), etc.  With these assurances we demonstrate the possibility of economic incentives for software adopters to have deeper and clearer expectations about a network’s resilience and security. Foundations in Game Theory Many of the ideas in our approach can be traced back to John von Neumann, a Princeton University mathematician who, with his colleague Oskar Morgenstern, created the basic foundations of modern game theory, which studies how rational agents make strategic choices as they interact. An example of one such strategic choice is the concept of mutual assured destruction (MAD), which describes a doctrine thata war in which two sides would annihilate each other would leave no incentive for either side to start a war. Once the two sides have come to such a mutually self-enforcing strategy, neither party will deviate as long as the opponent does not. Such a state-of-affairs is described in game-theory by the concept of Nash equilibrium. We aim to cast the cyber-security problem in a game-theoretic setting so that every "player" will choose to be honest, will check that they are honest and not an unwitting host for malware, can prove to others that they are honest and accept confidently the proofs that others are honest and not acting deceptively. A Collaborative Approach Through our collaboration with the research team at NYU—Dr. Mishra and Thomson Nguyen—we can access their extensive knowledge in game theory, pattern finding, model checking, and machine learning. We are also collaborating with Anthony Smith of the National Intelligence University. This collaboration will allow us to access Smith’s advanced knowledge in network security science and technology. Researchers from the SEI’s CERT Division involved in the project include Michael Appel, Jeff Gennari, Leigh Metcalf, Jose Morales, Jonathan Spring, Rhiannon Weaver , and Evan Wright. Building on the deep domain knowledge from CERT about the nature and origin of malicious attacks and how often those attacks occur, our collaboration with the Courant Institute will provide a better understanding of the implications of such attacks in a larger system. The theoretical framework for this approach is based on model checking and follows strategies similar to what Dr. Mishra developed for hardware verification as a graduate student at CMU. One of our preliminary tasks was to develop the mathematical frameworks to describe vulnerabilities including attack surface, trace data, software errors and faults, and malicious traces. The ability to rigorously define these patterns allowed us to formalize and detect a large class of malicious patterns as they transfer from user to user. As an interesting use-case, we focused on several critical patterns to identify malicious behaviors in traces. By studying the Zeus Trojan horse, which was used to steal users’ banking information, we were able to identify atomic actions that allow malicious users to persist on a system and compromise their web browsers. The enhanced system that we are proposing will additionally provide some degree of guaranteed resilience. When fully implemented, our approach will provide three key benefits to our stakeholders in government and industry; a well-understood theoretical model of interactions among benign and malicious users, providing a more accurate picture of forces (technological, economic, political and social) that shape the security landscape a scalable method for malware mitigation, including an adaptive system that can identify and address new threats a transparent mechanism to vet commercialized software, which touches upon our notions of trusted computing at multiple levels,  from firmware to Android applications. Measures for Resilience to Malicious AttacksOur new system has many attractive features: it does not simply stop after identifying a network attack. Instead, it motivates and enables deployment of measures of weaknesses using practical techniques such as vulnerability assessments for servers, fuzz testing binaries for weaknesses, and verifying adherence to best practice. These measures provide decision makers and users alike with means to adapt best practices and keep the network operational. Consequently, the system’s designers will also better understand what security features are needed in response to current threats and attack methods.  Many software vulnerabilities result from implicit assumptions made at the time of design. While it may not be possible to anticipate all the attacks against a design, we can begin to measure and minimize the time it takes for designers to respond to current attacks within our frame work of measures. In summary, we believe the proposed system will not just deter malicious attackers, but will also motivate users to ensure that their computers and mobile devices have not been purposefully or unintentionally compromised.  In addition designers will benefit from adding in security as user demands for security increase. Challenges There is no widely accepted definition of what constitutes malicious behaviors stated in a way that can be understood and guarded against by average users. We need to be able to help average users and those in government and industry gain a better understanding of whether behavior is malicious or benign. A simpler and much more popular concept used in the physical context is trust. If trust is perceived to be a valuable economic incentive in the cyber context, and users can assess whether they can trust a server or a software application, then a trust-based technique can be used and can benefit a diverse group of users, ranging from individual users to personnel in industry and government. Our approach will be especially powerful in trusted computing situations, where trust may be based on crytopgraphic signatures that validate the source code that operates a device. Granted that complete certainty may still be elusive. Note that users can entertain some assurance about the health of their network, because a third party can verify and certify that all are components are trustworthy and are behaving well. Our Next Steps Our next step is to check our design, as well as our implicit assumptions about how individuals behave in this framework. To this end, we have been developing a system that we can simulate in silico—initially aimed at understanding only the incentives to attack and counter attacks with mitigation —so that we can better understand how the individuals strategize to select equilibrium (a strategy profile from which no single player is incentivized to deviate). In the future, we plan to invest in deep, multi-trace modeling, extending the game theory to include temporal patterns of attacks on software and systems, which will involve simulation modeling and model checking.  By simulation modeling we can estimate the resource needs, overheads and other requirements of the system for practical deployments. Additional Resources For more information about the work of researchers in the SEI’s CERT Division, please visit www.cert.org For more information about New York University’s Courant Institute of Mathematical Sciences, please visithttp://www.cims.nyu.edu/

Examples From Real-World ProjectsBy Stephany BellomoSenior Member of the Technical StaffSoftware Solutions Division Agile projects with incremental development lifecycles are showing greater promise in enabling organizations to rapidly field software compared to waterfall projects. There is a lack of clarity, however, regarding the factors that constitute and contribute to success of Agile projects. A team of researchers from Carnegie Mellon University’s Software Engineering Institute, including Ipek Ozkaya, Robert Nord, and myself, interviewed project teams with incremental development lifecycles from five government and commercial organizations. This blog posting summarizes the findings from this study to understand key success and failure factors for rapid fielding on their projects. A key area in our interviews explored how Agile projects deal with the pressure to rapidly deliver high-value capability while maintaining project speed (delivering functionality to the users quickly) and product stability (providing reliable and flexible product architecture). For example, due to schedule pressure, we often see a pattern of high initial velocity (the amount of product backlog effort a team can handle in one sprint) for weeks or months, followed by a slowing of velocity due to stability issues. Business stakeholders often find these changes in velocity disruptive since the rate of capability delivery slows while the team addresses stability problems. We found that when experienced practitioners were faced with these challenges, they did not apply Agile practices in a silo. Instead they combined practices—Agile, architecture, or other—in creative ways to respond quickly to unanticipated stability problems, as described below. Balancing Speed and Stability The ability to balance speed and stability involves achieving and preserving a software development state that enables teams to deliver releases that stakeholders value at a tempo that makes sense for their business.  The desired software development state is different for each organization. This state is one in which architecture (often in the form of platforms and application frameworks), supporting tool environments, practices, processes, and team structures exist to support efficient and sustainable development of features. The entire organization—including development teams, management, and other stakeholders—must have visibility into the desired state, so that they neither over-optimize the supporting development infrastructure nor quit working on it. In organizations that operate in highly regulated environments, such as defense, avionics, financial services, and health care, software development teams often interact with system engineering, deployment, and quality assurance teams that may be operating under different tempos. One challenge that software development teams face is that these competing forces often result in a significant slowdown in delivery following a high initial velocity. When confronted with this slowdown, the organizations that we interviewed applied a variety of tactics and practices to get back to the desired state. For example they often applied Agile practices in combination with other practices, especially architecture practices, to rapidly field projects. A New View of Agile Many software developers steadfastly maintain that Agile requires small, co-located teams, downplays architectures, and provides no documentation. The reality is that organizations—especially those faced with the challenge of rapidly fielding software systems in highly-regulated environments—have been applying varied architecture practices that build on the foundations of Scrum and Extreme Programming (XP). The approaches revealed by our analysis of the interviews we conducted fall into three categories: When the project was going well, teams described using foundational Agile practices, such as Scrum status meetings, Scrum collaborative management style, small dedicated teams and limited scope, continuous integration, test-driven development, and so on to enable rapid development.  When teams described a problem, they would explain how they often combine Agile practices with architecture and other practices ranging from management to engineering to get back to desired state. Examples of these combined practices include release planning with architectural considerations, prototyping with a quality attribute  focus, release planning with external dependence management, test driven development with quality attribute focus, and technical debt monitoring with quality attribute focus.  Development projects incur technical debt when shortcuts (intended or otherwise) lead to degraded quality. We also collected some examples of inhibiting factors that prevented developers from rapidly delivering software products. These included slow business decision-making processes, limitations in measuring architectural technical debt, over-dependency on architecture for knowledge, and stability-related efforts that are not entirely visible to the business. Below are a few examples of Agile architecture practices that enable speed and stability that emerged from our interviews: Release planning with architecture considerations. This practice extends the feature release planning process by adding architectural information to the feature description document prior to release prioritization. Prototyping with a quality attribute focus. This practice extends the prototyping/demo user feedback practice commonly integrated into Scrum to include a focus prototyping on quality attributes, such as performance or security. Roadmap/Vision with External Dependency Management. This practice incorporates external dependency analysis into the roadmap planning process to reduce the risk of being blind-sided by unanticipated external changes. Test-Driven Development with Quality Attribute Focus. This practice merges test-driven practices, such as automated test-driven development and continuous integration, with a focus on runtime qualities, such as performance, scalability, and security.  Technical Debt Monitoring with Quality Attribute Focus. This practice merges the notion of tracking technical debt with a focus on quality attributes (such as performance, security, reliability) that go beyond defects and functional correctness. More examples of combined practices can be found in the paper A Study of Enabling Factors for Rapid Fielding: Combined Practices to Balance Tension Between Speed and Stability that I co-authored on this research with Ipek Ozkaya and Robert Nord. The combined practices listed above allow experienced developers to address the problem of velocity slowdown resulting from stability issues with minimal disruption to capability delivery. Over time, however, acceptance of this approach has evolved. The initial release of Scrum advocated that the practices must be applied exactly as described in the Scrum handbook (if this was not done the project was considered to have "Scrum But") syndrome, so the outcome would be questionable. In a recent blog posting, Ken Schwaber—one of the originators of Scrum—amended the initial Scrum doctrine by saying he would like to change the mindset of "Scrum But" to "Scrum And." He explained that "Scrum And" characterizes an organization that is on a path of continuous improvement in software development beyond the basic use of Scrum. In highly regulated environments, such as defense, avionics, financial services, and health care, organizations must often employ "Scrum And" approaches to balance speed and stability in an effort to meet budget, quality and timeline expectations. Inhibiting Factors Through our interviews with organizations, we also identified several factors that prevented development teams from rapidly delivering a software product. Many of these inhibiting factors were the result of incorrect or inconsistent applications of Agile or architecture practices, including (but not limited to) the following: a desire for features that limits requirements analysis or stability-related work slow business decision, feedback, or review-response time problems that resulted from challenges with external dependency management stability-related effort that was not entirely visible limitations in measuring architectural technical debt inadequate analysis, design, or proof-of-concept inconsistent testing practices and/or deficiency in quality attribute focus poor testing consistency Concluding Thoughts While agile architecture practices can help organizations assure the stability of fielded systems , it is important to understand the root causes of the inability to deliver at the expected pace and management of the tension between speed and stability. Organizations must also make problems more visible to developers, management, and stakeholders. When considering whether to combine Agile and architecture practices, organizations should consider the following questions: Are we delivering software to our customer at an expected pace? Are we aware of problems that are cropping up as a result of losing focus on architecting when Agile adoption activities become the primary focus? Does our technical roadmap address short-term and long-term issues? Does the team of software developers have skills that would enable them to successfully implement Agile and architecture? Do we have the visibility into not only the project management of the system, but also the quality expected from the system? We hope that by codifying and sharing the practices exemplified above, other organizations can learn to apply these approaches to contend with the often conflicting demands of rapidly delivering software that is reliable, stable, and flexible in a rapidly changing environment. Future posts in this series will explore other hybrid Agile approaches identified by the organizations that we interviewed including prototyping with a quality attribute focus. If you have experience applying a hybrid Agile architecture approach in your organization, please share your story with us in the comments section below. Additional Resources This article is an excerpted version of an article that appeared in the June 2013 issue of the Cutter IT Journal. For more information, please visit http://www.cutter.com/itjournal.html To read more about the SEI’s research in architectural technical debt, please visithttp://www.sei.cmu.edu/architecture/research/arch_tech_debt/ To read more about the review of architecture-centric risk factors, please read the article "Architecture for Large Scale Agile Architecture Development: A Risk-Driven Approach," which was published in the May/June edition of Crosstalk by Mike Gagliardi, Mike, Robert Nord, and Ipek Ozkaya, please visithttp://www.crosstalkonline.org/issues/mayjune-2013.html To read the article "A Study of Enabling Factors for Rapid Fielding: Combined Practices to Balance Tension Between Speed and Stability," by Stephany Bellomo, Robert Nord, and Ipek Ozkaya, please visit http://www.sei.cmu.edu/library/assets/whitepapers/ICSE2013%20Study%20of%20Rapid%20Fielding%20Practices_camera%20ready.pdf

By Will DormannSenior Member of the Technical StaffCERT Vulnerability Analysis Team Occasionally this blog will highlight different posts from the SEI blogosphere. Today we are highlighting a recent post by Will Dormann, a senior member of the technical staff in the SEI’s CERT Division, from the CERT/CC  Blog. This post describes a few of the more interesting cases that Dormann has encountered in his work investigating attack vectors for potential vulnerabilities. An attack vector is the method that malicious code uses to propagate itself or infect a computer to deliver a payload or harmful outcome by exploiting system vulnerabilities.

By Julien Delange, Senior Member of the Technical StaffSoftware Solutions Division When life- and safety-critical systems fail (and this happens in many domains), the results can be dire, including loss of property and life. These types of systems are increasingly prevalent, and can be found in the altitude and control systems of a satellite, the software-reliant systems of a car (such as its cruise control and anti-lock braking system), or medical devices that emit radiation. When developing such systems, software and systems architects must balance the need for stability and safety with stakeholder demands and time-to-market constraints. The Architectural Analysis & Design Language (AADL) helps software and system architects address the challenges of designing life- and safety-critical systems by providing a modeling notation with well-defined real-time and architectural semantics that employ textual and graphic representations. This blog posting, part of an ongoing series on AADL, focuses on the initial foundations of AADL. The SEI’s Work on AADL The SEI has been one of the lead developers of AADL and has participated on the AADL standardization committee since its inception. SEI researchers also developed the Open Source AADL Tool Environment (OSATE), which is the reference implementation for supporting AADL within the Eclipse environment and a number of AADL-based analysis capabilities. The SEI and other researchers and developers have used AADL as a foundation to design and analyze life- and safety-critical systems. Likewise, several projects have used AADL to design software architecture and analyze, validate, or improve various quality attributes, such as reliability, latency/performance, or security. While AADL was initially conceived for use in avionics, this blog series has demonstrated its applicability in other domains that rely on life- and/or safety-critical systems, including the medical domain. Likewise, AADL tools have been developed by organizations such as Rockwell Collins, the European Space Agency, Honeywell, and Ellidiss Software to provide modeling concepts that describe the runtime architecture of application systems in terms of concurrent tasks, their interactions, and their mapping onto an execution platform. Software with Reuse: AADL Beginnings Bruce Lewis, an experienced software practitioner working for the U.S. Army Aviation & Missile Research at the Development & Engineering Center (AMRDEC) in Huntsville Alabama, is a key player in the development and evolution of AADL. Lewis, who formed the AADL Standards Committee and co-drafted the first-ever requirements document for the standard, attended a standards committee meeting at the SEI’s Pittsburgh headquarters in July 2013 and described his purpose in the standardization of AADL: Originally, we were working on different reuse techniques and ways to build software with reuse. This was in the early '90s. We found that the different reuse approaches weren't really effective without an architecture description where we could actually analyze how components would fit together and also analyze the effect of putting them together. DARPA [the Defense Advanced Research Projects Agency] also was not satisfied with the current state-of-the-art for domain specific sets of components. They also wanted an architecture approach. So, I went to DARPA to talk to them about a possible solution. They pointed me to the Domain-Specific Software Architecture (DSSA) project, which was inventing the concept of an architecture description language. We began to experiment with architecture description languages for real-time, mission-, and safety-critical systems. That's how I met Peter [Feiler], who also was working on the DSSA project. DARPA offered me the opportunity to lead the research projects for the Honeywell MetaH language over several DARPA programs. MetaH became the foundation for AADL. Our primary project involved essentially reengineering then enhancing a missile system using a reference architecture that we had developed. We then developed and hosted the missile system to multiple targets on single and distributed multi-processor platforms. We verified the system flew correctly by running the environment simulation and the tactical code on the embedded processors for each processor target. In each case, the missile flew correctly after rapid ports to the new processing environment. MetaH was so effective in our army experiments that I felt like it was very important to move it to a standard and push it forward in the industry as a more powerful, more effective means for real-time system building and evolution.  Lewis and Feiler shepherded AADL into a language that could be used to ensure the reliability of other safety-critical systems in other industries including aerospace, telecommunications, and medical equipment. For these industries, the exponential growth and increasing complexity of software and systems development had become a major area of concern. A team of SEI researchers led by Feiler detailed the exponential growth and complexity involved in developing safety-critical systems in the technical report, System Architecture Virtual Integration: An Industrial Case Study. In that technical report, the authors described how aerospace software-reliant systems have experienced exponential growth in size and complexity, and also unfortunately in errors, rework, and cost, primarily due to integration issues that are detectable through architecture-centric analysis. New development of safe aircraft is reaching the limit of affordability. They also noted that the size of software systems, measured in source lines of code (SLOC), had doubled every four years since the mid-1990s and reached around 27 million SLOC of software by 2010. Rework costs were in the multi-billions of dollars. Discovering even a small percentage (10 percent) of these issues would yield a major payoff. A key challenge faced by developers and testers of aerospace software systems resides in the integration of industrial and  commercial off-the-shelf (COTS) components from multiple suppliers. Developers increase produce systems by integrating new and existing components rather than starting from scratch. This approach allows some benefits, such as reuse of existing development efforts, ease in the creation of product lines, and the reduction of component production costs. There are, however, some drawbacks to this approach: Components are loosely related and never intended to work together in the new context. Assumptions constraining usability may not be documented. Newly assembled artifacts have different and potentially conflicting requirements. From an OEM’s perspective, multiple alternative components need to be evaluated for architectural effects. Existing components were validated and tested in one system but could not work in other environments (e.g. deploying a function on different processors, different timelines, messaging infrastructures, scheduling, error handling environments, or execution runtime) All other integration and/or deployment issues engineers had already experienced during operational projects, such as memory, bus, processor utilization, data integrity, safety and security requirements, shared resource contention, end-to-end latency, jitter sensitivity, fault-tolerance requirements, execution behavior, definitions of data, functional integration correctness, and underlying assumptions internal/external to the component. These errors are likely discovered during the integration test phase, late in the development process. As a result, they increase the need for testing, introduce the need for rework and new development, increase costs, and postpone product delivery. To address the problem of integration, AADL needed to represent the architecture early and incrementally to find problems typically reported during the system integration phase. When this approach is applied during the requirements and design phase, it will result in errors that can be repaired at a time when they are less costly to fix. Lewis described the evolution of AADL to address system integration: The DARPA/Honeywell MetaH language provided very powerful concepts for architecture analysis and rapid generative integration driven by the analytical models, but the language was limited in flexibility and scope. We had 30 extensions from DARPA experiments that we knew we wanted to make plus industry and academic input on the committee. Based on his wealth of experience, Feiler became the primary AADL language architect. We knew that the AADL needed to support common analysis and integration of specifications across contractors based on precise core standard semantics for real-time systems. AADL also needed to provide flexibility for language extension for properties and annexes for specialized domains and tools. Since architectures must support evaluation of many quality attributes and requirements, AADL was developed for multi-domain analysis from a centralized model so we could detect side effects across these domains through a centralized, shared architecture model. It was also developed to support ease of incremental component refinement through extension mechanisms and to support analysis of incomplete specifications. Throughout the development process, we verify the integration of components into the virtual real-time system. During our DARPA years, we moved into the more complex and challenging aviation domain. We wanted a domain that was really challenging where we could exploit the power of architecture expression to demonstrate many emerging architectural qualities in complex systems. This proved to be well targeted and timely. The aviation industry, at the same time, was beginning to recognize that they were suffering from what were essentially failures to effectively integrate the software system architecture. Costs were extremely high, and there were many expensive programs that demonstrated the fact that we need some kind of an architectural approach that was quantitative and qualitative. This approach would help us understand the architecture as early as possible, do virtual integration, and then ultimately build the system.  Our next post in this series will describe the use of AADL to improve safety and reliability in the aerospace industry as part of the System Architecture Virtual Integration (SAVI) project. Additional Resources To view the AADL Wiki, please visit https://wiki.sei.cmu.edu/aadl/index.php/Main_Page For more information about AADL, please visit http://www.aadl.info To read the SEI technical report, System Architecture Virtual Integration: An Industrial Case Study, please visithttp://resources.sei.cmu.edu/library/asset-view.cfm?assetID=9145

By Ian GortonSenior Member of the Technical Staff Software Solutions Division(This blog post was co-authored by John Klein) New data sources, ranging from diverse business transactions to social media, high-resolution sensors, and the Internet of Things, are creating a digital tidal wave of big data that must be captured, processed, integrated, analyzed, and archived. Big data systems storing and analyzing petabytes of data are becoming increasingly common in many application areas. These systems represent major, long-term investments requiring considerable financial commitments and massive scale software and system deployments. With analysts estimating data storage growth at 30 to 60 percent per year, organizations must develop a long-term strategy to address the challenge of managing projects that analyze exponentially growing data sets with predictable, linear costs. This blog post describes a lightweight risk reduction approach called Lightweight Evaluation and Architecture Prototyping (for Big Data) we developed with fellow researchers at the SEI. The approach is based on principles drawn from proven architecture and technology analysis and evaluation techniques to help the Department of Defense (DoD) and other enterprises develop and evolve systems to manage big data. The Challenges of Big Data For the DoD, the challenges of big data are daunting. Military operations, intelligence analysis, logistics, and health care all represent big data applications with data growing at exponential rates and the need for scalable software solutions to sustain future operations. In 2012, the DoD announced a $250 million annual research and development investment in big data that is targeted at specific mission needs such as autonomous systems and decision support. For example, in Data-to-Decisions S&T Priority Initiative, the DoD has developed a roadmap through 2018 that identifies the need for distributed, multi-petabyte data stores that can underpin mission needs for scalable knowledge discovery, analytics, and distributed computations. The following examples illustrate two different DoD data-intensive missions: Electronic health records. The Military Health System provides care for more than 9.7 million active-duty personnel, their dependents, and retirees. The 15-year-old repository operates a continouously growing petascale database, with more than 100 application interfaces. System workload types include care delivery, force readiness, and research/analytics. The system does not currently meet the target of 24/7 availability. Flight data management. Modern military avionics systems capture tens of gigabytes (GBs) of data per hour of operation. This data is collected in in-flight data analysis systems, which perform data filtering and organization, correlation with other data sources, and identification of significant events. These capabilities support user-driven analytics for root cause detection and defect prediction to reduce engineering and maintenance costs. To address these big data challenges, a new generation of scalable data management technologies has emerged in the last five years.  Relational database management systems, which provide strong data-consistency guarantees based on vertical scaling of compute and storage hardware, are being replaced by NoSQL (variously interpreted as "No SQL", or "Not Only SQL") data stores running on horizontally-scaled commodity hardware. These NoSQL databases achieve high scalability and performance using simpler data models, clusters of low-cost hardware, and mechanisms for relaxed data consistency that enhance performance and availability. A challenging technology adoption problem is being created, however, by a complex and rapidly evolving landscape of non-standardized technologies built on radically different data models. Selecting a database technology that can best support mission needs for timely development, cost-effective delivery, and future growth is non-trivial. Using these new technologies to design and construct a massively scalable big data system creates an immense software architecture challenge for software architects and DoD program managers. Why Scale Matters in Big Data Management Scale has many implications for software architecture, and we describe two of them in this blog post. The first revolves around the fundamental changes that scale enforces on how we design software systems. The second is based upon economics, where small optimizations in resource usage at very large scales can lead to huge cost reductions in absolute terms. The following briefly explores these two issues: Designing for scale. Big data systems are inherently distributed systems. Hence, software architects must explicitly deal with issues of partial failures, unpredictable communications latencies, concurrency, consistency, and replication in the system design. These issues are exacerbated as systems grow to utilize thousands of processing nodes and disks, geographically distributed across data centers. For example, the probability of failure of a hardware component increases with scale. Studies such as this 2010 research paper, "Characterizing Cloud Computing Hardware Reliability," find that 8 percent of all servers in a data center experience a hardware problem annually, with the most common cause being disk failure. In addition, applications must deal with unpredictable communication latencies and network partitions due to link failures. These requirements mandate that scalable applications treat failures as common events that must be handled gracefully to ensure that the application operation is not interrupted. To address such requirements, resilient big data software architectures must: Replicate data across clusters and data centers to ensure availability in the case of disk failure or network partitions. Replicas must be kept consistent using either master-slave or multi-master protocols. The latter requires mechanisms to handle inconsistencies due to concurrent writes, typically based on Lamport clocks. Design components to be stateless and replicated and to tolerate failures by dependent services, for example, by using the Circuit Breaker pattern described by Michael T. Nygard in his book Release IT and returning cached or default results whenever failures are detected. This pattern ensures that failures do not rapidly propagate across components and allow applications an opportunity for recovery. Economics at scale. Big data applications employ many thousands of compute-and-storage resources. Regardless of whether these resources are capital purchases or resources hosted by a commercial cloud provider, they remain a major cost and hence a target for reduction. Straightforward resource reduction approaches (such as data compression) are common ways to reduce storage costs. Elasticity is another way that big data applications optimize resource usage, by dynamically deploying new servers to handle increases in load and releasing them as load decreases. Elastic solutions require servers that boot quickly and application-specific strategies for avoiding premature resource release. Other strategies seek to optimize the performance of common tools and components to maintain productivity while decreasing resource utilization. For example, Facebook built HipHop, a PHP-to-C++ transformation engine that reduced its CPU load for serving web pages by 50 percent. At the scale of Facebook’s deployment, this represents a very significant resource reduction and cost savings. Other targets for reduction are software license costs that can become cost prohibitive at scale. Cost reduction has seen a proliferation of database and middleware technologies developed recently by leading internet organizations, and many of these have been released as freely available open source. Netflix and Linkedin provide examples of powerful, scalable technologies for big data systems. Other implications of scale for big data software architectures revolve around testing and fault diagnosis. Due to the deployment footprint of applications and the massive data sets they manage, it’s impossible to create comprehensive test environments to validate software changes before deployment.  Approaches such as canary testing and simian armies represent the state of the art in testing at scale. When the inevitable problems occur in production, rapid diagnosis can only be achieved by advanced monitoring and logging. In a large-scale system, log analysis itself can quickly become a big data problem as log sizes can easily reach hundreds of GBs per day. Logging solutions must include a low overhead, scalable infrastructure such as Blitz4J, and the ability to rapidly reconfigure a system to redirect requests away from faulty services. Necessarily large investments and magnified risks that accompany the construction of massive, scalable data management and analysis systems exacerbate these challenges of scale. For this reason, software engineering approaches that explicitly address the fundamental issues of scale and new technologies are a prerequisite for project success. Designing for Scalability with Big Data To mitigate the risks associated with scale and technology, a systematic, iterative approach is needed to ensure that initial design models and database selections can support the long-term scalability and analysis needs of a big data application. A modest investment in upfront design can produce unprecedented returns on investment in terms of greatly reduced redesign, implementation, and operational costs over the long lifetime of a large-scale system. Because the scale of the target system prevents the creation of full-fidelity prototypes, a well-structured software engineering approach is needed to frame the technical issues, identify the architecture decision criteria, and rapidly construct and execute relevant but focused prototypes. Without this structured approach, it is easy to fall into the trap of chasing after a deep understanding of the underlying technology instead of answering the key go/no-go questions about a particular candidate technology. Getting the right decisions for the minimum cost should be the aim of this exercise.  At the SEI, we have developed a lightweight risk reduction approach that we have initially named Lightweight Evaluation and Architecture Prototyping (for Big Data), or LEAP(4BD). Our approach is based on principles drawn from proven architecture and technology analysis and evaluation techniques such as the Quality Attribute Workshop and the Architecture Tradeoff Analysis Method. LEAP(4BD) leverages our extensive experience with architecture-based design and assessment and customizes these methods with deep knowledge and experience of the architectural and database technology issues most pertinent to big data systems. Working with an organization’s key business and technical stakeholders, this approach involves the following general guidelines: Assess the existing and future data landscape. This step identifies the application’s fundamental data holdings, their relationships, the most frequent queries and access patterns, and their required performance and quantifies expected data and transaction growth. The outcome sets the scope and context for the rest of the analysis and evaluation and provides initial insights into the suitability of a range of contemporary data models (e.g., key-value, graph, document-oriented, column-oriented, etc.) that can support the application’s requirements. Identify the architecturally-significant requirements and develop decision criteria. Focusing on scalability, performance, security, availability, and data consistency, stakeholders characterize the application’s quality attribute requirements that drive the system architecture and big data technology selection. Combining these architecture requirements with the characteristics of the data model (previous step) provides the necessary information for initial architecture design and technology selection. Evaluate candidate technologies against quality attribute decision criteria. Working with the system architects, this step identifies and evaluates candidate big data technologies against the applications’ data and quality attribute requirements, and selects a small number of candidates (typically two to four) for validation through prototyping and testing. The evaluation is streamlined by using an evaluation criteria framework for big data technologies that we are developing as part of our internal R&D. This framework focuses the assessment activities for specific database products against a generic collection of behavioral attributes and measures. Validate architecture decisions and technology selections. Through focused prototyping, our approach ensures that the system design and selected database technologies can meet the defined quality attribute needs. By evaluating the prototype’s behavior against a set of carefully designed, application-specific criteria (e.g., performance, scalability, etc.), this step provides concrete evidence that can support the downstream investment decisions required to build, operate, and evolve the system. During the construction and execution of the prototypes, the project team develops experience working with the specific big data technologies under consideration. Benefits LEAP(4BD) provides a rigorous methodology for organizations to design enduring big data management systems that can scale and evolve to meet long-term requirements. The key benefits are: A focus on the database and key analysis services addresses the major risk areas for an application. This keeps the method lean and produces rapid design insights that become the basis for downstream decisions. A highly transparent and systematic analysis and evaluation method significantly reduces the burden of justification for the necessary investments to build, deploy, and operate the application. Maximizing the potential for leveraging modern big data technologies reduces costs and ensures that an application can satisfy its quality attribute requirements. Greatly increased confidence in architecture design and database technology selection, and hands-on experience working with the technology during prototype development, reduces development risks. The identification of outstanding project risks that must be mitigated in design and implementation, along with detailed mitigation strategies and measures that allow for continual assessment. Looking Ahead We are currently piloting LEAP(4BD) with a federal agency. This project involves both scalable architecture design and focused NoSQL database technology benchmarking, as well as an assessment of features to meet the key quality attributes for scalable big data systems. We are interested in working with organizational leaders who want to ensure appropriate technology selection and software architecture design for their big data systems. If you are interested in collaborating with us on this research, please leave us feedback in the comments section below or send an email to us at igorton@sei.cmu.edu or jklein@sei.cmu.edu. Additional References To read the article "Time, clocks, and the ordering of events in a distributed system" by Leslie Lamport, please visit http://research.microsoft.com/en-us/um/people/lamport/pubs/time-clocks.pdf To read the paper "Characterizing cloud computing hardware reliability," by Kashi Venkatesh Vishwanath and Nachiappan Nagappan, which was presented at the Proceedings of the First Association for Computing Machinery Symposium on Cloud Computing, please visithttp://dl.acm.org/citation.cfm?doid=1807128.1807161 To read more about the challenges of ultra-large-scale systems please visit http://www.sei.cmu.edu/uls  

By Timur SnokeMember of the Technical StaffCERT Network Situational Awareness Team Occasionally this blog will highlight different posts from the SEI blogosphere. Today we are highlighting a post from the CERT/CC Blog by Timur Snoke, a member of the technical staff in the SEI’s CERT Division. This post describes maps that Timur has developed using Border Gateway Protocol (BGP) routing tables to show the evolution of public-facing autonomous system numbers (ASN).  These maps help analysts inspect the BPG routing tables to reveal disruptions to an organization’s infrastructure.  They also help analysts glean geopolitical information for an organization, country, or a city-state, which helps them identify how and when network traffic is subverted to travel nefarious alternative paths to place communications deliberately at risk.  

By Julien Delange, Senior Member of the Technical StaffSoftware Solutions Division The size and complexity of aerospace software systems has increased significantly in recent years. When looking at source lines of code (SLOC), the size of systems has doubled every four years since the mid 1990s, according to a recent SEI technical report. The 27 million SLOC that will be produced from 2010 to 2020 is expected to exceed $10 billion. These increases in size and cost have also been accompanied by significant increases in errors and rework after a system has been deployed. Mismatched assumptions between hardware, software, and their interactions often result in system problems that are detected only after the system has been deployed when rework is much more expensive to complete. To address this problem, the Society of Automotive Engineers (SAE) released the Architecture Analysis & Design Language (AADL), which helps software and system architects address the challenges of designing life- and safety-critical systems by providing a modeling notation with well defined real-time and architectural semantics that employ textual and graphic representations. This blog posting, part of an ongoing series on AADL, describes the use of AADL in the aerospace industry to improve safety and reliability. The Evolution of AADL The SEI has been a leader in the development of AADL and has participated on the AADL standardization committee since its inception. Likewise, several projects have used AADL to design software architecture and analyze, validate, or improve various quality attributes, such as reliability, latency/performance, or security. While AADL was initially conceived for use in avionics, this blog series has demonstrated its applicability in other domains that rely on life- and/or safety-critical systems, including the medical domain. This post focuses on the evolution of AADL from its initial foundations (covered in our previous blog posting) to its use in the System Architecture Virtual Integration (SAVI) project. SAVI is part of the collaborative and applied research performed by an aerospace industry research cooperative known as the Aerospace Vehicle Systems Institute (AVSI). SAVI is a multi-year project separated into different tasks that aim to improve system and software development and significantly reduce development costs through the early discovery of what will eventually become integration issues. Each sub-task addresses a particular issue companies are actually struggling with. The remainder of this post shows how the design rationale of AADL fits with SAVI’s goals and describes the uses of AADL within the project to improve software safety and reliability. A Focus on Safety A major integration challenge in fielding aerospace software systems stems from safety requirements. Detecting and analyzing faults/errors within a component can prove demanding for software architects. These tasks are especially hard during component integration, when it is necessary to understand how the faults would propagate among the architecture without testing or simulation. This challenge served as a catalyst for the development of demanding certification standards such as DO178C, also known as "Software Considerations in Airborne Systems and Equipment Certification." SEI researchers, led by Peter Feiler, a senior member of the technical staff and author of AADL, made the decision to document the benefits of the architecture model and bind safety-related specifications (such as faults occurrence and propagation across the architecture) to the architecture model. As Bruce Lewis—an experienced software practitioner working for the U.S. Army Aviation & Missile Research at the Development & Engineering Center (AMRDEC) in Huntsville Alabama and a key player in the development and evolution of AADL—reported: The error models are placed on components, but when we integrate the components, we know what kinds of errors they can accept on their input and what kinds of errors they can propagate out. Then we can do an integration of error models to actually look at how errors might propagate to the system. SEI researchers extended AADL to describe system faults and analyze the safety of systems. This work consisted of two parts: enhancing the core of the technology with an updated sub-language (the Error-Model Annex Version 2, an AADL annex being standardized by the SAE) to better describe architecture safety concerns developing new tools to process this additional information within the model, validate system safety, and help engineers produce documents for validating the system These enhanced capabilities enable AADL users to augment their software architectures with fault description and specify error source and propagation across hardware and software components. These enhancements can also be used to improve system validation and detect faults that are potentially overlooked during the initial development process and that may lead to errors. These safety analyses can now be evaluated incrementally as the architecture is refined during the development process, again permitting early discovery of issues and reducing manual effort. A Safety-Dedicated Sub-language for AADL To improve system validation and detect faults, an updated annex language has been proposed by the SEI and is currently under review by the AADL Standardization Committee. The effort involved contributors from several companies and domains, and was intended to make the document more user friendly for safety analysts. SAE will publish this addition to the core AADL language in the coming months as an official standard annex. In addition, the SEI will publish technical reports to present this sub-language and demonstrate its use on real systems, such as the Wheel Brake System example. This example (originating from the SAE AIR6110 standard that demonstrates the safety validation process for the avionics domain) demonstrates the relevance of our approach and that the approach scales with current industrial practices Researchers from the SEI also recently enhanced the AADL reference environment, OSATE, to analyze system safety and produce documents required by safety evaluation standards, such as SAE ARP4761. For now, the toolset provides several functions that allow software engineers to extract safety-related information from an architecture model and show the impact of a fault in documents, such as Functional Hazard Assessment provides a list of faults with their description, reference to design document and severity classification Fault Tree Analysis provides a hierarchical description of fault dependencies. Failure Mode and Effects Analysis provides a list of all faults with their propagation paths across the architecture. Reliability Block Diagram provides an analysis that evaluates the failure probability of a component according to its failure events and incoming error propagations. These functions have been demonstrated in the context of the SAVI project. Also, as these functions can be interfaced with open-source tools that are available at no cost, they can be tested using the public release of OSATE. In that context, the SEI researchers detailed how to reproduce the demonstration using a public model of the Wheel Brake System from the SAE AIR6110 standard. Beyond Safety: Analyzing System Behavior The next iteration of SAVI will focus on system behavior. Many projects from the avionics or aerospace communities reuse software and hardware components from different projects, trying to reduce new development and take advantage of existing components. When software and hardware components are integrated, however, tests show that behavior discrepancies can generate issues that were not predicted in their initial execution environment. For example, different development teams may use different assumptions regarding quality attributes (such as timing, resources dimensions, etc.) resulting in issues (late arrival of data, insufficient computing capacity, etc.) during integration. This discrepancy may lead to significant increases in testing efforts and development costs as well as a postponement in the delivery of the system. To overcome these issues, SEI researchers are working to extend AADL by integrating component behavior descriptions into a new "behavior annex." This new addition to the AADL language would provide software architects with the ability to describe the software behavior in terms of modes (operational, failed, degraded, etc.), states (running, idle, waiting for inputs, etc.) and operations (call to system functions or subprograms, send/receive values from interfaces, etc.). This additional architecture information would also allow architects to check the consistency of integrated components behaviors and detect potential defects or requirements mismatches. Part of this work may lead to the development of analysis tools that will integrate an existing behavior specification (such as state machine) into an architecture model and detect potential defects. Our next post will present the actual outcome of this work and the envisioned tool support. Additional Resources To view the AADL Wiki, please visit https://wiki.sei.cmu.edu/aadl/index.php/Main_Page For more information about AADL, please visit http://www.aadl.info To read the SEI technical report, System Architecture Virtual Integration: An Industrial Case Study, please visithttp://resources.sei.cmu.edu/library/asset-view.cfm?assetID=9145

By Jose Morales Senior Member of the Technical Staff CERT Division In early 2012, a backdoor Trojan malware named Flame was discovered in the wild. When fully deployed, Flame proved very hard for malware researchers to analyze. In December of that year, Wired magazine reported that before Flame had been unleashed, samples of the malware had been lurking, undiscovered, in repositories for at least two years. As Wired also reported, this was not an isolated event. Every day, major anti-virus companies and research organizations are inundated with new malware samples. Although estimates vary, according to an article published in the October 2013 issue of IEEE Spectrum, approximately 150,000 new malware strains are released each day. Not enough manpower exists to manually address the sheer volume of new malware samples that arrive daily in analysts’ queues. Malware analysts instead need an approach that allows them to sort out samples in a fundamental way so they can assign priority to the most malicious of binary files. This blog post describes research I am conducting with fellow researchers at the Carnegie Mellon University (CMU) Software Engineering Institute (SEI) and CMU’s Robotics Institute.  This research is aimed at developing an approach to prioritizing malware samples in an analyst’s queue (allowing them to home in on the most destructive malware first) based on the file’s execution behavior. Existing Approaches to Prioritizing Malware Analysis Before beginning work on developing an approach for prioritizing malware analysis, our team of researchers examined existing approaches and found very few. Most institutions in academia, government, and industry analyze malware by randomly selecting samples, ordering them alphabetically, or analyzing a binary file in response to a request for a specific file based on its MD5 , SHA-1, or SHA-2 cryptographic hash value. Our team decided to take a systematic approach to malware analysis that takes incoming samples and analyze them using runtime analysis. At a high-level, this approach collects and categorizes salient features. Using a clustering algorithm, our approach then ideally prioritizes the malware sample appropriately in an analyst’s queue based on their description of the type of malware they want to analyze upon its arrival to their repository. A key idea in our approach involved the use of dynamic analysis to measure the maliciousness of a malware sample based on its execution behavior. The assumption is that all malware samples have certain malicious events they must perform to carry out nefarious deeds. The implementation of these deeds is captured at runtime and may prove to be useful assessment characteristics. Salient and Inferred Features When we initially began this research, we extracted more than two dozen features that were a mix of actual runtime data that could be easily identified as suspicious. An example is modification of a registry key such as Windows\CurrentVersion\Run. inferred features that are derived by analyzing the details of one or more actual runtime suspicious data.  An example is a malware setting itself up to start at system reboot by analyzing the details of the Windows\CurrentVersion\Run registry key modification and concluding the edit was the malware placing its own file system path in this key assuring it will execute on reboot. It’s important to note that malware doesn’t typically commit all of its nefarious deeds with just one running process. In examining features, we also created malware infection trees to gain a better understanding of the processes and files created by the malware. The paper, "Building Malware Infection Trees," which I co-authored, allowed us to view the malware sample as "a directed tree structure" (see Figure 1 below) with each node representing a file or process that the malware had created and each edge representing file creation, process creation, self-replication or dynamic code injection behavior.  Figure 1. A Malware Infection Tree for Poison Malware.Our research focused on the following three areas of suspicious behavior, all of these behaviors were recorded for the executing malware process and any related processes in its malware infection tree.  We assume these behaviors being performed by a member of the malware infection tree can lead to suspicious behaviors usable in assessing the malware sample: File systems. We looked at the typical open, write, create, move, and delete file operations. These are typical behaviors for any executable, but when performed by malware they can occur in greater numbers or target specific files as part of a broader malicious behavior.  We also examined instances of self-replication where malware copies itself into a new file or copies itself into an existing file as well as a piece of malware deleting itself from the system. We also considered cases where a child process in the malware infection tree deleted the static file image of an ancestor process which was deemed a possible attempt to avoid detection. Networks. We examined domain name service (DNS) queries and reverse DNS (rDNS) queries.  Specifically, we searched for high occurrences of DNS and rDNS requests, especially failed requests which indicate that malware may be testing a set of potentially active IP addresses to connect with a command-and-control server or some other malicious server.We also considered multiple failed Transmission Control Protocol (TCP) connection attempts, specifically cases where a URL was DNS queried multiples times and then, when an attempt was made to connect to the IP address, the attempt failed. This scenario indicates that the malware might be harvesting IP addresses through DNS requests, but the IP addresses it harvests are servers that have not been activated or have been taken down, blocked, removed, or blacklisted. A piece of malware sometimes keeps repeating this action until it connects with an IP address or times out. We also identified use of non-typical network protocols.  Processes. We primarily looked at whether the malware created a completely new process, deleted an already running process, or started a new process thread. When a malware sample deleted a process, we checked to see if the process that was being deleted belonged to a known piece of anti-malware software. If that was the case, we inferred that the malware was actually trying to disable the system’s security measures to ensure its safety and longevity.Our analysis of processes also included dynamic code injection where the malware process running on a system identifies already existing processes that are likely to be running on any version of the target system. Specifically, we looked at several Windows platform standard processes that start at runtime, such as svchost.exe, winlogon.exe, and taskhost.exe.In dynamic code injection, the malware process chooses an already running process and writes malicious commands into the process’s memory. Next, the malware creates a new thread that will execute the malicious instructions that have just been written to its memory. These steps result in the modified process behaving in non-typical ways. With dynamic code injection, the malware actually delegates malicious acts to other process so that that the system will detect the other process as doing something suspicious and not the original malware that is running on the system.We also examined whether, during our runtime analysis, the malware tried to modify the registry key Windows\CurrentVersion\Run. A modification of this registry key is obviously something that we want to know about because it can be an indicator of malicious behavior.   A Technical Dive into Our Approach Once we determined the features we wanted to look at, we turned our attention to creating a training set. To do so, our team compiled a set of malware samples classified as advanced persistent threats (APTs), botnets, and Trojans.  We also included known malware that was ranked in the Top 5 most dangerous and most persistent in the wild in the past five years, according to Kaspersky’s securelist.org website. We submitted, executed, and analyzed each sample in our runtime analysis framework and extracted our chosen features. Once we created our training set, we assembled 11,000 malware samples that we clustered based on the training set. The training set allowed us to gain an initial understanding about what our algorithms tell us about how we could prioritize the malware.  We examined execution behavior at the user and kernel levels for a three-minute, run-time analysis. We then determined, for the whole set, if behaviors were identified mostly from user- or kernel-level collected data. Next, we created a set of characteristics from the most pervasive observed execution behaviors and repeated our experiment using a larger mixed malware sample set and clustered results based on our set of characteristics.  At the end of analysis we collected the feature sets for our training sets and our test set of 11,000 samples, and we submitted them for analysis with various machine learning algorithms to determine which one is best for prioritizing malware samples based on our features. Collaborations Dr. Jeff Schneider, a researcher at the Robotics Institute in CMU’s School of Computer Science and an expert on classification and clustering, agreed to analyze our feature sets and create a classifier to prioritize malware samples. We are also working with software engineers within the CERT Division’s Digital Intelligence and Investigations Directorate Software Engineering group who wrote code for us including the feature extraction code. Next Steps The goal of our research was to allow analysts greater efficiency in establishing a priority queue for malware analysis. These priorities can vary based on whether the malware analyst works in the financial industry and is interested in Distributed Denial of Service (DDoS) attacks, botnets, or Trojans, or whether the analyst works in the DoD’s cyber command and is interested in APTs and protecting national assets. If our approach works as planned, we will accelerate efforts to formalize an automated prioritization system using our established set of salient and inferred features that could be integrated into current analysis frameworks using as input a live continuous malware feed. We welcome your feedback on our approach in the comments section below. Additional Resources To read the paper, "Building Malware Infection Trees," by Jose Andre Morales, Michael Main, Weiliang Luo. Shouhuai Xu, and Ravi Sandhu, please visit http://ieeexplore.ieee.org/xpls/abs_all.jsp?arnumber=6112326 To read about other malware research initiatives at the SEI, please visithttp://blog.sei.cmu.edu/archives.cfm/category/malware