By Kurt WallnauSenior Member of the Technical StaffResearch, Technology, and System Solutions and CERT Science of Cyber-Security For more than 10 years, scientists, researchers, and engineers used the TeraGrid supercomputer network funded by the National Science Foundation (NSF) to conduct advanced computational science. The SEI has joined a partnership of 17 organizations and helped develop the successor to the TeraGrid called the Extreme Science and Engineering Discovery Environment (XSEDE). This posting, which is the first in a multi-part series, describes our work on XSEDE that allows researchers open access—directly from their desktops—to the suite of advanced computational tools and digital resources and services provided via XSEDE. This series is not so much concerned with supercomputers and supercomputing middleware, but rather with the nature of software engineering practice at the scale of the socio-technical ecosystem. Background: Bringing Disciplined Engineering to TeraGrid From 2001 to 2011 the NSF’s network of supercomputers, services, and middleware—known as the TeraGrid—played a key role in establishing and supporting the emerging discipline of computational science.  Computational scientists used the TeraGrid to study the atomic-level structure of protein needed for sound perception, investigate how gas giant formations like Jupiter and Saturn are formed, and gain new insights into the early stages of HIV-1 infection to prevent the virus’ ability to "hijack" other healthy cells. By 2008, however, it was becoming clear to NSF’s Office of Cyberinfrastructure (OCI) that TeraGrid’s success had generated new demands from computational scientists that could not be satisfied by the existing infrastructure.  While TeraGrid had provided technologies and services to integrate the nation’s most advanced supercomputing resources, by 2008 there was a growing perception among computational scientists that not only did the national cyberinfrastructure ecosystem need to provide access to supercomputing resources, there needed to be a easier way to integrate a much broader array of digital assets and services in conjunction with much greater stability and other quality attributes in order to support top-tier research programs. In 2008, NSF/OCI issued a call for proposals for TeraGrid Phase III: eXtreme Digital (XD) Resources for Science and Engineering. As expected, the Phase III solicitation called for a continuation of and substantial improvements to core TeraGrid technologies, as well as to its outreach, education, consulting, and infrastructure-management services.  The solicitation also included something surprising—a strong emphasis on software and system engineering in Phase III.  NSF reinforced this new emphasis in a series of town hall meetings for potential submitters. Among the takeaways from those meetings, one thing was certain: The winning proposal would need disciplined engineering processes. But which processes?  It was this question that Pittsburgh Supercomputing Center director Michael Levine posed to the SEI. The SEI’s foundational role in identifying "process" as a lever for improving practice—and in achieving industrial-scale adoption of software engineering process through the Capability Maturity Model Integration (CMMI) and its many derivatives—made it a natural choice for Levine and his team.  Levine, a key leader in TeraGrid, also understood that NSF’s culture and the TeraGrid community in particular required something different than the conventional notion of "process." It was this understanding that ultimately led Levine to the SEI’s Research, Technology, and Systems Solution (RTSS) program. After a few rounds of discussion, RTSS joined a team that included the National Center for Supercomputing Applications (NCSA), Pittsburgh Supercomputing Center (PSC), Texas Advanced Computing Center (TACC), and the National Institute for Computational Sciences (NICS).  The team assembled by Towns at the NCSA ultimately developed the winning proposal. The successor to TeraGrid would now be known as the eXtreme Science and Engineering Discovery Environment (XSEDE). Our Challenge: Creating an Engineering Culture NSF decided to award one-year planning grants to the two most competitive preliminary proposals: XSEDE and XROADS. NSF had awarded the planning grant with the stipulation that the results were to be shovel-ready implementation plans, ready on Day 1. That is, the winning proposal would need to go operational with no disruption of service to any current customer of TeraGrid.  The XSEDE team recognized the following challenges of developing a credible management and technical plan: For the management plan, this meant completely working out all the details about project structure and governance. For the technical plan, this meant defining the software and system architecture needed to support immediate implementation. The SEI team believed that the challenge was of a crosscutting nature: Project governance, software and system architecture, and engineering practice must be mutually reinforcing.  Our first challenge was to persuade the XSEDE team to abandon the belief (as it turns out, a belief that was widely held by both XSEDE and XROADS teams) that "doing" software engineering was nothing more than following a process, and, by extension, the SEI could define "The Process." To overturn this mistaken belief would require persistence.  We undertook two efforts to do so: We developed a traditional System Engineering Management Plan (SEMP) that described top-level engineering processes and organizations. We piloted architecture-based engineering practices to help the architects engage internal stakeholders to decide what mattered most in the architecture. As expected, the XSEDE team was mostly indifferent to the content of the SEMP, except to note that it lacked narrative "pizazz." They were surprised, however, by the effectiveness of techniques—such as the Mission Thread Workshops led by the SEI’s Mike Gagliardi—at shedding light on the organizational implications of engineering decisions.  Most importantly, the team began to appreciate that the steps of "The Process" do not define most of what we regard as software engineering. Instead, real software engineering practices reside within these steps—the parts that often aren’t written down.  Our New Challenge: Doing it Again, In Operation With the one-year planning grant complete, in 2011 XSEDE and XROADS teams submitted and briefed their plans to the NSF Review Panel.  When asked what worried him most about the review process, John Towns (XSEDE PI) replied he was concerned that the panel would decide to fund part of each of the XSEDE and XROADS efforts. That funding model would violate the by-now integrated management, technical, engineering and governance structures that XSEDE had developed in its plan. As it turned out, Towns’ fears were partly realized when NSF announced its intent to award XD to XSEDE with the proviso that it incorporate the most meritorious elements  of the XROADS effort and team.  We were now confronted with the need to revisit planning assumptions and decisions made with a full year of deliberations—in effect to "re-plan" the effort in the span of a few short weeks. In hindsight it is clear that NSF made a wise decision in merging the XSEDE and XROADS proposals, both in terms of avoiding the schism in the community that might have arisen from a "winner-take-all" outcome, and in terms of incorporating several exciting architectural ideas introduced by XROADS, such as delivering cloud resources as an alternative to leadership class "big iron" computers, and adopting "software as a service" as a business model and product delivery channel. On the other hand, much of what we had accomplished in terms of introducing the seeds of an engineering culture in XSEDE had been thoroughly disrupted. After a year of engaging the XSEDE team on the nuances of software engineering methods, these methods had begun to pay dividends. Of course this progress relied on the skills and perspectives of just one of the two teams with which we had engaged.  Now our engineering teams would be reconstituted with people from both teams.  Not only had we potentially lost a "common sense" of software engineering, but we now had engineering teams with "competing views" of what to accomplish.   We had to reconstitute an engineering culture while simultaneously realigning our technical objectives. Moreover, because the TeraGrid effort was coming to its contractual end, we needed to do this "live," which is like having to redesign and rebuild an aircraft while in flight.  To be fair, although we knew that the merger was going to be challenging, we also knew that we were substantially further along than we had been one year earlier. Both teams had spent their planning efforts wisely, and their overall approaches were more similar than different, and where different, they were mostly complementary.  There was also something at work that is difficult to objectively quantify: The members of the XSEDE and XROADS teams were mostly well known to one another, and there was a universal desire to move beyond the competition of the past years, roll up our sleeves, and get to work on the real mission of advancing the nation’s computational science. Results after One Year What did we manage to accomplish?  Did we build a redesigned aircraft while in mid-flight without crashing and burning?  After one year, we can safely claim a good measure of success, the results of which are summarized in the figure below. Figure 1 shows a portion of the XSEDE program that is engaged in engineering work on a day-to-day basis, as described below: The SEI helped introduce an explicit and rigorous treatment of software architecture as a keystone of the XSEDE engineering approach. We reinforced concrete practices for documenting software architecture and for understanding, documenting, and using quality attributes to design and evaluate how well an architecture serves its strategic aims. The SEI helped introduce a range of concrete software development practices, beginning with unit, integration, and acceptance testing, which had been completely lacking in TeraGrid.  We augmented these practices with a rudimentary software lifecycle that included basic development phases and verification gates for detailed design and various test phases. The SEI also helped XSEDE leadership evolve to a more realistic and nuanced understanding of "process."  XSEDE had evolved from its early quest for The Process to a program that appreciated the importance of establishing an engineering culture, as well as the need for that culture to define and evolve its own engineering processes. It is fair to say that on Day 1 of the XSEDE/XROADS union, none of the practice areas reflected in the graphic and outlined above were in place, and indeed, few if any had any legacy in TeraGrid. This substantial progress is due not to the SEI but to XSEDE leadership and staff.  Our plans for Year 2 combine two main thrusts Consolidation by documenting and applying practices and making small improvements as needed. Planning for new and potentially groundbreaking practices to scale architectural design to allow contributions from hundreds to thousands of members of the XSEDE community and from the larger computational science ecosystem of science grids, communities, and technology providers. Upcoming Topics in the XSEDE Thread My next post will explore establishing an engineering culture within an emerging social-technical system. Other topics that I plan to post about are: What are the quality attributes of a socio-technical system that will produce not just one intended system but many systems? Can the web of cultural norms and incentives that make up a socio-technical system be described? Can their mutual alignments be measured or influenced?  Can these measures predict the quality of engineering decisions? What is the role of an integrating technology platform in the formation and maintenance of a socio-technical ecosystem?  How can a socio-technical ecosystem be established to design the platform for another ecosystem? (In fact, this is the XSEDE case.) Are persistent institutions required to sustain engineering practice in a socio-technical ecosystem?  Can a practice be sustained without such institutions? In addition to the above posts, I’d like to enlist the help of other contributors to the XSEDE effort, as well as researchers of engineering practices in and for socio-technical systems.  Please feel free to leave your thoughts in the comments section below. Due Credit Before Closing While most of the credit belongs to XSEDE leadership and staff, I will take a few moments here to give credit to SEI colleagues who have contributed to this result: Mike Gagliardi introduced XSEDE to large-scale architecture thinking though Mission Thread Workshops. Joe Batman demonstrated mature systems engineering practices by showing the connectedness of engineering decision-making and project governance. Paul Clements, and later Felix Bachmann, introduced disciplined approaches for documenting architecture in a way that serves stakeholder interests. Scott Hissam defined the Software Development and Integration (SDI) approach (see Figure 1) to integration testing, acceptance testing, and version management, and invaluable assistance in environment integration. Rhonda Brown and Mike Konrad drew on their experience in CMMI, TSP and more recently, the combination of TSP with software architecture practices, to improve the agility of SDI’s overall approach. Additional Resources For more information about the XSEDE project, please visit https://www.xsede.org/.For more information about the SEI’s work in architecture centric engineering (ACE), please visit www.sei.cmu.edu/about/organization/rtss/ace.cfmFor more information about the SEI’s work in system-of-systems engineering, please visit www.sei.cmu.edu/sos/.For more information about the SEI’s work in ultra-large-scale systems, please visit www.sei.cmu.edu/uls/.

By Bill Pollak, Transition Manager Research, Technology, & System Solutions A search on the term "software architecture" on the web as it existed in 1992 yielded 88,700 results. In May, during a panel providing a 20-year retrospective on software architecture hosted at the SEI Architecture Technology User Network (SATURN) conference, moderator Rick Kazman noted that on the day of the panel discussion—May 9, 2012— that same search yielded 2,380,000 results. This 30-fold increase stems from various factors, including the steady growth in system complexity, the increased awareness of the importance of software architecture on system quality attributes, and the quality and impact of efforts by the SEI and other groups conducting research and transition activities on software architecture. This blog posting—the first in a series—provides a lightly edited transcription of the presentation of the first panelist, Linda Northrop, director of the SEI’s Research, Technology, & System Solutions (RTSS) Program at the SEI, who provided an overview of the evolution of software architecture work at the SEI during the past twenty years. 20 Years of Architecture at the SEI as Presented by Linda Northrop In 1992, the SEI was working on structural modeling and applying this concept to flight simulators for the purpose of re-architecting Air Force systems that had become brittle and unmodifiable. Others at the SEI were then studying what we called "quality attributes," looking at such attributes as performance, reliability, and modifiability and trying to determine what the underlying analytic models were. Soon after, we started a software architecture project at the SEI that studied architecture description languages, definitions of software architecture, and ways to evaluate software architectures. Around 1995, we were asked by the U.S. Department of Defense (DoD) to write something about software architecture. In response, Paul Clements and I wrote Software Architecture: An Executive Overview. Then Len Bass, Paul Clements, and Rick Kazman wrote the first edition of the book Software Architecture in Practice. We considered a number of existing definitions of software architecture, but in the end formulated our own:  The software architecture of a program or computing system is the structure or structures of the system, which comprise software elements, the externally visible properties of those elements, and the relationships among them. Architecture is concerned with the public side of interfaces; private details of elements—details having to do solely with internal implementation—are not architectural. This work constituted the beginning of a cohesive body of SEI research and transition efforts focused on software architecture.   We started our research with architecture evaluation, because we reasoned that by evaluating software architectures early, we could alleviate subsequent testing and integration problems. We developed the Architecture Tradeoff Analysis Method (ATAM ) and from this grew many other methods based on questions that people asked during our application of the ATAM , such as, "What are the significant quality attributes?" (Quality Attribute Workshop) and "How should an architecture be documented?" (Views & Beyond method). Over time, we created a repertoire of methods, technologies, and techniques that grew into a discipline that we now call architecture-centric engineering (ACE). Characteristics of ACE discipline include an explicit focus on quality attributes direct linkage of architecture to business and mission goals explicit involvement of system stakeholders a grounding in state-of-the-art quality attribute models and reasoning frameworks and a firm foundation in how these frameworks can guide architectural decisions Other organizations, such as Bosch, Raytheon, Siemens, and Lockheed Martin, were also exploring and embracing practices in software architecture during these years. In 2006, I led a study on ultra-large-scale (ULS) systems, which we defined as a system at least one of whose dimensions is of such a large scale that constructing the system using development processes and techniques prevailing at the start of the 21st century is problematic. This study included panelist Doug Schmidt and a number of other experts from an assortment of disciplines. We studied the characteristics of software-reliant systems of the future and learned that these systems would be inherently decentralized that you’d never be able to fully know the diverse range of requirements, many of which will conflict that they’d be continuously evolved, developed, and deployed -- in other words they can’t be completed shutdown and restarted they’d be heterogeneous and developed and governed by many organizations and stakeholders failure will be a normal part of system operation people would be a part of the system, as opposed to being "mere" users. ULS systems were a complete mind change for me. I was thinking that, if we are going to architect these systems effectively, what aspects of our current body of knowledge on software architecture could be applied and what new breakthroughs would be required? Much of what we described in the study report has come to pass; there are ULS systems today with the characteristics we identified. In addition, several game-changing technologies, such as cloud computing, social media, mobile computing, and multicore have emerged that can be used to improve ULS system qualities, as well as to gain competitive advantage. Many people here at the conference have talked about what’s next. Jeromy Carriere said that you can’t nail everything down about the architecture when you have an inherently distributed team and everyone has to operate autonomously, but you can agree on the hard decisions and consistent boundaries. So in the future we have to understand What are appropriate architectures for socio-technical, cyber-physical systems (especially ones that form critical parts of systems-of-systems and ULS systems)? What are quality attributes of these systems that apply today? We need to consider attributes like privacy and we need to incentivize humans in these system to do the right things. How to get stakeholders involved when they are so numerous and widely distributed we can’t get them in a room to hold a Quality Attribute Workshop or conduct an ATAM? Perhaps we can exploit social media - "crowdsource" the applications of our methods.    Our charge as software architects is to take all the ULS system characteristics and challenges and figure out how we architect successfully in that space. What’s Next In my next blog posting, I’ll provide a transcript of the presentation by Douglas C. Schmidt, former CTO of the SEI and currently a professor of computer science at Vanderbilt University, who will talk about advances in software architecture practice for distributed real-time and embedded systems.  Subsequent blog postings will present the talks by Ian Gorton research and development R&D lead Data Intensive Computing at Pacific Northwest National Lab; Bob Schwanke of Siemens Corporate Research; and Jeromy Carrière, chief architect at X.commerce, a subsidiary of eBay. Additional Resources The abstract and slides for Linda Northrop’s talk are available at www.sei.cmu.edu/library/abstracts/presentations/northrop-panel-saturn2012.cfm. The Ultra-Large-Scale Systems report is available at www.sei.cmu.edu/uls/. For more information about the book Software Architecture in Practice, please visitwww.sei.cmu.edu/library/abstracts/books/0321154959.cfm. SATURN 2013, presented in collaboration with IEEE Software magazine, will be held April 29 through May 3, 2013 in Minneapolis, Minnesota. For more information see the SATURN 2013 software architecture conference website at www.sei.cmu.edu/saturn/2013.

By Bill PollakTransition ManagerResearch Technology & System SolutionsLast week, we presented the first posting in a series from a panel at SATURN 2012 titled "Reflections on 20 Years of Software Architecture." In her remarks on the panel summarizing the evolution of software architecture work at the SEI, Linda Northrop, director of the SEI's Research, Technology, and System Solutions (RTSS) Program, referred to the steady growth in system scale and complexity over the past two decades and the increased awareness of architecture as a primary means for achieving desired quality attributes, such as performance, reliability, evolvability, and security. It’s undeniable that the field of software architecture has grown during the past 20 years. In 2010, CNN/Money Magazine identified "software architect" as the most desirable job in the U.S. Since 2004, the SEI has trained people from more than 900 organizations in the principles and practices of software architecture, and more than 1,800 people have earned the SEI Software Architecture Professional certificate. It is widely recognized today that architecture serves as the blueprint for both the system and the project developing it, defining the work assignments that must be performed by design and implementation teams. Architecture is the primary purveyor of system quality attributes which are hard to achieve without a unifying architecture; it’s also the conceptual glue that holds every phase of projects together for their many stakeholders. This blog posting—the second in a series—provides a lightly edited transcription of a presentation by Douglas C. Schmidt, former chief technology officer of the SEI and currently a professor of computer science at Vanderbilt University, who discussed advances in software architecture practice for distributed real-time embedded systems during the past two decades. Reflections on 20 Years of Architecture for Distributed Real-time & Embedded Systems as Presented by Douglas C. Schmidt My talk focuses on advances in software architecture practice for distributed real-time embedded (DRE) systems, with a retrospective of the past 20 years in that space. The essence of a DRE system is that the right answer delivered too late becomes the wrong answer. There are many military examples of DRE systems, such as integrated air and missile defense, as well as civilian examples, such as SCADA (supervisory control and data acquisition), air-traffic management, and electronic trading systems. Twenty years ago, many engineers developing DRE systems didn’t understand what architecture meant from a software perspective. I’ll never forget the first time I went to a meeting at McDonnell-Douglas where I was employed to help develop a common software architecture for their legacy avionics mission computing systems, and I asked, "Show me your architecture?" They put up a circuit diagram, and I thought, "Boy, this is going to be a challenging project!" In legacy DRE systems, operational performance was paramount, whereas software issues were secondary, so there was much uncertainty and confusion about key software design and implementation concerns. It was hard to sort those things out 20 years ago since we lacked a common vocabulary and organizing paradigm for reasoning about software architecture for DRE systems. Perhaps due to the lack of awareness of software concerns, engineers developing DRE systems in the early- to mid-1990s tended to substitute adherence to particular standards or products for a thorough understanding of software architecture. For example, there was a push to apply standards, such as early versions of the CORBA (Common Object Request Broker Architecture) standard.  Unfortunately, the CORBA specification initially de-emphasized implementation details to the point where standard CORBA object request brokers couldn’t be used effectively for DRE systems because they were woefully underspecified with respect to key quality attributes that Linda mentioned in her panel presentation. Another thing that was occurring in the early 1990s was a focus on domain-specific approaches to architecture. For example, there was a project at DARPA called Domain-Specific Software Architecture (DSSA) that examined how to structure software by focusing on domain-specific entities such as radars, trackers, launchers, etc. DSSA was a useful paradigm; in fact, it foreshadowed today’s interest in domain-specific modeling languages, model-driven engineering, and domain-driven design. There was a tendency 20 years ago, however, to overlook the underlying reusable domain-independent abstractions that weren’t adequately captured by the DSSA paradigm.  These domain-independent abstractions included reactive event demultiplexers and active object concurrency models that exist at a lower level than a specific application domain. During the latter part of the 1990s, key advances in software architecture for DRE systems stemmed from the growing focus on design and architecture patterns. The classic Gang of Four patterns catalog was published in 1994, followed by the first Pattern-Oriented Software Architecture (POSA) volume in 1996. As a result, engineers developing DRE systems began adopting common vocabularies to reason intentionally about software architecture from a practitioner perspective. For example, experience gained from work at Boeing, Lockheed Martin, Raytheon, Siemens, and many telecom equipment providers showed the benefits of applying patterns to describe product lines and common software architectures for DRE systems. The growing awareness of patterns more clearly differentiated design aspects from implementation aspects. Combining this distinction with a focus on optimizing and hardening DRE systems by choosing the right patterns to implement also helped improve the quality attribute support of DRE system standards, such as Real-time CORBA and the Data Distribution Service. This experience demonstrated that pattern-oriented architectural abstractions could scale up to larger, networked topologies, yet still assure traditional real-time concerns, such as avoiding priority inversion and ensuring schedulability. By the end of the 1990s, the DRE systems community recognized that patterns enabled software developers and systems engineers to understand how fundamental abstractions could be reused in a domain-independent manner, as well as combined to support integrated higher-level domain-specific abstractions.  The availability of popular, pattern-oriented, domain-independent middleware toolkits (such as ACE and TAO) in open-source form increased awareness of the value of applying patterns to production DRE systems. Starting around the year 2000, the patterns community began documenting software architectures for DRE systems more cohesively via pattern languages, which provide a vocabulary and process for the orderly resolution of software development problems.  For example, the POSA4 book describes brokers, publishers/subscribers, and messaging as key architectural elements in a pattern language for distributed computing. These types of pattern languages codify, coordinate, and generate larger ensembles of software and system elements than individual patterns and pattern catalogs. In recent years, engineers have applied various domain-specific modeling languages and tools (such as the Generic Modeling Environment and associated model-driven engineering technologies) to automate various elements of pattern languages.  The combination of pattern languages and model-driven tools has enabled the analysis and synthesis of frameworks and components that are more effective than earlier computer-aided software engineering (CASE) tools. So, over the past 20 years of research and transition efforts, software engineers have increasingly identified and applied patterns and pattern languages to document architectures and build DRE systems that are directly relevant to application developers and practitioners. The knowledge and technology base has matured to the point where the techniques and tools we have now provide substantial advantage over what was available 20 years ago, though they are no substitute for experience and insight. Looking ahead, I think we’re just beginning to comprehend and codify the pattern languages and architectures for the current generation of network-centric DRE systems-of-systems, as well as the next generation of ultra-large-scale (ULS) systems that Linda Northrop discussed in her panel presentation. As we reach this scale, complexity, and mission-criticality, we need the software architecture community to step forward and begin understanding and documenting the appropriate solutions in pattern form. Right now we know many anti-patterns for how not to build these types of systems, but we still struggle with how to build them at scale and reason about how they’ll work with all the quality attributes they need to succeed. What’s Next In my next blog posting, I’ll provide a transcript of presentations by Ian Gorton research and development R&D lead Data Intensive Computing at Pacific Northwest National Lab; Bob Schwanke of Siemens Corporate Research; and Jeromy Carrière, chief architect at X.commerce, a subsidiary of eBay. Additional Resources The abstract and slides for Douglas C. Schmidt’s talk are available at www.sei.cmu.edu/library/abstracts/presentations/schmidt-panel-saturn2012.cfm. Many patterns and pattern languages—as well as open-source implementations of these patterns—for DRE systems are available at www.dre.vanderbilt.edu/~schmidt/patterns.html. Douglas C. Schmidt is teaching a massively open online course (MOOC) on patterns and frameworks for concurrent and networked software for Coursera starting February, 2013. Course information and instructions for free registration are available at https://www.coursera.org/course/posa/. A Software Engineering Radio podcast describing a pattern language for distributed systems is available at www.se-radio.net/2007/07/episode-63-a-pattern-language-for-distributed-systems-with-henney-and-buschmann/. A podcast describing the applications of patterns and pattern languages to DRE systems is available at www.se-radio.net/2006/01/episode-3-interview-doug-schmidt/. SATURN 2013, presented in collaboration with IEEE Software magazine, will be held April 29 through May 3, 2013 in Minneapolis, Minnesota. For more information see the SATURN 2013 software architecture conference website at www.sei.cmu.edu/saturn/2013.

By David Keaton, Senior Member of the Technical Staff CERT Secure Coding Team By analyzing vulnerability reports for the C, C++, Perl, and Java programming languages, the CERT Secure Coding Team observed that a relatively small number of programming errors leads to most vulnerabilities. Our research focuses on identifying insecure coding practices and developing secure alternatives that software programmers can use to reduce or eliminate vulnerabilities before software is deployed. In a previous post, I described our work to identify vulnerabilities that informed the revision of the International Organization for Standardization (ISO) and International Electrotechnical Commission (IEC) standard for the C programming language. The CERT Secure Coding Team has also been working on the CERT C Secure Coding Standard, which contains a set of rules and guidelines to help developers code securely. This posting describes our latest set of rules and recommendations, which aims to help developers avoid undefined and/or unexpected behavior in deployed code.  History of Addressing Security Issues in C The C programming language began to take shape in 1969, long before security concerns became important for its applications. C was first standardized in 1989, too soon to take into account the then-budding security problems on the ARPANET.  Due to lack of customer demand for security, even the 1999 revision of the C standard contained only one security-related feature, the snprintf() function mentioned in my previous blog post.  In recent years, however, C developers have been forced to turn their attention to security issues. The CERT C Secure Coding Standard addresses this need by providing rules and recommendations for avoiding security problems in the following categories: preprocessor - issues dealing with macros declarations and initialization - choosing the right storage duration and type qualifiers, and C language rules for the uniqueness of variable names expressions - order of evaluation, safe use of C syntax integers - arithmetic issues such as avoiding integer overflow floating point - quirks of computer arithmetic that are often overlooked by people who are used to using integers arrays - allocating and communicating the correct size, and using the correct types characters and strings - ensuring that character sequences are null-terminated, and proper use of narrow and wide characters memory management - avoiding memory leaks, double free, and underallocation input output (I/O) - proper use of C’s file I/O library environment - interfacing with the operating system signals - best practices for handling asynchronous events error handling - ensuring correct detection of error conditions application programming interfaces - security-conscious design of the interfaces between parts of a program concurrency - issues that arise in multithreaded programs. miscellaneous - issues not covered by other categories, such as assertions, and maintaining the security of function pointers. POSIX - issues specific to the POSIX operating system, which is widely used with C. Examples of CERT C Secure Coding Rules The remainder of this blog posting gives some examples of the types of secure coding rules we’ve defined for C. Preprocessor macros. One part of the CERT C Secure Coding Standard focuses on the C preprocessor, which is a macro expander that executes at the beginning of the compilation process. Far too often, programmers overlook security-related consequences of preprocessor misuse. If a programmer passes an expression that has a side effect as an argument to a macro, the macro may cause the side effect to occur multiple times, depending on how it uses that argument. The CERT Secure Coding team recently developed rules to ensure that a programmer doesn’t accidentally pass an argument that slips a side effect into a macro, or that the programmer writes the code in such a way that the side effects only occur one time. While these and other rules described in this post aren’t ones that would normally be considered security risks, they have resulted in security problems when the code is deployed and activated. For example, we have developed the following rules and recommendations for developers to follow when they write code that involves the C preprocessor: Avoid side effects in arguments to unsafe macros (Identifier PRE31-C). This hard-and-fast rule states that if a developer is using a macro that uses its arguments more than once, then the developer must avoid passing any arguments with side effects to that macro. An example of the application of PRE31-C is#define ABS(x) (((x) < 0) ? -(x) : (x))/* ... */m = ABS(++n); /* undefined behavior */ Do not define unsafe macros (Identifier PRE12-C). This recommendation relates to PRE31-C that defines an unsafe macro as one that evaluates any of its arguments more than one time. Macro replacement lists should be parenthesized (Identifier PRE02-C). This recommendation suggests that developers should use parentheses around macro replacement lists; otherwise, operator precedence may cause the expression to be computed in unexpected ways. For example, if an argument contains a plus sign (+), and a macro contains a multiplication sign (*), and the argument has not been parenthesized, then the multiplication will occur first, followed by the addition, which may not be what a developer expected. An example of the application of PRE02-C is listed below:#define CUBE(X) (X) * (X) * (X)int i = 3;int a = 81 / CUBE(i); /* evaluates to 243 */ Declarations. C provides a range of mechanisms to declare data types and variables of these data types. For example, a developer might declare a variable in an outer scope and also declare another variable of the same name in a nested inner scope. In such a case, the variable in the inner scope hides the variable in the outer scope. When a developer makes changes to that variable, he or she might assume that changes are being made to the outer-scope variable when, in fact, only the inner-scope variable is being changed. The declarations problem is compounded by the fact that in C, there is a limit to how many characters are required to be unique in a variable name. The C standard has several requirements including the following: A macro name has 63 significant initial characters. If a program has two macro names and they differ only in the 64th character, the compiler is allowed to think that those are the same name. Programs also have 31 significant initial characters in an external identifier. If a program has two variables whose names differ only in the 32nd character or after, the compiler is allowed to think that those are the same variable. The situation described above can cause a problem wherein a developer has declared two variables that can inadvertently reside in the same scope. While the two variables might not have the same name in English, the compiler might truncate the name to such a degree that the names are the same. Characters and strings. C defines a set of functions that operate on strings composed of characters. It is common practice for developers to count the number of characters needed in a string and allocate exactly the same number of bytes. For example, if a developer allocates enough space to store the text version of the IPv4 address 255.255.255.255, the developer might allocate 15 bytes when he or she actually needs 16 bytes to accommodate the null terminator at the end. One additional character is needed for the null terminator, a byte whose value is zero that defines the end of the string. While languages such as Fortran store a count of how many characters the string contains as part of the string data structure, C doesn’t do that. If the marker that indicates the end of the string is missing, then the software doesn’t know that it needs to stop. Instead, it keeps searching through the memory. While the previous rules and recommendations are intended to prevent vulnerabilities that eventually lead to security problems, neglecting to include a byte for the null terminator leads directly to a buffer overflow. Future Work Since publishing the first version of the CERT C Secure Coding Standard, we’ve learned and improved our approach to vulnerability analysis and developing rules and recommendations. A new version of the CERT Secure Coding Standard will eventually be published to update the existing rules and recommendations, as well as to add some for new C features, such as the standard C multithreading library. Meanwhile, the current work has already met with success. Cisco and Oracle have adopted the CERT C Secure Coding Standard as part of their internal processes.  We continue to hear of additional interest from various organizations. Additional ResourcesFor more information about the CERT Secure Coding initiative, please visithttps://www.cert.org/secure-coding/

By David FrenchSenior Malware ResearcherCERT In previous blog posts, I have written about applying similarity measures to malicious code to identify related files and reduce analysis expense. Another way to observe similarity in malicious code is to leverage analyst insights by identifying files that possess some property in common with a particular file of interest. One way to do this is by using YARA, an open-source project that helps researchers identify and classify malware. YARA has gained enormous popularity in recent years as a way for malware researchers and network defenders to communicate their knowledge about malicious files, from identifiers for specific families to signatures capturing common tools, techniques, and procedures (TTPs). This blog post provides guidelines for using YARA effectively, focusing on selection of objective criteria derived from malware, the type of criteria most useful in identifying related malware (including strings, resources, and functions), and guidelines for creating YARA signatures using these criteria. Benefits of Reverse Engineering for Malware Analysis Reverse engineering is arguably the most expensive form of analysis to apply to malicious files. It is also the process by which the greatest insights can be made against a particular malicious file. Since analysis time is so expensive, however, we constantly seek ways to reduce this cost or to leverage the benefits beyond the initially analyzed file. When classifying and identifying malware, therefore, it is useful to group related files together to cut down on analysis time and leverage analysis of one file against many files. To express such relationships between files, we use the concept of a "malware family", which is loosely defined as "a set of files related by objective criteria derived from the files themselves." Using this definition, we can apply different criteria to different sets of files to form a family. For example, as I described in my last blog post, Scraze is considered a malware family. In this case, the objective criteria forming the family are the functions that Scraze files have in common.  Likewise, NSIS files can also be considered a family, though a reasonable person might conclude that a common installation method for multiple different programs (such as NSIS provides) is an irrelevant relationship between those programs. In this case, we may also use function sharing as the objective criteria forming the family. Whatever the criteria selected to identify a malware family, we generally find that it is desirable for these criteria to have the following properties: The criteria should be necessary to the behavior of the malware. Ideal candidates are structural properties (such as a particular section layout or resources needed for the program to run) or behavioral properties (such as function bytes for important malware behavior). The criteria should also be sufficient to distinguish the malware family from other families. We often find that malicious files accomplish similar things but perhaps in different ways. For example, many malicious files detect that they are running in a virtual environment, and there have been many published techniques on how to implement this behavior. Since these techniques are published on the Internet, it is trivial for a malware author to incorporate them into his own programs, and we expect this to occur. Thus, using these published examples as criteria indicative of one particular malware family is probably not sufficient to distinguish that family. Applying YARA Signatures to Malware Analyst experience and intuition can guide the selection of good criteria, as discussed in my prior work on the process of selecting and applying criteria. The goal of developing these criteria is to use them to identify related files. In practice, we generally find that good criteria for distinguishing and identifying malware families are excellent targets for creating signatures that identify the families. One way to encode signatures that identify malware families is by using the open source tool YARA. YARA provides a robust language (based on Perl Compatible Regular Expressions) for creating signatures with which to identify malware. These signatures are encoded as text files, which makes them easy to read and communicate with other malware analysts. Since YARA applies static signatures to binary files, the criteria statically derived from malicious files are the easiest and most effective criteria to convert into YARA signatures. We have found three different types of criteria are most suitable for YARA signature development: strings, resources, and function bytes. The simplest usage of YARA is to encode strings that appear in malicious files. The usefulness of matching strings, however, is highly dependent on which strings are chosen. For example, selecting strings that represent unique configuration items, or commands for a remote access tool, are likely to be indicative of, and specific to, a particular malware family. Conversely, strings in a malicious file that result from the way the file was created (such as version information stored by the Microsoft MSVC++ compiler) are generally poor candidates for YARA signatures. Here is an example of a YARA signature for the malware family Scraze, based on strings derived from the malware: rule Scraze{  strings:    $strval1 = "C:\Windows\ScreenBlazeUpgrader.bat"    $strval2 = "\ScreenBlaze.exe "  condition:    all of them} Another effective use of YARA is to encode resources that are stored in malicious files. These resources may include things like distinctive icons, configuration information, or even other files. To encode these resources as YARA signatures, we first extract the resources (using any available tool, for example Resource Hacker) and then convert the bytes of the resource (or a portion thereof) to a hexadecimal string that can be represented directly in a YARA signature. Icons in particular make excellent targets for such signatures. However, resources are easily modified in a program. For example, Microsoft Windows allows a program’s icon to change by simple copy/paste operations in Windows Explorer. The same care must be taken in selecting program resources as is taken when selecting strings. Finally, an effective use of YARA signatures is to encode bytes implementing a function called by the malicious program. Functions tend to satisfy our desire for criteria both necessary and sufficient to describe the malware family. The best functions to encode are those that perform some action deemed indicative of the overall character of the malware. For example, the best function for malware whose primary purpose is to download another file may be one that performs the download or processes downloaded files. Likewise, for remote access tools, it may be a function to encrypt network communications or to process received commands. For viruses, it may be code (which may not even be an actual function) involved with decrypting a payload or re-infecting additional files. We may also identify packers by encoding bytes representing the unpack stub. Here is an example of a YARA signature encoding part of a function used to check the integrity of an NSIS installer created with NSIS version 2.46. This basic block has had address bytes wildcarded based on the PIC algorithm (described in the article "Function Hashing for Malicious Code Analysis" in the 2009 CERT research report). rule NSIS_246{   strings:       $NSIS_246_CheckIntegrity = { 57 53 E8 ?? ?? ?? ?? 85 C0 0F 84 ?? ?? ?? ?? 83 3D ?? ?? ?? ?? 00 75 ?? 6A 1C 8D 45 D8 53 50 E8 ?? ?? ?? ?? 8B 45 D8 A9 F0 FF FF FF 75 ?? 81 7D DC EF BE AD DE 75 ?? 81 7D E8 49 6E 73 74 75 ?? 81 7D E4 73 6F 66 74 75 ?? 81 7D E0 4E 75 6C 6C 75 ?? 09 45 08 8B 45 08 8B 0D ?? ?? ?? ?? 83 E0 02 09 05 ?? ?? ?? ?? 8B 45 F0 3B C6 89 0D ?? ?? ?? ?? 0F 8F ?? ?? ?? ?? F6 45 08 08 75 ?? F6 45 08 04 75 ?? }   condition:       $NSIS_246_CheckIntegrity} Caveats in Creating YARA Signatures Regardless of the criteria used to create YARA signature, there are always caveats, especially for criteria derived from program data, such as strings. For example, malware with statically coded password lists may have a large number of strings, including those that may seem to unique to a family. Moreover, since strings are easily mutable, a string (such as the filename or path into which a malicious file is installed, or common encoding strings used in HTTP requests) considered as indicative may change at any time. The analyst must assess the significance of the presence or absence of a particular string, rather than delegate responsibility of understanding to a YARA signature. Likewise, when using code instead of data to create YARA signatures, we must also be aware of the caveats. It is important to use functions that are not provided from a common implementation detail, such as common libraries that have been statically linked into the program. It is also important to account for differences in malicious files due to process changes, such as recompilation. One way to sift out these differences is to use algorithms that normalize address references (for example, the PIC algorithm) when selecting function bytes. Malicious code may change over time, so particular functions may come and go. It is therefore better to select a number of functions to encode as signatures and derive the "best" signatures by surveying matching files and refining the analysis as the importance of the signatures is revealed. This method is the essence of family analysis, and is an important application of YARA to malware analysis. Open Issues and Future Work While YARA continues to gain in popularity, there aren’t many guidelines on how to use it most effectively. The issue of false positives continues to vex malware analysts. For example, YARA signatures may match files in which the specified criteria exist and yet do not possess the same semantics as expressed in the original file. Another issue with using YARA signatures is that they can be fragile in the face of changing codebases. A malware author can and does change his/her code to suit an ever-shifting set of goals, and keeping up with these changes is particularly challenging. While YARA signatures are a powerful means of capturing and communicating analyst insights, care must be taken that they do not drift too far away from the current reality of a particular malware family.  To address these open issues, our future work is to continue refining the process by which malware families may be most reliably identified. This work includes developing metrics for the best type of criteria to define families, measuring the resilience of these criteria in the face of changes, and evaluating the cost of developing the criteria vs. the cost to change the malware. We are also developing techniques to prioritize malware analysis based on such metrics, as well as triage systems to associate related files by automatically producing and testing high-quality signatures. Additional Resources: Function Hashing for Malicious Code Analysis, CERT Research Report, pp 26-29www.cert.org/research/2009research-report.pdf

By Linda Parker GatesSenior Member of the Technical StaffAcquisition Support Program Organizational improvement efforts should be driven by business needs, not by the content of improvement models.  While improvement models, such as the Capability Maturity Model Integration (CMMI) or the Baldrige Criteria for Performance Excellence, provide excellent guidance and best practice standards, the way in which those models are implemented must be guided by the same drivers that influence any other business decision.  Business drivers are the collection of people, information, and conditions that initiate and support activities that help an organization accomplish its mission. These drivers should be the guiding force behind performance improvement because they represent key factors or influences that matter to an organization’s success.  But how do we identify these drivers? This blog posting, the latest in a continuing series on the SEI’s work on strategic planning, describes how we are using integrated strategic planning and the associated information framework to derive the most vital business drivers for performance improvement.  An Integrated Strategic Planning Method The strategic planning method we’ve been using at the SEI integrates the following two complementary techniques that provide a framework for identifying business drivers for performance improvement: Critical success factors (CSFs), which are indicators that measure how well an organization is accomplishing its goals. For example, a CSF for agile software projects is achieving a high-level of client-developer interaction. Future scenarios, which allow organizations to explore multiple potential futures and generate robust strategies and the early warning signs that indicate how the future may unfold. For example, weather experts will create scenarios based on the critical uncertainties associated with a major weather system and plan for the range of possibilities, while monitoring the variables and narrowing on the most likely scenario over time.  Our integrated strategic planning approach (described in my February 2011 blog post and a November 2010 SEI technical report) sets the stage for initiatives, such as architecture analysis of alternatives, performance improvement, risk management, and portfolio management, that an organization can apply to improve its performance at multiple scales, ranging from individuals and teams up to the entire enterprise.  Tying performance improvement to organizational strategy by identifying key business drivers creates an environment that is business-driven and model-based, but not model-driven. In other words, improvement decisions are informed by best practice models—such as the CMMI, the Baldrige Criteria for Performance Excellence, the Information Technology Infrastructure Library (ITIL), and the Project Management Body of Knowledge—but are driven by business concerns, rather than by an attempt to apply a particular model for its own sake. Linking Strategic Planning to Performance Improvement Through the SEI’s strategic planning and performance improvement work with federal acquisition program offices, we’ve observed that key business drivers can and should be elicited from integrated strategic plans.  In particular, aligning improvement activities with organizational strategic goals and CSFs helps ensure improvement activities achieve business goals. We’ve also learned that there is no one-size-fits-all improvement solution, that is, no single model improves performance across-the-board. Instead, organizations often see better results when they apply the most applicable parts of multiple models based on strategic business-driven information. When improvement initiatives and activities are directly derived from organizational goals, objectives, and CSFs, they can support and complement strategic initiatives and actions.  We particularly like how broad frameworks, such as the Malcolm Baldrige Criteria for Performance Excellence, can be used to identify general initiatives. The Baldrige Customer Focus criteria, considered with regard to the organization’s customer goals and coupled with input from a strategic plan, might lead the organization to improve the resilience of their customer-facing web services, which can then be augmented with specific actions (such as ensuring that high priority alerts from incident detection systems are resolved within 60 minutes) guided by the CERT Resilience Management Model. This multi-model combination enables an organization to select the improvement model(s) and practices according to what will best support their business objectives (such as preserving the confidentiality of customer data), rather than according to model-based criteria (such as maturity levels).  Business-Driven Performance Improvement To showcase the way that integrated strategic planning can help an organization understand its business drivers for improvement, consider an information technology (IT) group with the mission of acquiring IT systems that support the services provided to the broader company’s customers.  The IT group’s CSFs would reflect the following operational areas that must function well to meet its mission: acquiring IT systems that serve customer needs  managing and tracking a budget that is adequate for the mission formally managing relationships with key internal and external stakeholders through communication, managing expectations, and personal interaction Likewise, the IT group’s strategic goals might include the following: Deliver service at customer locations, not just at traditional in-house IT data centers and facilities Enable highly usable self-service systems to maximize the ability of customers to transact business without intervention by IT group staff Develop and certify qualified, competent project managers, project team members, and portfolio and project oversight staff members to manage IT projects successfully Progress toward the goals and attention to the CSFs described above might involve improvement actions associated with maturing the IT group’s requirements definition process. Specifically, the strategic goals outlined above could lead the IT group to the Baldrige Customer Focus category for criteria on Customer Listening. This goal would also indicate value in CMMI-ACQ process areas, such as CMMI for Acquisition (CMMI-ACQ) and the following practices: ARD - Acquisition Requirements Development o  Specific Practice 1: Develop Customer Requirementso  Specific Practice 3: Analyze and Validate Requirementso  Generic Practice 2.5: Train Peopleo  Generic Practice 2.7: Identify and Involve Relevant Stakeholderso  Generic Practice 3.1: Establish a Defined Process  The goal might also lead the organization to practices in CMMI for Services (CMMI-SVC). Future scenarios might help expose a driver about the potential for a dramatic change in the IT group’s workforce over the next 5 to7 years.  While scenarios do not present certainties, they present opportunities to develop robust strategies that will serve the organization well, regardless of the outcome of a high-impact uncertainty.  Awareness of a critical uncertainty around the size of the workforce (due to economic conditions, aging, competition for talent, etc.) might help the IT group justify focus on the Baldrige Workforce Focus category for criteria on Workforce Capability and Capacity, as well as the People Capability Maturity Model (P-CMM). Ideally, the IT group will have fully aligned strategic plans and improvement plans that identify the essential knowledge and best practices contained in the performance improvement models that are most relevant for their specific business drivers.  Current and Future Work on Integrated Strategic Planning We are currently working with an organization that is undertaking 48 improvement actions, each derived from organizational goals and CSFs and tied to one of four performance improvement models, including the P-CMM, CMMI-ACQ, CMMI-SVC, and the Baldrige Criteria for Performance Excellence.  By aligning the work associated with the improvement models to the organization’s strategic plans, the improvement measures are tied to organizational goals and CSFs. Our work on this project thus far has provided compelling evidence that the integrated strategic planning process described in this blog posting is well-suited to identifying business drivers for performance improvement.  More broadly, an organization’s store of knowledge and experience, embodied in people and captured for communication and use in the processes, practices, and procedures of the organization, help an organization react to change.  In today’s dynamic mission contexts, the ability of an organization to react to changes in its mission environment is a critical capability.  We are exploring the use of an agile strategic planning process that links performance improvement to organizational agility.  We look forward to sharing our results with you in a forthcoming post. Additional Resources For more blog postings on strategic planning, please visit http://blog.sei.cmu.edu/archives.cfm/category/strategic-planning I recently attended the SEPG Europe conference in Madrid, Spain, where I delivered a presentation about using strategic planning techniques to identify business drivers for multi-model performance improvement.  My presentation was on Tuesday. http://www.sei.cmu.edu/sepg/europe/2012/

By Bill PollakTransition Manager Research, Technology, & System Solutions It is widely recognized today that software architecture serves as the blueprint for both the system and the project developing it, defining the work assignments that must be performed by design and implementation teams. Architecture is the primary purveyor of system quality attributes that are hard to achieve without a unifying architecture; it’s also the conceptual glue that holds every phase of projects together for their many stakeholders. Last month, we presented two posting in a series from a panel at SATURN 2012 titled "Reflections on 20 Years of Software Architecture" that discussed the increased awareness of architecture as a primary means for achieving desired quality attributes and advances in software architecture practice for distributed real-time embedded systems during the past two decades. This blog posting—the next in the series—provides a lightly edited transcription of a presentation by Robert Schwanke, who reflected on four general problems in software architecture: modularity, systems of systems, maintainable architecture descriptions, and system architecture. Robert Schwanke, Siemens Corporate Research We’ve been using the term "software architecture" for about 20 years, but the foundations of the concept go back another 20 years, to the information-hiding principle introduced by David Parnas in 1972. So, we’ve actually had 40 years of software architecture. Parnas also talked about hierarchical structure in 1974 and data encapsulation in 1975. Some classic papers from that era are listed at the end of this article. We still lean on these and other early principles. In fact, if we look around for general principles of software architecture, there are not many new ones. But we do have important, unsolved, general problems in software architecture.  Today I want to draw your attention to four problem areas: modularity, systems of systems, maintainable architecture descriptions, and system architecture. What is so hard about modularity today? According to Parnas, modules were supposed to decouple development tasks. But somewhere along the way, we got the idea that modularity is about syntactic dependency. It's not. It’s about dividing systems into separate chunks so that people can work on them independently, most of the time, not needing to talk to each other very often. To create a good decomposition, we need to know which development tasks should be decoupled, because we can’t decouple them all. Modular decomposition has to be a tree. At every node in the tree, we decide which tasks are most important to decouple and divide the subsystem into smaller pieces accordingly. Modularity is also about anticipating change. The marketplace, stakeholders, and technology can all change, altering the software’s requirements and the criteria for success. To get a perfect architecture, you must have perfect insight into the future to know what is going to change. How far into the future should you look when selecting tasks to decouple? If you look too far, you get an over-engineered system; if you don’t look far enough, your project may fail before its first delivery. My team is now working on measuring modularity. Past efforts at measuring it looked at coupling and cohesion, design similarity, and other measures, but we never really validated any of those measures—we could never show what the measurements were good for. These days we are looking at detecting modularity errors by contrasting code structure with change sets. For example, if certain pairs of files get changed together often—and there’s no syntactic explanation for why they’re being changed together—we suspect a modularity error. Modularity is supposed to keep things independent, but such pairs of files are not independent. We are combining one line of work by Yuanfang Cai at Drexel University, and another line by Alan MacCormack’s team at MIT and Harvard Business School that are both studying how to predict future change—where changes will most likely happen in the system—using structure measures and change-history measures together. Preliminary indications are that file size is still the best single predictor of future bugs. This seems intuitive—bigger files means more bugs—except that the bug density in large files turns out to be lower than in small files. Not the number of bugs—that is still higher—but the bug density is lower. Another interesting predictor, coming from social-network research, is "betweenness centrality." Centrality is how much a node in the network is in the middle of the network—specifically, the frequency with which it appears on the shortest path between any pair of nodes in the network—and it’s a pretty strong predictor of future change. The reason centrality is a good predictor is that if changes are likely to propagate through this node, the node is likely to change. Another hard problem is technology stacks. Specialization forces us to rely heavily on third-party components and technologies, and not just in ultra-large scale systems. I worked on a small system recently in which the first draft implementation, installed on my desktop, used 15 third-party technologies. By the time we delivered the system, it contained 300 open-source components, protected by 30 distinct open-source licenses, which gave us many headaches even though we hadn’t modified any of the source files. When that happens, your system loses control over aggregated quality attributes. When I was working on a VOIP (Voice Over Internet Protocol) telephone-switch product a few years ago, there were only four VOIP-switch vendors selling complete hardware and software solutions. They were all relying on third-party server hardware, specified by the VOIP vendor but sold by the server vendor directly to the VOIP customer. Servers today have a market window of 18 months, after which the vendor will change the design, typically to take advantage of new and better components. Due to that short market window, the server vendors have reduced what they spend on reliability analysis of the hardware. The telephone business was once famous for its five-nines (.99999) reliability. Not anymore, because they can afford neither to build their own servers nor to keep re-analyzing the reliability of third-party servers. There was one server vendor that was poised to take over the entire telephony server market just by offering a server with a 5-year market window and a  reliability specification. The next problem is maintainable architecture descriptions. We’ve been trying for a long time to put good, useful descriptions into the hands of architects and developers and get people to maintain them. Instead, the current practice is to figure out the architecture once, document it, put it on a shelf, and never change it again. Or, actually, we do change the architecture, but the description doesn’t change, and then it’s useless. The biggest obstacle to maintainable architecture descriptions is that the subsystem tree, often reflected in the directory structure of the project, is almost enough. Much of the value of the architecture description resides in the module decomposition tree. Making the rest of the architecture description accurate, enforceable, maintainable, and usable is really hard, and we have not yet demonstrated enough of a return on the expense. Finally, there is the challenge of system architecture. We realized recently that with the way the systems engineering field now defines itself [INCOSE standard], system architecture and software architecture are almost the same thing. That is, most large systems are now dominated by software, making the software architecture and system architecture almost the same. The domain-specific physical technologies define many of the components’ quality attributes, but the software provides the integration and control that synthesizes the system qualities out of the components. So we need to worry, as software architects, that we’re about to become system architects. In our emerging role, we need to add the physical, mechanical, and electrical components to our system architectures, but more importantly, on the people side, we must develop cross-domain communication, trust, and engagement. This requires a real engineering education that most software people don’t have. Instead, we have engineers with good, practical engineering training but an inadequate appreciation of software, and software guys who understand abstraction, dependencies, modularity, and so forth, but think they can build anything, whether it’s feasible or not. We software architects probably know a lot more about system architecture, in general, but we can’t speak the language that systems engineers have been talking for decades. The next post in this series will include presentations by Jeromy Carriere and Ian Gorton. Additional Resources Schwanke’s Presentation http://www.sei.cmu.edu/about/organization/rtss/?location=tertiary-nav&source=11347 Classic papers cited in Schwanke’s Presentation Information-hiding PrincipleThe Secret History of Information Hiding by David Parnashttp://www-sst.informatik.tu-cottbus.de/~db/doc/People/Broy/Software-Pioneers/Parnas_new.pdf Hierarchical StructureOn a "buzzword": hierarchical structure by David Parnashttp://gala.cs.iastate.edu/References/Parnas_Buzzword.pdf Data EncapsulationSome conclusions from an experiment in software engineering techniques by David Parnashttp://dl.acm.org/citation.cfm?id=1480035Modularization and hierarchy in a family of operating systems by A. N. Habermann, Lawrence Flon, and Lee Coopriderhttp://dl.acm.org/citation.cfm?id=360076 Separate Dependency Specs from CodeProgramming-in-the large versus programming-in-the-small by Frank DeRemer and Hans Kron     http://dl.acm.org/citation.cfm?id=808431 Module GuideThe modular structure of complex systems by Paul Clements, David Parnas, and David Weiss http://dl.acm.org/citation.cfm?id=801999

By Dr. Bill Claycomb Senior Member of the Technical StaffCERT Insider Threat Center Sabotage of IT systems by employees (the so-called "inside threat") is a serious problem facing many companies today.  Not only can data or computing systems be damaged, but outward-facing systems can be compromised to such an extent that customers cannot access an organization’s resources or products.  Previous blog postings  on the topic of insider threat have discussed mitigation patterns, controls that help identify insiders at risk of committing cyber crime, and the protection of next-generation DoD enterprise systems against insider threats through the capture, validation, and application of enterprise architectural patterns. This blog post describes our latest research in determining the indicators that insiders might demonstrate prior to attacks. The immediate and lasting impacts of IT sabotage by insiders can be catastrophic. In one case analyzed by researchers at the CERT Insider Threat Center, a disgruntled employee sabotaged IT systems hosting a child abuse hotline, preventing access to the organization’s website.  Another similar case resulted in severe limitations of 911 emergency services in four major cities.  In another instance, a company went out of business because an insider deleted all research data and stole all backups.  Since 2001, researchers at the CERT Insider Threat Center have documented malicious insider activity by examining publicly available information such as media reports and court transcripts. We have also conducted interviews with the United States Secret Service, victims’ organizations, and convicted felons. The goals of our research have been to answer the following questions: Are there patterns an employee exhibits prior to an attack? If so, what are they? If a pattern is applied, can it distinguish malicious insiders from all others?  And if so, is there a point at which the malicious insiders could have been detected or at which their behavior could have been identified as problematic? Previous research has been conducted on enabling and measuring early detection of insider threats, but several of the studies lacked access to the large collection of real-world cases our team has collected over the past 11 years. As part of our research, we focused on the timeline of events in cases of IT sabotage. Specifically, we looked at the following: the types of behavior that were observable prior to an attack patterns or models we can abstract from the saboteurs the points at which the employer could have taken action to prevent or deter the employee from executing an insider attack, either by positively mitigating the employee’s disgruntlement or by protecting IT systems. Our analysis was based on more than 50 cases of insider IT sabotage (other types of insider threat behavior include fraud and theft of intellectual property). From the selected cases, we created a chronology of events for each incident.  The number of events per insider incident ranged from 5 to more than 40, with an average of 15. We began by trying to identify specific events in each case that represented key points of the incident.  These key points are described as follows:   Tipping Point (TP) the first observed event at which the insider clearly became disgruntledIn one case we examined, it was reported that an "insider had a dispute with management regarding salary and compensation." Malicious Act (MA) the first significant observed event that clearly enabled the attackIn the case described above, it was reported that the "insider inserted a logic bomb into production code." Attack Zero Hour (0H) when cyber damage begins to occurIn our example, the next sequence of events involved "logic bomb fires, deleting massive amounts of critical company data." Attack Detected (AD) when the organization realizes something is wrongIn our example, the "system administrator arrives at work and discovers the system is down." Attack Ends (AE) when cyber damage stops (not when recovery ends or even begins)In our example, the "system administrator finds and deletes logic bomb, preventing future damage." Action on Insider (AI) the first instance of organizational response to the insider (fired, arrested, etc.)In our example, the insider was fired and a search warrant was executed "to find missing backup tapes at insiders residence." Initial Findings After we determined which events corresponded to each key point, we analyzed each case to determine whether the events or behaviors prior to each event indicated a predisposition for sabotage.  For this project, we considered predispositions to be characteristics of the individual that can contribute to the risk of behaviors leading to malicious activity.  For example, one issue we examined is whether an employee—prior to the point of clear disgruntlement—demonstrated a serious mental health disorder, an addiction to drugs or alcohol, or a history of rule conflicts. Although we are still conducting analysis, two patterns have emerged: In general, insiders begin conducting attacks soon after reaching a tipping point of disgruntlement. Insiders tend to exhibit behavioral indicators prior to exhibiting technical indicators.  In particular, concerning behaviors of an interpersonal nature were generally observed prior to concerning behaviors on IT systems. Addressing Challenges One of the challenges we face in our work is measuring early detection with respect to the moment of attack. Specifically, we have found the following factors particularly troublesome: Sabotage attacks are often complicated, and it’s hard to pinpoint specific event timing. Defining an attack is not simple In particular, does the attack include the time spent on planning?  Is planting a logic bomb considered the attack, or does the attack begin when the logic bomb executes? Early detection times may vary according to analysis parameters, and timing fidelity is inconsistent.  For some cases we have timing information down to the minute; for others, we only know the day the event occurred. Deciding which events were observable is hard.  Does this mean "capable of being observed" (some system exists to observe the behavior), or "capable of being observed in each specific case" (the organization possessed and correctly utilized the tool to detect the behavior)? Measuring employee disgruntlement is hard.  Behavior that indicates disgruntlement for one person may be normal behavior for another.  Trying to identify a point (or set of points) in a case timeline where the insider clearly became disgruntled, or where the disgruntlement became markedly worse, is therefore highly subjective. Another difficulty we have experienced is a scarcity of detailed data. While we have several hundred cases to choose from, our sources for these cases are sometimes limited to court documents, media reports, etc., which generally do not contain detailed technical or behavioral descriptions of the insiders’ actions prior to attack.   To help maintain integrity of our results and evaluation methodologies, we are collaborating with Roy Maxion from CMU’s School of Computer Science, who is well known for his research in research methodologies and data quality issues. Caveats One important factor to note is that our data is somewhat biased, as we only consider malicious insiders who have been convicted of a crime related to their insider activity, and we are limited in the scope of data sources used.  The number of insiders who are not detected, or detected but not reported, is probably much greater than the number of insiders convicted.  Moreover, our results are not generalizable to the entire scope of IT sabotage. They do, however, provide some of the best evidence available for researchers and practitioners to develop novel controls for preventing and detecting some types of sabotage - including the types of high-impact crimes that result in prosecution and conviction of the insider. Impact of our Work and Future Plans Through this research, we plan to equip organizations, companies, and even government agencies with improved insider threat detection capabilities. We believe that our work will be of particular relevance to programs in multiple sectors of industry and finance throughout the United States. We also hope that our findings will establish a foundation for future research. Specifically, we are interested in leveraging our findings to develop controls, such as technical and non-technical methods of preventing and detecting insider threat. Additional Resources CERT Insider Threat Blog www.cert.org/blogs/insider_threat/ Chronological Examination of Insider Threat Sabotage: Preliminary Observations presented at the International Workshop on Managing Insider Security Threats www.cert.org/archive/pdf/CERT_CodingSabotage.pdf The "Big Picture" of Insider IT Sabotage Across U.S. Critical Infrastructures, Technical Report, May 2008 www.cert.org/archive/pdf/08tr009.pdf Comparing Insider IT Sabotage and Espionage: A Model-Based Analysiswww.cert.org/archive/pdf/06tr026.pdf

By Troy Townsend, Senior Analyst SEI Innovation Center The majority of research in cyber security focuses on incident response or network defense, either trying to keep the bad guys out or facilitating the isolation and clean-up when a computer is compromised. It’s hard to find a technology website that’s not touting articles on fielding better firewalls, patching operating systems, updating anti-virus signatures, and a slew of other technologies to help detect or block malicious actors from getting on your network. What’s missing from this picture is a proactive understanding of who the threats are and how they intend to use the cyber domain to get what they want. Our team of researchers—which included Andrew Mellinger, Melissa Ludwick, Jay McAllister, and Kate Ambrose Sereno—sought to help organizations bolster their cyber security posture by leveraging best practices in methodologies and technologies that provide a greater understanding of potential risks and threats in the cyber domain. This blog posting describes how we are approaching this challenge and what we have discovered thus far.Earlier this year, representatives from the government approached the SEI Innovation Center about conducting research to assess the state of the practice of cyber intelligence. Specifically, we were asked us to accomplish three core tasks: capture the state of the practice of cyber intelligence, specifically how cyber intelligence is being performed across private industry and government create an implementation framework that captures best practices and advances the state of the art prototype technology solutions that advance the state of the science The overall intent is to expose industry to the best practices in capabilities and methodologies developed by the government, and for the government to learn from the process efficiencies and tools used in industry.  In areas where both the government and industry are experiencing challenges, the SEI can leverage its expertise to develop and prototype innovative technologies and processes that can benefit all participants in the program.ScopeWe identified 25 organizations to participate in our research including federal agencies    international law firms universities    financial sector non-profits    energy sector commercial intelligence providers     retail sector Our intent was to not rate participating organizations as good or bad, but rather to capture their processes, tools, and understanding of cyber intelligence as a means of enhancing cyber security. To accomplish this, we created a cyber intelligence framework that captured the core, fundamental components of a cyber intelligence process. Based on this framework, we devised interview questions researchers used to learn how organizations accomplished those core components, which we identified as: defining the cyber environment. Questions in this area—by far the broadest category—focused on the organization’s cyber footprint, their identified risks, threats, and overall cyber intelligence organization. We asked organizations about everything from the composition of their cyber intelligence component to the methods and techniques analysts use for identifying emerging cyber threats. data gathering. Data gathering is largely derived from requirements identified by defining the organization’s environment. If the environment is too broadly defined ("everything is a threat!"), then data gathering becomes inefficient, and analysts are burdened with more data than they can possibly use.  If the environment is too narrowly defined, chances are the right data is not being collected, and the organization may be missing indicators of adversary activity. Questions in this area focused on data sources, tools, and processes of collecting and correlating data.  functional analysis. Functional analysis is the technical assessment or niche analysis of data, which includes functions such as malware analysis, insider threat analysis, reverse engineering, supply chain, intrusion analysis, and forensics. This type of analysis tends to answer the "what is happening" and "how is this happening" questions.  The primary utility in this analysis is to contribute directly to cyber security efforts and assist network defenders in remediating security gaps. strategic analysis. If functional analysis looks at the "what" and the "how," strategic analysis attempts to answer the "who is doing this" and "why are they doing this" questions.  Strategic analysis often incorporates functional analysis and conveys the complexity of the technical details to leadership in a way that they can understand and appreciate, such as how a cyber event impacts the organization’s strategic goals. The Cyber Intelligence Framework Mind the GapsFor each participating organization, we applied the organization’s workflows and processes to this framework. One challenge that we identified early on in both government and industry is that a language gap exists between functional analysts and decision makers. Often, leadership and decision makers don’t understand the technical nature of the functional analysis, such as what malware is and how and why it works. In the more effective cyber intelligence programs we observed, strategic analysts are able to translate that functional data in such a manner that decision makers can understand it and use it to make smarter security and business decisions. This translation helps the organization’s leadership better understand events such as distributed denial-of-service attacks (DDoS attacks). In a DDoS attack, the functional analysis provides technical details of the attack, including its provenance and effects on the server. Strategic analysis then applies those functional details to a broader view of how the DDoS attack impacts an organization’s business, how much money was lost as a result, and what could have been done to prevent it. Initial Findings In December, we plan to present the results of our findings to our customer. We will then begin working with organizations to address the challenges that we identified and incorporate best practices into their operations. Some of the initial challenges that we identified include a lack of consistent training for the strategic analysis role. Cyber intelligence is a relatively new area of expertise, and there is a dearth of senior mentors that can provide guidance to new analysts. Moreover, a consistent method of training has not been developed and the skills necessary to perform this function are not yet well defined. There are ongoing attempts to address these inconsistencies, and we plan to share our data to help professionalize the tradecraft of cyber intelligence. reliance on traditional intelligence methodologies. Intelligence methodologies were developed in an era when governments were looking at the inventories of the tanks, missiles, and airplanes held by hostile countries and predicting what the leaders of those countries planned to do with them. Applying these same processes, workflows, and tradecraft to the cyber domain is not always feasible. Technology changes so fast in the cyber domain that by the time a strategic-level product on an emerging threat makes it through the publication process, it’s already out of date. data gluttony. When we looked at the data gathering phase, in particular, we realized that organizations were inundated with data. Some organizations collected so much data that they simply discarded it without looking at it. Other organizations saved the data but did not use it effectively, and it continued to worthlessly accumulate in their servers. Other organizations collected data, but failed to correlate it. For example, an organization that requires employees to badge in at work may not be cross-checking those badge logs against employees who are logging in to the network remotely with a virtual private network (VPN). It stands to reason that an incident where an employee that has badged in to work but is also logged creating a VPN session from overseas may warrant some investigation. Future Work Our work thus far has focused on helping government leaders make smarter investments of the resources they use to secure the cyber infrastructure. In the coming months, after presenting our data to our sponsor, we will work with participating organizations to apply the best practices that we identified across their organizations. We’ve received permission from our sponsor to publish our results, so we intend to publish an SEI state of the practice report on cyber intelligence. In addition to the SEI state of the practice report, we aim to present our findings to a broader audience through presentations and panel discussions hosted by professional associations and Information Security conferences around.  For the coming year, we have received further sponsorship to develop prototype solutions to address some of the challenges we identified in this phase of our research. In January, we will begin working with engineers and participants from the study to develop and pilot these prototypes. Our work on the Cyber Intelligence Tradecraft Project won’t be the silver bullet solution to everyone’s cyber security problems. Instead, we hope that our research is a significant voice in an on-going conversation of how cyber intelligence analysis benefits risk mitigation and resource allocation in the cyber environment. We welcome your input to this conversation too!  Please add your comments in the section below. Additional Resources For information on the SEI Innovation Center, please visitwww.sei.cmu.edu/about/organization/innovationcenter/For information on the Atlantic Council History of Cyber Intelligence, please visithttp://ctovision.com/2012/10/lessons-from-our-cyber-past-history-of-cyber-intelligence/For another perspective on the value of cyber intelligence from RSA, please see http://blogs.rsa.com/stalking-the-kill-chain-position-before-submission/

By Paulo Merson, Visiting ScientistResearch, Technology, & System Solutions Occasionally this blog will highlight different posts from the SEI blogosphere. Today’s post by Paulo Merson, a senior member of the technical staff in the SEI’s Research, Technology, and System Solutions Program, is from the SATURN Network blog. This post explores Merson’s experience using Checkstyle and pre-commit hooks on Subversion to verify the conformance between code and architecture. Read More....

By Douglas C. Schmidt Principal Researcher As part of our mission to advance the practice of software engineering and cybersecurity through research and technology transition, our work focuses on ensuring the development and operation of software-reliant Department of Defense (DoD) systems with predictable and improved quality, schedule, and cost. To achieve this mission, the SEI conducts research and development (R&D) activities involving the DoD, federal agencies, industry, and academia. As we look back on 2012, this blog posting highlights our many R&D accomplishments. Our R&D benefits the DoD and other sponsors by identifying and solving key technical challenges facing developers and managers of current and future software-reliant systems. R&D work at the SEI focuses on the following four general areas of software engineering and cybersecurity: Securing the cyber infrastructure. This area focuses on enabling informed trust and confidence in using information and communication technology to ensure a securely connected world to protect and sustain vital U.S. cyber assets and services in the face of full-spectrum attacks from sophisticated adversaries. Advancing disciplined methods for engineering software. This area focuses on improving the availability, affordability, and sustainability of software-reliant systems through data-driven models, measurement, and management methods to reduce the cost, acquisition time, and risk of our major defense acquisition programs. Accelerating assured software delivery and sustainment for the mission. This area focuses on ensuring predictable mission performance in the acquisition, operation, and sustainment of software-reliant systems to expedite delivery of technical capabilities to win the current fight. Innovating software for competitive advantage. This area focuses on producing innovations that revolutionize development of assured software-reliant systems to maintain the U.S. competitive edge in software technologies vital to national security. Following is a sampling of the SEI’s R&D accomplishments in each of these areas during 2012, with links to additional information about these projects. Securing the Cyber Infrastructure A large percentage of cybersecurity attacks against DoD and other government organizations are caused by disgruntled, greedy, or subversive insiders, employees, or contractors with access to that organization’s network systems or data. Over the past 12 years, researchers at the CERT Insider Threat Center have collected incidents related to malicious activity by insiders from a number of sources, including media reports, the courts, the United States Secret Service, victim organizations, and interviews with convicted felons. The blog post Developing Controls to Prevent Theft of Intellectual Property described controls that researchers have developed to prevent, identify, or detect theft of intellectual property. A subsequent posting, A New SIEM Signature Developed to Address Insider Threats, explored controls developed to prevent, identify, or detect IT sabotage. Another aspect of insider threat research focused on identifying enterprise architecture patterns that protect an organization’s systems from malicious insiders. Enterprise architecture patterns are organizational patterns that involve the full scope of enterprise architecture concerns, including people, processes, technology, and facilities. Our goal with this pattern work is to equip organizations with the tools necessary to institute controls that will reduce the incidence of insider compromise. A blog post described research to create and validate an insider threat mitigation pattern language that focuses on helping organizations balance the cost of security controls with the risk of insider compromise. A final post described exploratory research to determine the indicators that insiders might demonstrate prior to attacks. New malicious code analysis techniques and tools being developed at the SEI will better counter and exploit adversarial use of information and communication technologies. Through our work in cybersecurity, we have amassed millions of pieces of malicious software in a large malware database. Analyzing this code manually for potential similarities and identifying malware provenance is a painstaking process. The blog post Modeling Malware with Suffix Trees outlined a method to create effective and efficient tools that analysts can use to identify malware more effectively. Another approach for identifying similarity in malicious code involves leveraging analyst insights by identifying files that possess some property in common with a particular file of interest. One way to do this is by using YARA, an open-source project that helps researchers identify and classify malware. YARA has gained enormous popularity in recent years as a way for malware researchers and network defenders to communicate their knowledge about malicious files, from identifiers for specific families to signatures capturing common tools, techniques, and procedures. The post Writing Effective YARA Signatures to Identify Malware provided guidelines for using YARA effectively, focusing on selection of objective criteria derived from malware, the type of criteria most useful in identifying related malware (including strings, resources, and functions), and guidelines for creating YARA signatures using these criteria. Our security experts in the CERT Program are often called upon to audit software and provide expertise on secure coding practices. The blog posting Improving Security in the Latest C Programming Language Standard detailed security enhancements— bounds-checking interfaces and analyzability—from the December 2011 revision of the C programming language standard, which is known informally as C11. Another post described our work on the CERT Perl Secure Coding Standard, which provides a core of well-documented and enforceable coding rules and recommendations for Perl, which is a popular scripting language. Advancing Disciplined Methods for Engineering Software According to a February 2011 presentation by Gary Bliss, director of Program Assessment and Root Cause Analysis, to the DoD Cost Analysis Symposium, unrealistic cost or schedule estimates frequently cause a program to breach a performance criterion. To help the DoD address this need, the SEI has continued its research into improving the accuracy of early cost estimates (whether for a DoD acquisition program or commercial product development) and ease the burden of additional re-estimations during a program’s lifecycle. The blog posting Quantifying Uncertainty in Early Lifecycle Cost Estimation (QUELCE) outlines a multi-year project conducted by the SEI Software Engineering Measurement and Analysis (SEMA) team. QUELCE is a method for improving pre-Milestone A software cost estimates through research designed to improve judgment regarding uncertainty in key assumptions (which are termed program change drivers), the relationships among the program change drivers, and their impact on cost. QUELCE asks domain experts to provide judgment not only on uncertain cost factors for a nominal program execution scenario, but also for the drivers of cost factors across a set of anticipated scenarios. A second installment in the series on QUELCE described efforts to improve the accuracy and reliability of expert judgment within this expanded role of early lifecycle cost estimation. On a separate front, a series of blog posts detailed a pilot of the Team Software Process (TSP) approach at Nedbank, one of the four largest banks in South Africa. The first post described how TSP principles allowed developers at the bank to address challenges, improve productivity, and thrive in an agile environment. The second post detailed how the SEI worked with Nedbank to address challenges with expanding and scaling the use of TSP at an organizational level. The third post explored challenges common to many organizations seeking to improve performance and become more agile and concluded by demonstrating how SEI researchers addressed these challenges in the TSP rollout at Nedbank. The TSP pilot teams at Nedbank made significant behavioral changes that not only improved the quality of the software but also team members' work lives by decreasing the need for evening and weekend overtime. The teams were able to make these improvements because they had project-specific measurements to guide their decisions, and they had the authority to implement those decisions. Based on the results of the pilots, Nedbank decided to implement TSP throughout the organization. Accelerating Assured Software Delivery and Sustainment for the Mission Another area of exploration for the SEI was methods and processes that enable large-scale software-reliant DoD systems to innovate rapidly and adapt products and systems to emerging needs within compressed time frames. The SEI has focused research efforts on improving the overall value delivered to users by strategically managing technical debt, which involves decisions made to defer necessary work during the planning or execution of a software project, as well as describing the level of skill needed to develop software using Agile for DoD acquisition programs and the importance of maintaining strong competency in a core set of software engineering processes. A blog post on this topic discussed how an architecture-focused analysis approach helps manage technical debt by enabling software engineers to decide the best time to rearchitect—in other words, to pay down technical debt. SEI researchers also focused their efforts on common problems faced by acquisition programs related to the development of IT systems, including communications, command, and control; avionics; and electronic warfare systems. Long development cycles don’t fit with user expectations, and in the DoD, it can take up to 81 months to acquire or develop a new technology.  To help bridge this gap, the SEI in 2012 hosted the Agile Research Forum, which brought together researchers and practitioners from around the world to discuss when and how to best apply agile methods in the mission-critical environments found in government and many industries. A multi-part series of blog posts highlighted key ideas and issues addressed at the forum, including Applying Agile at-Scale for Mission-Critical Software-Reliant Systems. This post, which recapped presentations by Anita Carleton, director of the SEI’s Software Engineering Process Management Program and Teresa M. Takai, chief information officer for the DoD, highlighted key ideas and issues associated  with applying agile methods to address the challenges of complexity, exacting regulations, and schedule pressures. Agile Methods: Tools, Techniques, and Practices for the DoD Community. This post, which recapped a presentation by Mary Ann Lapham, highlighted the importance of collaboration with end users, as well as among cross-functional teams, to facilitate the adoption of agile approaches into DoD acquisition programs. Strategic Management of Architectural Technical Debt. This post, which recapped a presentation by Ipek Ozkaya, discussed the use of agile architecture practices to manage strategic, intentional technical debt. Balancing Agility and Discipline at Scale. This post, which recapped a presentation by James Over, manager of TSP, advocated the building of self-managed teams, planning and measuring project process, designing before building, and making quality the top priority, among other principles associated with applying agile methods at-scale. Applying Agility to Common Operating Platform Environment Initiatives. This post, which recapped a presentation I gave at the forum, highlighted the importance of applying agile methods to common operating platform environments (COPEs) that have become increasingly important for the DoD to deliver enhanced integrated warfighting capability at lower cost, reduce acquisition and new technology insertion cycle time, and establish sustainable business and workforce strategies to support these goals. One problem that occurs when organizations are trying to adopt new practices is a disconnect in business strategy, values, and management style. These aren’t the only disconnects, but they are representative of what we often see when working with DoD or other regulated organizations trying to adopt agile methods. In the first installment in an upcoming series of blog posts, we introduced an analysis method called Readiness and Fit Analysis (RFA) that has been used for multiple technologies and sets of practices, most notably for adoption of CMMI practices, to identify and suggest strategies for mitigating common adoption risks. Engineering the architecture for a large and complex system is a hard, lengthy, and complex undertaking. System architects must perform many tasks and use many techniques if they are to create a sufficient set of architectural models and related documents that are complete, consistent, correct, unambiguous, verifiable, usable, and useful to the architecture’s many stakeholders. The SEI has developed the Method Framework for Engineering System Architectures (MFESA), which is a situational process engineering framework for developing system-specific methods to engineer system architectures. The first post in a two-part series on MFESA provided a brief historical description of situational method engineering, explained why no single system architectural engineering method is adequate, and introduced MFESA via a top-level overview of its components, describing its applicability, and explaining how it simultaneously provides the benefits of standardization and flexibility.  The second post in the series took a deeper dive into the four components that comprise MFESA: the MFESA ontology, which defines the foundational concepts underlying system architecture engineering the MFESA metamodel, which defines the base superclasses of method components the MFESA repository, which defines reusable method components the MFESA metamethod, which defines a process for creating project-specific methods using method components from the MFESA repository Innovating Software for Competitive Advantage For more than 10 years, scientists, researchers, and engineers used the TeraGrid supercomputer network funded by the National Science Foundation (NSF) to conduct advanced computational science. The SEI recently joined a partnership of 17 organizations and helped develop the successor to the TeraGrid called the Extreme Science and Engineering Discovery Environment (XSEDE). The first post in a multi-part series focused on the nature of software engineering practice in the context of the TeraGrid socio-technical ecosystem. From war zones in Afghanistan to disaster relief in countries like Haiti and Japan, today’s warfighter faces many challenges. The Advanced Mobile Systems Initiative at the SEI continues to focus on meeting the needs of the warfighter at the tactical edge, which is a term used to describe hostile environments with limited resources. Warfighters who use handheld devices face problems ranging from information obscurity (i.e., a lack of awareness of the available information) to information overload (i.e., too much information, coupled with an inability to locate truly vital information.). Through the development of context-aware mobile applications, which was described in a recent blog post, SEI researchers are exploring alternative sources of data that would not only push the limit of what could be done with user context, but also focus on the extremely challenging environment at the tactical edge. Another research effort is aimed at addressing a common phenomenon that takes place when the software development and maintenance effort involves several programmers and spans months or years: the source code exhibits an actual architecture that gradually diverges from the intended architecture. A post on the SATURN Network Blog describes one researcher’s experience using Checkstyle and pre-commit hooks on Subversion to verify the conformance between code and architecture. Concluding Remarks As you can see from this summary of accomplishments, 2012 has been a highly productive and exciting year for the SEI technical staff. Moreover, this blog posting just scratches the surface of SEI R&D activities. Please come back regularly to the SEI blog for coverage of these and many other topics we’ll be doing in the coming year. As always, we’re interested in new insights and new opportunities to partner on emerging technologies and interests. We welcome your feedback and look forward to engaging with you on the blog, so please feel free to add your comments below. Additional Resources For the latest SEI technical reports and papers, please visitwww.sei.cmu.edu/library/reportspapers.cfm For more information about R&D at the SEI as well as opportunities for collaboration, please visitwww.sei.cmu.edu/research/

By Sagar Chaki, Senior Member of the Technical StaffResearch, Technology & System Solutions A malicious program disrupts computer operations, gains access to private computational resources, or collects sensitive information. In February 2012, nearly 300 million malicious programs were detected, according to a report compiled by SECURELIST. To help organizations protect against malware, I and other researchers at the SEI have focused our efforts on trying to determine the origin of the malware. In particular, I’ve recently worked with my colleagues—Arie Gurfinkel, who works with me in the SEI’s Research, Technology, & System Solutions Program, and Cory Cohen, a malware analyst with the CERT Program—to use the semantics of programming languages to determine the origin of malware. This blog post describes our exploratory research to derive precise and timely actionable intelligence to understand and respond to malware. In a previous blog post, I described our efforts to use classification (a form of machine learning) to detect provenance similarities in binaries. Broadly, two functions are provenance similar if they have been compiled from similar source code using similar compilers. Malware programmers often draw upon similar source code with minor differences, different compilers (such as various versions of Microsoft Visual C++), or different levels of optimization.  In creating a training set to learn (or train) a classifier to predict the similarity of binaries, we realized several positive results including high accuracy and automation in performing classification. We also recognized, however, the following limitations: Machine learning is not 100 percent precise. In some instances the classifier reported that two binaries were similar that weren’t or vice versa. In a lab environment, machine learning is hard to validate at scale due to the substantial time commitment of researchers who must manually check thousands of samples. Our approach was limited to pairwise comparison (detecting whether any given pair of malware/functions are similar), which impedes scalability to full production-size data sets since detecting similarity among a set of size N requires O(N2) comparisons. Another approach that we researched, computing and comparing attribute vectors, also yielded low rates of accuracy and did not capture semantics. Our New Approach Our new approach is based on extending traditional syntactic clustering techniques that have been used to classify malware in the past with execution semantics. This approach involves two tasks: function clustering based on semantic hashes semantic difference analysis of functions    We summarize each approach below. Function Clustering Based on Semantic Hashes The first task of our work—function clustering based on semantic hashes—is based on semantic reasoning about programs, which is also called static analysis. We chose this approach because semantic summaries significantly reduce the number of clusters compared to syntactic hash-based clustering (For more information on the Pithos signature, please read the article by Cory Cohen and Jeffrey Havilla on page 28 of the 2009 CERT Research Annual Report)  while maintaining the quality of the cluster. Our approach involves using static analysis to determine the effect of a function (i.e., some relationship between its inputs and outputs) and then mapping functions that have the same behavior to the same equivalence classes. We thus determine that two functions are similar if we arrive at a similar result when statically computing the effect of each function.For example, the following functions can be identified as similar two functions that modify the same variable two functions that output the same value for the same input close outputs for close inputs (If a final input/output is 5 and another input output is 4, we’ll say that’s "close." If one output is 10 and one is 20, we’ll say "far.") Another aspect of this first task involves validating the static analysis and constructing a semantic hash or equivalence class. The validation involves constructing a benchmark from the CERT malware database and then performing clustering on that benchmark. There are many choices we can make, so we need to find the ones that actually work. This approach allows us to avoid the inefficient O(N2) pair-wise comparison problem.  In particular, equivalence classes are determined by hashing every binary/function individually and determining which ones have the same hash, not by comparing every pair of binaries/functions. Semantic Difference Analysis of Functions The second task in our approach focused on solving the pairing problem by semantically comparing two functions using a technique called regression verification, which converts the problem of comparing two functions to that of checking the equivalence of their input-output relationship expressed as logical formulas. This approach has been applied to other domains and problems including checking the equivalence of two programs available as source code. We wanted to broaden its scope by applying regression verification to binaries, which are low-level, executable code. To accomplish this task, we relied upon a platform called ROSE, which is a framework for handling binaries, as well as a binary analysis platform. The ROSE framework provides disassembly, instruction semantics, and other static analysis capabilities for us to build our semantic function comparison technique. Pairing our first task with semantic hashing allowed us to verify that our approach worked. In other words, we used the second technique to verify the first technique. The application of regression verification eliminates the need for researchers to spend countless hours manually verifying the correctness of every pair. It also provides a mechanism by which we could be reasonably confident in an automated method for verifying the correctness of pairs.  Combining the Two Tasks Semantic hashing provides a precise—but rough—idea of similarity among functions. For example, given a million functions, semantic hashing could identify 10,000 groups each with 10 functions.  In a particular group with 10 functions, all functions may not be similar. Researchers can then do a pairwise comparison, however, because each group is small in size. With 10 functions, researchers would have 45 pairs to check. Researchers can then apply regression verification (the second task) to take a closer look at each pair and prune out and refine each equivalence class. Mitigating Risks One challenge that we face is applying regression verification to binaries. While prior research has applied this technique to compare source code, applying it to binaries is non-trivial due to low-level details (e.g., bit-precise semantics that treat registers as 32-bit values instead of infinite-domain integers) and lack of structure (e.g., much harder to compute accurate control-flow graphs). Our approach to overcome this challenge consists of two parts: (i) logically partition system memory into three parts - the stack, the heap, and the region where parameters passed to functions are stored; and (ii) develop a semantics of executable code in terms of its effect on these three partitions of system memory. One risk we’ve identified is that the application of regression verification might not work well on very small functions since many have no inherent behavior (e.g., they wrap more complex functions), and they are trivially equivalent if we just consider their bodies. To mitigate that risk, we identified and pruned small functions from summarization. Our expectation is that this will have a marginal effect on analysis since small functions are not very helpful to analysts in general. Note that obfuscations that split large complex functions into many smaller ones are beyond the scope of this project; we assume that there are orthogonal deobfuscation techniques that deal with such issues. A second risk that we’ve identified is that the cost of computing precise semantic summaries may be expensive.  To mitigate this risk, our approach favored scalability over precision. We reasoned that it is better to create a scalable solution that yields precise summaries for 80 percent of functions than a non-scalable approach that could yield precise summaries for 99 percent of functions but only works on 10 percent of functions in a reasonable time. Impact to DoD Through this research we hope to reduce the amount of manual malware analysis that is required by the organizations seeking to protect valuable information system assets.  Reducing this analysis will lead to more cost-effective identifications and more timely responses to intrusions. This approach will also provide increased visibility into intruder behavior, leading to more effective defense against future intrusions, which are of increasing concern. Initial Results/Future Work We have implemented a tool (on top of ROSE) that computes two different types of semantic hashes, and empirically evaluated them on a benchmark derived from the CERT artifact catalog. Our paper, "Binary Function Clustering using Semantic Hashes," describing this research will appear in the Proceedings of the 11th International Conference on Machine Learning and Applications. Additional Resources To read the paper Binary Function Clustering Using Semantic Hashes, Wesley Jin, Sagar Chaki, Cory Cohen, Arie Gurfinkel, Jeffrey Havrilla, Charles Hines, Priya Narasimhan, Proceedings of the 11th International Conference on Machine Learning and Applications (ICMLA), December 12 to 15, 2012. To read the paper Supervised learning for provenance-similarity of binaries by Sagar Chaki, Cory Cohen, & Arie Gurfinkel, please visithttp://dl.acm.org/citation.cfm?id=2020419 To read the paper Regression verification by  Benny Godlin & Ofer Strichman, please visit http://dl.acm.org/citation.cfm?doid=1629911.1630034 To read about the research on Function Hashing for Malicious Code Analysis by CERT analysts Cory Cohen and Jeffrey Havilla, please see page 28 of the CERT Research Annual Report, which can be read at www.cert.org/research/2009research-report.pdf

By Bill PollakTransition ManagerResearch Technology & System Solutions It’s undeniable that the field of software architecture has grown during the past 20 years. In 2010, CNN/Money magazine identified "software architect" as the most desirable job in the U.S. Since 2004, the SEI has trained people from more than 900 organizations in the principles and practices of software architecture, and more than 1,800 people have earned the SEI Software Architecture Professional certificate. It is widely recognized today that architecture serves as the blueprint for both the system and the project developing it, defining the work assignments that must be performed by design and implementation teams. Architecture is the primary purveyor of system quality attributes, which are hard to achieve without a unifying architecture; it’s also the conceptual glue that holds every phase of projects together for their many stakeholders. This blog posting—the final installment in a series—provides lightly edited transcriptions of presentations by Jeromy Carriere and Ian Gorton at a SATURN 2012 roundtable, "Reflections on 20 Years of Software Architecture." Jeromy Carrierehttp://www.sei.cmu.edu/library/abstracts/presentations/carriere-panel-saturn2012.cfm In 1999 and 2000, I had just left the SEI to join a startup company that I co-founded. I came out of the SEI with great respect for software architecture. This [Please view Slide 2 from Carriere’s presentation.] was the top-level, logical view of the back end of my startup company’s software. Not many startup companies then—and certainly not now—do architecture models of this sort. This was an early model-driven design; it actually drilled down into mechanisms between components, and it generated code. That was cool, I had a little experimental testbed, and it was fun. All of these factors led me to develop a different working definition of architecture: A system’s architecture codifies a set of decisions that are both hardest to change and that have the most significant impact on the way the system manifests its quality attributes. This diagram [Please view Slide 6 from Carriere’s presentation.] helped us sell the company for $200 million to AOL. This is completely informal, but it reflects what’s important about architecture; that it was quality-attribute driven. We were making decisions as a small startup company based on what was important. What was paramount for us was flexibility. Above all else, we knew that we didn’t know where the business was going. In the very short life of the company, the business model changed substantially four times; the software architecture did not change once. Because the software architecture of the system was designed specifically to adapt—from a physical-device company to a voice-based thing, B2B, B2C… But this [refers to architecture diagram in slide] was maintained throughout. And that was enough to get us the deal. We sold the company to AOL, and then it failed. To put it bluntly, it was designed to be sold. We built our first product—AOL By Phone—in 90 days. From the day that we did the deal to the day that we deployed the system, including deploying hardware, it was 90 days. Later in my career, I still loved diagrams. This [Please view Slide 7 from Carriere’s presentation.] is a fantastic diagram labeled "critical systems interrelations overview," but it is useless. What good is that [refers to slide 7] besides being decorative? I wanted to be able to find a reason to use it in my day job, but it didn’t have any use. This was a period in my architectural career where I came to this conclusion and started to take on a different perspective. Last year at SATURN, I talked about low-ceremony architecture, which is how do you architecture like a startup. This is based on my Quack.com experience. [For more information about Carriere and his experience co-founding Quack.com, read the article "SEI Architecture Practices Propel Successful Startup."] If you want to get a job at Facebook, Twitter, Google, or Netflix, do not put "architect" on your resume. You will not get hired. Call yourself a lead engineer or something else, but not architect. In Silicon Valley, there’s a lingering allergic reaction to "Big A" Architecture, which is characterized by setting standards and making rules and drawing diagrams, those kinds of activities that we have all experienced. That disconnection from the actual engineering practice of developing software and what became architecture in some companies that were predecessors in the net space is deeply rooted in Silicon Valley. What I find most ironic is that these kinds of companies need architecture practice the most. This goes back to the Quack [Quack.com] story; we needed a perspective on quality attributes, prioritizing for flexibility, thinking about how you’re prioritizing architectural decision making because you’re living in a very uncertain world. Ian Gorton, Pacific Northwest National Laboratoryhttp://www.sei.cmu.edu/library/abstracts/presentations/gorton-panel-saturn2012.cfm We've seen our profession mature over the past 20 years. We've made advances in methods, processes and tools, but the scale of systems has grown enormously. But has there been anything truly innovative in the past five years in practice or research? While we apply the methods and tools that we know and love, the software world is getting more complex, and our solutions are very much based on qualitative engineering approaches. In the future, maybe we need to become more quantitative, applying more objectivity in our assessments and designs. There has been some initial work in many areas. I've done work in combining decision theory with optimization techniques to enable architects to quantitatively assess complex, multidimensional design alternatives. There are tools for modeling performance with multiple modeling approaches built into the tools, which makes them easy to use. So I think there's a lot of work we can leverage and extend here if we actually look toward future problems. It's not that we have to invent any mathematics or statistics; there's a rich body of math and stats that we can exploit. We just need to know when and how quantitative methods can be usefully applied. Quantification is in some ways harder, but it doesn't stop others from addressing really hard problems. Climate modelers, for example, build simulations that are being used by governments around the world to predict future climate and mitigating policies. These climate models are tested and validated against historical data. When they don't produce results that exactly match the historical data, they are calibrated to provide reasonable outputs. But there's no guarantee that these calibrated models are going to be able to predict the future. So if climate scientists can do this, we can too. Our current and emerging design and quantification models and methods may not (or may never) produce 100 percent accurate predictions of software architecture characteristics, but they will be much better than our current qualitative approaches. We have to move toward quantification because there are challenges in the exponential growth of system scale. There are huge software systems being employed in clouds that are so complex that our current architecture design methods are inadequate. To more rigorously address these systems of massive scale, we need concerted R&D [research and development] to move software architecture to more quantitative foundations. This is hard because, as you get to larger scales, things we know and understand well (such as design approaches, technologies, platforms) must be reexamined to address scale. This community is perfectly positioned to discover underlying principles and make these available in systematic approaches that can be broadly adapted to build the highly scalable systems of the future. Additional Resources Carriere’s presentation may be viewed at http://www.sei.cmu.edu/library/abstracts/presentations/carriere-panel-saturn2012.cfm Gorton’s presentation may be viewed athttp://www.sei.cmu.edu/library/abstracts/presentations/gorton-panel-saturn2012.cfm For more information about Carriere and his experience co-founding Quack.com, read the article "SEI Architecture Practices Propel Successful Startup" athttp://www.sei.cmu.edu/library/abstracts/news-at-sei/feature11q02.cfm

By James Edmondson,Senior Member of the Technical StaffResearch, Technology, & System Solutions An autonomous system is a computational system that performs a desired task, often without human guidance. We use varying degrees of autonomy in robotic systems for manufacturing, exploration of planets and space debris, water treatment, ambient sensing, and even cleaning floors. This blog post discusses practical autonomous systems that we are actively developing at the SEI. Specifically, this post focuses on a new research effort at the SEI called Self-governing Mobile Adhocs with Sensors and Handhelds (SMASH) that is forging collaborations with researchers, professors, and students with the goal of enabling more effective search-and-rescue crews. Motivating Scenario: An Earthquake Hits a Populated Area Recent earthquakes and earthquake-induced events like tsunamis in Haiti, Japan, and Indonesia have shown both the destructive power of nature and the compassion of communities to lend aid to people in need.  Compassion alone, however, won’t help emergency responders locate and help all of the potential survivors. Search-and-rescue crews thus need technology to help them utilize the often meager resources they have in the best way possible. Figure 1. Ground crews search for survivors after an earthquake hits a metropolitan area. Automated aerial robots aid crews with thermal cameras that can penetrate rubble to find the trapped and injured. Tackling a Massive Problem Search-and-rescue is a massive undertaking in every conceivable way.  There are issues of scale in geographic area coverage and number of people, devices, and sensors involved. Despite the numbers of people in need, emergency responders must deal with limited mobility and availability of ground crews and a complete loss of most telecommunications infrastructure, which is common after an earthquake. There are only so many people available to help, and it can take days to completely search a single building. Even in a small town, a crew of thousands of emergency responders could take weeks to find survivors, and by then it may be too late.To assist in this type of important mission, the SMASH project is exploring a combination of effective software design practices, artificial intelligence, and inexpensive hardware. These capabilities will allow one responder to control a fleet of quadcopters  or other robotic vehicles that search the rubble (for example, using thermal sensors, ground-penetrating radar, and other types of sensor payloads) and present the results to a smartphone or other human-computer device in a manner that’s useful and efficient to human operators, as shown in Figure 2. Figure 2. Automated aerial robots send information back to a smartphone, tablet, or other visual interface to provide ground crews with the thermal context of the area around the ground crews. We want to extend the capabilities of emergency responders by equipping them with dozens of robotic vehicles per operator, which would allow them to infer context from thermal images and locate survivors in the rubble. A Focus on Human-in-the-loop Autonomy Our central idea is to extend—not replace—the capabilities of human operators, most obviously by providing them with additional coverage. We also provide ground operators the ability to penetrate the rubble and look into dangerous or out-of-reach areas and locate survivors. Enhanced vision is useless, however, if robotic vehicles are not intelligent enough to search locations collaboratively with other agents.  Likewise, the vehicles need to use maps of building locations to proactively search likely survivor zones quickly and take ground operator feedback and guidance on the likely location of survivors. These are central components to our SMASH autonomous design philosophy. Figure 3. Aerial robots can land to form sensor networks for routing information from other personnel or sensors. Robust telecommunications infrastructure is another capability SMASH aims to provide. The robotic vehicles we are working on form their own wireless access points and also have a long-range radio (2 km) that may be useful in communicating with other ground crews or even by survivors with cellphones who need to send important messages, e.g., that they are trapped in a car of a parking garage. Part of the autonomy we are developing focuses on landing the robotic vehicles or using throwable wireless access points to form sensor networks that route information based on the priority of information. Figure 4. Routing and bandwidth allocation are dictated by contextual priority. In this image, two aerial robots detect thermal targets while the other three detect nothing. The two aerial robots have higher priority information and are given preferential bandwidth treatment. Our research on SMASH will also address issues of timing and scale. In a search-and-rescue operation, timing is critical, and data volumes from thousands of ground crews and new sensors can cause an ad hoc telecommunication infrastructure to fail. Consequently, autonomy is only useful if it is designed to work within the constraints of the available bandwidth of existing networking infrastructure.  Solving this problem depends on focusing on prioritization, quality-of-service, and effective bandwidth utilization. For example, in Figure 4, only two of the aerial robot quadcopters are detecting thermal targets. The other quadcopters are not detecting anything of use to the ground crews. Our solution approach focuses on only sending information that is relevant (such as thermal targets) and not constantly broadcasting the full thermal or video images unless specifically requested by the human operators. Moreover, the quadcopters that are perceiving potential human beings have higher priority and the ability to send more data to the operator within the networking bandwidth available. This type of autonomous self-interest is important to provide a scalable distributed infrastructure. Open-source dissemination Open architectures, i.e., open standards and open source initiatives that allow end-users to extend products and services,  are also important for critical infrastructures—not only because transparency helps in creating a better product, but also to reduce the cost of providing solutions to government agencies and non-profits. Consequently, SMASH is built on top of open-source initiatives including the Multi-Agent Distributed Adaptive Resource Allocation (MADARA) project, which provides real-time knowledge and reasoning services for distributed systems, and the Drone-RK project, which provides a programming interface for controlling and processing data from the Parrot AR.Drone 2. We are augmenting these open-source software projects with additional tools that the software community can use to enable human-in-the-loop autonomous system development and deployment. Collaborations To create these new autonomous techniques and tools, we are partnering with researchers and students from around the world. Within Carnegie Mellon University’s department of Electrical & Computer Engineering (ECE) within the College of Engineering, we have formed a core group of professors and students including Anthony Rowe, Larry Pileggi, and Kenneth Mai of the ECE departments, who will be working on extending the Parrot AR.Drone 2 and developing a new type of thermal, wireless-enabled throwable sensor. In the fall of 2013, Tony Lattanze of CMU’s Master of Software Engineering Program will work with students to investigate area coverage problems with collaborative robots and group artificial intelligence. Students from CMU will be joined by graduate students from research universities around the world for summer programs that investigate the search-and-rescue problem space in hopes of extending the state-of-the-art and effecting real world solutions to hard research problems. Contacts and Conclusion The SMASH project is driving the state-of-the-art in human-aided autonomy with an expected software payload that will be deployed to real-world and often mission-critical scenarios. If you have questions about the effort or would like to discuss potential collaboration possibilities, please contact  info@sei.cmu.edu or leave a comment below. Additional Resources For more information about the open-source knowledge and reasoning engine that we are using for distributed artificial intelligence in real-time systems, please visit the Madara project site at https://code.google.com/p/madara/ For more information on the software development kit used  for manipulating and reading images and data from the Parrot AR.Drone 2,please  visit the Drone-RK project site at http://drone-rk.org/ For a cool robotics emulation site that includes models of the Parrot AR.Drone, please try out the V-REP emulator (trial version lasts for 1 to 2 months). For more information please visithttp://www.v-rep.eu/

By Bill ScherlisSEI Principal Researcher and Director, Institute for Software Research The Department of Defense (DoD) has become deeply reliant on software. As a federally funded research and development center (FFRDC), the SEI is chartered to work with the DoD to meet the challenges of designing, producing, assuring, and evolving software-reliant systems in an affordable and dependable manner. This blog post is the second in a multi-part series that describes key elements of our forthcoming Strategic Research Plan that address these challenges through research, acquisition support, and collaboration with the DoD, other federal agencies, industry, and academia.  The first post in this series focused on Architecture-Led Incremental Iterative Development.  This part focuses on the remaining three elements of our strategic plan: (1) designed-in security and quality (evidence-based software assurance), (2) a set of DoD critical component capabilities relating to cyber-physical systems (CPS), autonomous systems, and big data analytics, and (3) cybersecurity tradecraft and analytics. Evidence-based Software Assurance and Certification The goal of the second element in the SEI Strategy Research Plan is a dramatic reduction in the cost and difficulty of making assurance judgments related to quality and security attributes. Achieving this goal is particularly important as systems become more complex and evolve more rapidly. Current approaches for certification and accreditation are largely based on an after-the-fact evaluation of a snapshot of a system. While after-the-fact approaches are effective for certain well-defined categories of components and systems, they tend to break down as systems increase in complexity, scale, and dynamism. They also tend to hinder ongoing evolution, rapid reconfiguration, dynamic loading of components, autonomy, and composition and interlinking of systems-of-systems. Put simply, these established techniques do not scale up, and they do not work well for the emerging software framework-based systems now prevalent in commercial and infrastructural applications. The industry folklore has long asserted that quality-related activities, including security-related assurance, can consume half of total development costs for larger systems. For example, the IBM Systems Journal states that, in a typical commercial development organization, "the cost of providing [the assurance that the program will perform satisfactorily in terms of its functional and nonfunctional specifications within the expected deployment environments] via appropriate debugging, testing, and verification activities can easily range from 50 to 75 percent of the total development cost." Additionally, after-the-fact evaluation practices can add a year or more to the elapsed time required to develop and deploy software-reliant systems. Commercial systems, including products and software as a service (SaaS), cloud-based systems, tend to undergo a relatively rapid and continual evolution. For many of our DoD and infrastructural systems, we similarly need to support a continuous evolution. Some areas of particular technical emphasis include Architecture and composition principles that enable separate evaluation of individual components, with the possibility of combining results to achieve aggregate assurance judgments. These principles are motivated by the reality of modern software supply chains, which are rich and diverse in sourcing and geography. Modeling and analytics to support the diversity of quality attributes significant to DoD and infrastructural systems. These include modeling, simulation, and design techniques that support critical security attributes. Exploration of development approaches that incorporate creation of evidence in support of assurance claims into the process of development. This evidence-based assurance can harmonize incentives to create designs and implementations that can more readily support evaluation. Evaluation and other techniques to support the use of more opaque components in systems, including binary components and potentially dangerous components from unknown sources. Part of the SEI Strategy Research Plan addresses incentives for developers to adapt their architectural structures, development process, and tooling to better accommodate the idea of amassing evidence—during the development process—that can support an eventual assurance claim with respect to quality and security attributes critical to the operation of the particular system. Our work benefits from the fact that the development process can be naturally incremental through the composition of components and the incremental validation of assurance claims. This development process is supported by the partial accumulation of engineered artifacts and evidence. These ideas of "designed-in security" build on the fourth theme of the Networking Information Technology Research & Development (NITRD) Program plan, "Trustworthy Cyberspace: Strategic Plan for the Federal Cybersecurity Research and Development Program." Critical Component Capabilities The goal of the third element in the SEI Strategic Research Plan is to enhance DoD software capability in several areas that have critical and pervasive roles in DoD software-reliant systems. These areas include composable, cyber-physical systems (CPS), autonomous and distributed systems, and high-performance, data-intensive computing.Each of the areas presents challenges: Composable cyber-physical systems (CPS). "Cyber-physical" refers to the fact that embedded software operates in the context of physical sensors and affectors. CPS thus includes control systems, real-time systems, and many other categories of systems pervasive in DoD and critical infrastructure. These systems tend to have greater complexity due, for example, to higher coupling and the need to model and manage associated physical system components. They also tend to need to assure that real-time deadlines are met. A consequence is that they typically manifest greater internal coupling in their design, thwarting higher levels of capability, composition, and flexibility.There are diverse technical challenges related to design and evaluation, which include improving modeling and analysis for cyber-physical and control systems, modeling and managing hardware reliability issues, making safe use of concurrency and effective scheduling to assure that performance goals can be met, providing support for analysis and testing, and advancing the overall stack architectures beyond legacy concepts. Many mobile systems fall in the category of cyber-physical, including ad hoc networks connecting with large numbers of sensors and enhanced mobile devices deployed to the field. Several recent blog posts outline efforts by SEI researchers to address these challenges.  Autonomous systems. Autonomous systems are cyber-physical systems that can accept sensor data and mission guidance, and, with very limited (or no) human interaction, arrive at mission decisions and enact outcomes. These systems are increasingly critical to the mission, and yet they pose particular challenges for verification and validation, since they rely so much less on ongoing human interaction.  Indeed, the capability and complexity of these systems are often limited for this reason. Systems must both behave as expected and, additionally, not manifest unwanted behaviors that can be dangerous or threaten mission success. From a technical perspective, autonomy can be particularly challenging because of the vast state space, the number of possibilities of combinations of inputs, and challenges of error tolerance, and the difficulty of fully modeling the environmental circumstances of their operation. High-Performance Computing (HPC) for analytics and for modeling and simulation. Advances in sensor fidelity, rapid growth in network capacity, increasing convergence in data-center and high performance computing architectures, advances in large-scale data storage, and emerging frameworks for scalable distributed computing (such as MapReduce and GraphLab) have all resulted in the growing phenomenon of "big data." There are many significant applications of big-data techniques in DoD and infrastructural systems—and in the development of those systems as well. Indeed, many of the other features of the SEI Strategic Research Plan build on progress in big data. Cybersecurity Tradecraft and Analytics The goal of the fourth strategic element is to advance analytic capability in support of diverse aspects of the cybersecurity mission. These aspects include analytics and situational awareness for malware, vulnerability categorization and assessment, vulnerability information management, network activity analysis, threat characterization and assessment, organizational security, and many other dimensions of operational response, remediation, and recovery. This capability builds on a range of data assets related to adversarial tradecraft, malware, vulnerabilities, insider threats, and other results of experience with large numbers of cybersecurity-related incidents. There are diverse purposes of this strategic element, including: Improvement in our understanding and communication of threats and risks Development of better preventive approaches in the engineering of systems and in managing secure operations including considerations for security and assurance "at scale", improved indications and warning, and near-real-time data analysis Support for forensic and corpus analysis Support for hypothesis generation using machine learning, near-real-time analysis, and other advanced capabilities What’s Next The next blog posting in this series focuses on our approach for evaluating and validating SEI research projects. The SEI Strategic Research Plan is designed to ensure that the SEI conducts high-quality and high-impact work that benefits the DoD by identifying and solving key technical challenges facing current and future DoD software-reliant systems. This strategy itself undergoes continual evolution and improvement; the broad range of SEI engagements enable us to continually refine the strategy as the technology advances, the mission evolves, and our understanding improves. We welcome engagement from our partners and stakeholders in the improvement and refinement of this strategy. Please leave comments below or contact us directly at info@sei.cmu.edu. Additional Resources To download the report Critical Code: Software Producibility for Defense, please see www.nap.edu/catalog.php?record_id=12979 To download the Report of the Defense Science Board Task Force on Defense Software (2000), please visit http://oai.dtic.mil/oai/oai?verb=getRecord&metadataPrefix=html&identifier=ADA385923 To view on-demand presentations from the SEI Agile Research Forum, please visitwww.sei.cmu.edu/go/agile-research-forum For more information about the Networking Information & Technology Research & Development Program, please visit www.nitrd.gov/about/about_nitrd.aspx

By Austin WhisnantMember of the Technical StaffThe CERT Network Situational Awareness Team Knowing what assets are on a network, particularly which assets are visible to outsiders, is an important step in achieving network situational awareness. This awareness is particularly important for large, enterprise-class networks, such as those of telephone, mobile, and internet providers. These providers find it hard to track hosts, servers, data sets, and other vulnerable assets in the network. Exposed vulnerable assets make a network a target of opportunity, or "low-hanging fruit" for attackers. According to the 2012 Data Breach Investigations Report, of the 855 incidents of corporate data theft reported in 2012, 174 million records were compromised. Of that figure, 79 percent of victims were targets of opportunity because they had an easily exploitable weakness, according to the report. This blog post highlights recent research in how a network administrator can use network flow data to create a profile of externally-facing assets on mid- to large-sized networks. Network flow data, which is an aggregation of the header information contained in datagrams (packets), can be used to create profiles of network traffic, detect malicious activity, and determine appropriate traffic prioritization settings. Network flow data includes information about communicating pairs of IP addresses, and the ports and protocols on which they communicate, as well as aggregated byte counts and flags used. Network administrators can use network profiling to consider how decisions about configuration changes will affect the rest of the assets on their network. Security administrators can evaluate the profiles to identify assets that violate policy and suspicious activity, while business administrators can use the profiles to help guide long-term decisions regarding network security. The intent of this research by the CERT Network Situational Awareness Team was to create a step-by-step guide for using network flow to inventory or profile a network that includes thorough explanations of why certain steps were chosen so that administrators could understand the process and tailor the steps for their environments. We on the research team focused our analysis on creating a profile of externally facing assets on mid- to large-sized networks that serve thousands to hundreds of thousands of users. We used data from a medium-sized enterprise network that allowed us to access its typical data usage. By focusing on network flow data, we had much less data to deal with than if we collected all traffic on a network (full packet capture). Focusing on headers also allowed us to avoid issues with confidentiality and privacy because we were not actually collecting payload information. We then parsed the data we collected using the System for Internet-Level Knowledge (SiLK).  SiLK is an open-source tool developed by the CERT Network Situational Awareness Team that is an efficient network flow collection and storage infrastructure that will accept flow data from a variety of sensors. To produce relevant results, the process we developed for network profiling must complete within a fixed amount of time. For networks with relatively stable assets, this process could take place over one or two months. For fast-changing networks, the process could take place in as little as one to two weeks. By following these steps, a network administrator profiling a network will obtain a list of public-facing assets, information about the ports through which each asset is communicating, and other pertinent information, such as the external IP addresses to which the asset is connecting. Our approach can be broken down into the following steps: Gather available network information. It is important to gather available information about a network prior to beginning the profile. This information helps to define the scope for the rest of the process. Baseline information can include address space, network maps, lists of servers and proxies, and policies governing network design. Even if this information is incomplete or out of date, it still provides a reference baseline.Not everything about the network can be known. Consider conducting a quick assessment or penetration test to develop a network map and a list of exposed services on various machines. Doing so provides a background for the profile. Later, the network maps and lists of servers can be updated for a cycling-through process. Select an initial data set. This step shapes the entire analysis. A representative sample must be large enough to represent typical traffic, but small enough to support an iterative processing of queries. The general guidelines for selecting a sample data set include •    Duration. Start with at least an hour’s worth of data. Add more data up to a day’s worth, until the query time starts to slow.•    Timing. Select the busiest time of day to carve out the most representative network traffic.•    Direction. If the traffic is bidirectional, start by looking at outbound traffic.•    Sampling. Avoid starting with sample data because it may mask important routine behaviors.•    Network size. Consider separating a large IP-bound network into a few independent profiles and merging them after analysis is complete. Identify the active address space. Issues involved in monitoring the address space include whether sensors cover private address space, what traffic is expected on failover circuits during normal operations, and whether a business unit has connected a system without administrator knowledge. The steps involved in identifying and monitoring the active address space include1. Identify hosts that have active Transmission Control Protocol (TCP) connections and those that don’t.2. Identify hosts that generate a non-trivial amount of traffic on protocols other than TCP.3. Aggregate individual hosts into populated network blocks. 4. Examine additional information gathered in step one to confirm the list of active IP address blocks. Catalog common services. After the active hosts on the network have been identified, inventory the services that comprise the majority of bandwidth use and business operations, such as web traffic and email. Once these protocols are inventoried, start working on other services likely to run on the network and visible to instrumentation, including Virtual Private Network (VPN), Domain Name System (DNS), and File Transfer Protocol (FTP). Catalog remaining active assets. The list of assets in the profile thus far should cover almost all network traffic, as well as services that were profiled based on the most frequent services in the network. Prior to profiling any remaining hosts, expand the time frame for the sample data set to see if there are other active hosts that were not represented in the smaller data set. The expanded data set should include at least one month’s worth of data for the most accurate results. While profiling leftovers, note the findings in the profile, even if they are hard to verify or if they seem incorrect. This information can always be reviewed in more detail during further investigation. Maintain the profile. Six months after you create a profile, it may no longer be accurate. A majority of these steps can be automated to address this problem. For example, network flow analysis software may allow for scheduling filters to run weekly or monthly during this process. Automated tools can only do so much, so you need to consistently validate potential assets for a particular service before adding them to the profile. To update the profile at least once a month, either run through the profiling process again or examine trends over time. Report on findings. To increase the impact of the profile, consider adding more data points that pertain to security, including•    machine administrator•    update schedule•    intended purpose Challenges in this Approach Relying only on network flow data to create a network profile is inherently more inaccurate than using full packet capture. As long as administrators are aware of the limitations of using flow data (as explained in the report documenting the guide), useful results can be produced by following the steps in the guide. Future Research Our team is currently examining how administrators customize and use the results of network profiles. We are investigating how to automate these steps and implement a continual update process. We are also considering development of a new step-by-step guide using this same approach with a different tool. For example, ARGUS is another network flow analysis tool with a slightly different approach than SiLK. Please feel free to suggest other tools for investigation in the comments section below. Additional Resources The technical report describing this research, Network Profiling Using Flow, may be downloaded at http://www.sei.cmu.edu/library/abstracts/reports/12tr006.cfm Our approach is based on System for Internet-Level Knowledge (SiLK), an open-source tool developed by the CERT Network Situational Awareness Team. You can download SiLK at http://tools.netsa.cert.org/silk/. The Network Situational Awareness (NetSA) group in the CERT Program at the SEI developed and maintains a suite of open-source tools for monitoring large-scale networks using flow data. These tools have grown out of our work on the AirCERT project, the SiLK project, and the effort to integrate this work into a unified, standards-compliant flow collection and analysis platform. You can view that suite of tools at http://tools.netsa.cert.org/index.html

By Bill ScherlisSEI Principal Researcher and Director, Institute for Software Research Some of the principal challenges faced by developers, managers, and researchers in software engineering and cybersecurity involve measurement and evaluation.  In two previous blog posts, I summarized some features of the overall SEI Technology Strategy. This post focuses on how the SEI measures and evaluates its research program to help ensure these activities address the most significant and pervasive problems for the Department of Defense (DoD). Our goal is to conduct projects that are technically challenging and whose solution will make a significant difference in the development and operation of software-reliant systems. In this post we’ll describe the process used to measure and evaluate the progress of initiated projects at the SEI to help maximum their potential for success. The Importance of Measurement and Evaluation in Software and Cybersecurity Certain characteristics of software and cybersecurity are easy to measure—lines of code produced, errors found, errors corrected, port scan events, and time consumed. But much of what really matters in practice offers an even greater challenge to our measurement and evaluation capabilities. For example How far are we from completion of the project? Are the specifications consistent and feasibly implementable? How much engineering risk is associated with this requirements specification? How resilient is this architectural design with respect to flooding attacks? What kinds of robustness are offered by this application programming interface (API )definition? How many errors reside in the code base, and how many of them are vulnerabilities? How many of these vulnerabilities are easy coding errors, and how many are intrinsic to the architectural design? Is the component or system implementation safe for deployment in a secure environment? Will the control system meet deadlines? Over the years, much of the research activity at the SEI has been focused on enhancing our ability to measure and evaluate. Indeed, the genesis of the Capability Maturity Model (CMM) and its successors in the late 1980’s is based on a need—unmet at the time—to evaluate the capabilities of software development organizations to deliver on their promises. Models such as the Resilience Maturity Model (RMM) and architecture evaluation methodologies serve similar purposes. Much of the body of research work at the SEI is focused on these challenges. And so there is a natural question: How can we evaluation the potential significance of the research, development, testing, and evaluation (RDT&E) work we (and others) undertake to improve technology and practice in software engineering and cybersecurity?  This is a significant issue in all research, development, test, & evaluation (RDT&E) programs. We need to take risks, but we want these to be appropriate risks that will yield major rewards, and with a maximum likelihood. An overly conservative approach to RDT&E management can yield small increments of improvement to existing practices, when what is really needed, in many cases, are game-changing advances. Indeed, in software and cybersecurity, despite the admonition of Lord Kelvin ("If you cannot measure it, you cannot improve it"), purely "numbers-based" approaches can be dangerous (viz. the famous "light under the lamppost" story). Naïve application of management-by-the-numbers is especially damaging in our disciplines, where so many important measures are lacking. On the other hand, bold "open loop" approaches can yield risks with little likelihood of benefits. We must therefore employ a combination of measures, expert judgment, and careful identification of evaluation criteria—and we must structure the process of research to obtain early indicators and develop them when necessary (i.e., build new lampposts). This multidimensional evaluation process is essential because, in research, it is not just a matter of risk versus reward—thoughtful management in the definitional phases of a project can enable us to "push the curve out," simultaneously mitigating risk, identifying or developing meaningful measures in support of evaluation, and increasing potential for success and impact. A more general discussion on research management practice is the subject of chapter 4 of the 2002 National Academy report on Information Technology Research, Innovation, and E-Government. Heilmeier’s Catechism In this vein, more than two decades ago, George Heilmeier, former director of the Defense Advanced Research Projects Agency (DARPA), developed a set of questions that he posed to prospective leaders of research projects. If these questions can be well addressed, then the proposed projects were more likely to be both successful and significant. These questions have been so widely adopted in research management that they have become known as the "Heilmeier Catechism." Experience at DARPA and other innovative research organizations has shown that compelling answers to the following questions are likely to predict successful outcomes: What are you trying to do? Explain objectives using no jargon. How is it done today? What are the limits of current practice? What's new in your approach? Why do you think it will be successful? If you're successful, what difference will it make? To whom? What are the risks and the payoffs? How much will it cost? How long will it take? What are the midterm and final "exams" to assess progress? (A more detailed presentation of these questions appears in the National Academy Critical Code report, pages 114 - 115.) Our work at the SEI spans a spectrum ranging from deep, technically-focused research on the hard problems of software engineering and cybersecurity—the technology, tools, practices, and data analyses—to more operationally-focused efforts supporting the application of diverse technical concepts, methods, and tools to the challenges of practice, including development and operations for complex software-reliant systems. To ensure we help the DoD and other government and industry sponsors identify and solve key technical and development challenges facing their current and future software-reliant systems, we have adapted Heilmeier’s Catechism into an evaluation process that is more adapted to the particular measurement challenges in software and cybersecurity.  This evaluation process—which we summarize below—involves a combination of four factors: Mission relevance and potential significance of a proposed scope of work to the mission effectiveness of DoD, its supply chain, and other critical sectors Field significance, or capability to evaluate mission effectiveness of emerging solutions to DoD and other stakeholders, including the development of early indicators of mission significance Technical soundness of the research undertaken, based on accepted scientific approaches Technical significance and innovation of the work, for example according to the standards of quality and significance evident in relevant top publication venues Ensuring Validity The wide spectrum of activities in the SEI body of work motivates us to contemplate four different dimensions of validity in assessing proposals for research projects from SEI technical staff: Mission relevance. Are we working on the right problem? Means to enhance validity related to mission relevance include challenge problem workshops, along with early and ongoing collaboration with knowledgeable, mission-savvy stakeholders. This is analogous to the agile practice of involving the customer early and often; it dramatically lessens risk that we are solving the wrong problem. The SEI also benefits from having technical staff members with extensive operational experience in the development, sustainment, and modernization of systems, as well as in a wide range of cybersecurity mission activities. This experience, along with direct engagement with operational stakeholders, enhances validity related to mission relevance. Field significance. Will our solution have the intended impact when it is matured and fielded? Early indicators of field significance include the use of models and simulation, surrogate trials, and field exercises, as well as identification of metrics and validated evaluation criteria and perhaps, most importantly, collaboration with partners who can assist in appraisals, including test and evaluation organizations. Development of such early indicators can be very challenging and can require explicit investment in trials, exercises, and, development of surrogates. Technical soundness. Does our approach embody a sound scientific process that will lead to scientifically valid results? This dimension of validity forces attention to experiment design, mathematical soundness, measures and metrics, statistical significance, and other technical factors that influence the scientific acceptability of outcomes. Technical soundness of published results is generally validated through peer review. The best early predictors of sound outcomes relate to the caliber and experience of the technical staff and their research collaborators. Technical significance. Is our technical approach well situated in the literature or practice of the discipline, and will results be recognized as significant and scientifically impactful? The best evidence of technical significance derives from direct evaluation by recognized peer expert. For publishable results, this can come in the form of peer review and publication in the top venues, best-paper awards, keynote invitations, recognition from professional technical societies, and other types of peer recognition. Finally, we consider the use and development of metrics, which contribute generally both to our ability to evaluate research and to the advancement of practice generally in software and cybersecurity. Much of the research we do is focused on developing and validating new metrics. But, additionally, when we develop metrics useful in research, we find they can also have broad value in practice. In the management of research projects, there are four dimensions we consider: Success criteria.  How can we test for overall soundness, significance, and impact when a project reaches completion?  What are the observable features of final project results, and who must be involved in making an assessment?  Which of the observables are indisputably objective, and which require expert judgment? It is important to note that there is nothing wrong with the use of expert judgment to make evaluations of success and impact, especially when the alternative is a set of inaccurate and incomplete objective measures. Performance criteria.  What are early indicators and tracking mechanisms to monitor progress (the so-called "mid-term exams")? What are the various dimensions of performance, and how are they assessed? As with the success criteria, these can involve a combination of objective measures and expert judgment.   User inputs and feedback. Who are the early collaborative stakeholders, and how are they selected? What are the mechanisms to get feedback from them? This is an explicit criterion, because of the importance of expert judgment and stakeholder acceptance of much of the new work in software and cybersecurity. Metric development. Where are there dark shadows where the light of measurement is needed?  How will the project advance measurement capability as part of its work? At the SEI, this is a first-class activity for two reasons. First, the development of new metrics can have broad and deep impact on practice. For example, metric development (or, more generally, development of evaluation capability) is an explicit feature of some of the challenges mentioned in previous blog posts related to software assurance, cost-re-estimation, and architectural resilience. Advances in Metrics As noted at the outset, measurement is, in some ways, the deepest challenge in both software engineering and cybersecurity. For example, how can we measure the security of a system or the flexibility of a software framework? Likewise, how can we develop useful subsidiary measures that can help us understand the flow of data and information within a large system, or the extent of coupling among a set of federated software components?  This challenge is not only technically deep but profoundly significant to the advancement of practice. Particularly when earned value must be evaluated, the Architecture-Led Incremental Iterative Development (ALIID) approach depends on advances in metrics. Described in the first post in this series, ALLID (or, "agile at scale") enables iterative and incremental development of highly capable and innovative systems with acceptable levels of programmatic risk. Earned value based purely on accumulation of lines of code is unlikely to lead to project success, when much of the risk mitigation (accomplished through prototyping and modeling-and-simulation, for example) leads to reduced variances in estimates, though not necessarily reduced mean values. That is, the estimates become more predictive and reliable as a result of these kinds of activities, and thus greater levels of innovation and engineering-risk taking can be supported without a necessary threat to the potential for success of the overall effort. This is the essence of iteration and incrementality. Effective metrics can have a profound impact on the industry—witness the CMM and subsequent CMMI families of measures, just successfully transitioned out of the SEI into a separate activity at Carnegie Mellon. This is why we take as a principle that the development of new measures must be a first-class activity in the research portfolio. Unlike the work done decades ago, our disciplines are now data rich, and we can build on data collection, models and simulation, field tests, and other analytic approaches that, just a few years ago, were relatively less feasible. We see this reflected in the research portfolio at the SEI, which is moving more aggressively than ever to address these challenges and create rich possibilities for the development and operation of software-reliant systems for the DoD, other mission agencies, and their supply chains. In this series on the SEI Technical Strategic Plan, we have outlined how we are meeting the challenges of designing, producing, assuring, and evolving software-reliant systems in an affordable and dependable manner. The first post in this series outlined our strategy for ALLID or agile-at-scale, combining work on process, measurement, and architecture. The second post in the series detailed our strategies for to achieve game-changing approaches to designed-in security and quality (evidence-based software assurance). It also identified a set of DoD-critical component capabilities relating to cyber-physical systems (CPS), autonomous systems, and big data analytics. Finally, it outlined our approach to cybersecurity tradecraft and analytics. I am also very excited to take this opportunity to introduce you to Kevin Fall, who joined the SEI late last month as its new chief technology officer. In this role, Dr. Fall will take over responsibility from me for the evolution and execution of the SEI Technical Strategic Plan, as well as direction of the research and development portfolio of the SEI’s technical programs in cybersecurity, software architecture, process improvement, measurement and estimating, and unique technical support to sponsors. I will continue to take an active role in supporting the SEI and its essential mission—even including occasional future blog posts! Kevin will join the blogger team with a blog post in the near future summarizing his vision for research and transition activities at the SEI. Additional Resources We expect that the SEI Strategic Research Plan will be soon be available for download.

By Grace Lewis Technical Lead, Edge-Enabled Tactical Systems Research In 2011, Col. Timothy Hill, director of the Futures Directorate within the Army Intelligence and Security Command, urged industry to take a more open-standards approach to cloud computing. "Interoperability between clouds, as well as the portability of files from one cloud to another, has been a sticking point in general adoption of cloud computing," Hill said during a panel at the AFCEA International 2011 Joint Warfighting Conference. Hill’s view has been echoed by many in the cloud computing community, who believe that the absence of interoperability has become a barrier to adoption.  This posting reports on recent research exploring the role of standards in cloud computing and offers recommendations for future standardization efforts. Avoiding Vendor Lock-In Since the inception of the cloud, organizations have been transferring data and workloads to pooled, configurable computing resources. These resources include networks, servers, storage, applications, and services. One concern voiced by many organizations that use cloud-based services is vendor lock-in, which stems from the inability to move resources from one cloud provider to another. Users want to have the freedom to move between cloud providers for many reasons. For example, a relationship with a vendor may not be working, service-level agreements may not be met, other providers offer better prices, or a provider goes out of business. In an environment without common standards, there is little or no freedom to move between vendors. The cloud computing community has already developed numerous standards (some argue there are too many) by various forums, standards organizations, and nonprofit organizations including OpenStack, the Standards Acceleration to Jumpstart Adoption of Cloud Computing, and The Open Group Cloud Work Group, to name a few. One issue explored in my research is whether we should create new standards or just leverage existing standards. Some standardization efforts focus on codifying parts of a cloud-computing solution, such as workloads, authentication, and data access. Other standards focus on unifying disparate efforts to work together on a solution. In addition, due to their large market share in this space, interfaces used by Amazon have emerged as de facto standards. The technical report describing my research, The Role of Standards in Cloud Computing Interoperability, explains how answers to questions about how standards can enable interoperability depend on several factors.  Key factors include the type of service model that a cloud provider uses and the level of interoperability that an organization expects.  Note that the cloud community typically uses the term interoperability to refer to portability, i.e., the ability to move a system from one platform to another. Use Cases Initially, my research identified four typical cloud-computing interoperability use cases that are supported by standards: Workload migration. A workload that executes in one cloud provider can be uploaded to another cloud provider.  Some standardization efforts that support this use case are Amazon Machine Image (AMI), Open Virtualization Framework (OVF), and Virtual Hard Disk (VHD). Data migration.  Data that resides in one cloud provider can be moved to another cloud provider. A standardization effort that supports this use case is Cloud Data Management Interface (CDMI). In addition, even though SOAP and REST are not data-specific standards, multiple cloud-storage providers support data- and storage-management interfaces that use SOAP and REST. User authentication.  A user who has established an identity with a cloud provider can use the same identity with another cloud provider.  Standardization efforts that support this use case are Amazon Web Services Identity Access Management (AWS IAM), OAuth, OpenID, and WS-Security. Workload management. Custom tools developed for cloud workload management can be used to manage multiple cloud resources from different vendors.  Even though most environments provide a form of management console or command-line tools, they also provide APIs based on REST or SOAP. We found that workload migration and data migration can benefit the most from standardization.  For example, standardization of VM (virtual machine) image file formats would allow organizations to move workloads from one provider to another or from private to public clouds. Standardized APIs for cloud storage would do the same for data. Provider Service Models In examining the issue of standardization through the provider lens, we looked at the three main service models: Infrastructure-as-a-service (IaaS). IaaS stands to benefit the most from standardization because the main building blocks are workloads that are represented as VM images and storage units, whether type data or raw data. This finding also ties back to the first two use cases identified earlier, which were workload migration and data migration. Platform-as-a-service (PaaS). Organizations that buy into PaaS, do so for the perceived advantages of the development platform. The platform provides many capabilities out of the box such as managed application environments, user authentication, data storage, reliable messaging, and other functionality in the form of libraries that can be integrated into applications. Organizations that adopt PaaS are not thinking only of extending their IT resources, but are seeking value-added features (such as libraries and platforms) that can help them develop and deploy applications more quickly. Software-as-a-service (SaaS). SaaS stands to benefit the least from standardization. SaaS is different from IaaS and PaaS in that it represents a licensing agreement to third-party software instead of a different deployment model for existing resources that range from data storage to applications. Organizations that adopt SaaS are acquiring complete software solutions or services that can be integrated into applications. Organizations select PaaS and SaaS specifically for these value-added features, and end up in a commitment similar what one experiences when purchasing software. Expecting PaaS and SaaS providers to standardize these features would be equivalent to asking an enterprise resource-planning software vendor to standardize all of its features; it’s not going to happen because it’s not in their best interests. Future Research One challenge among standardization organizations is determining what areas of cloud computing to standardize first. In 2005, researchers from the European Union defined three generations of service-oriented systems. The development of cloud-based systems over time is analogous to the following classification of the way that service-oriented systems have evolved First-generation. The location and negotiation of cloud resources occur at design time. Cloud resources are provisioned and instantiated following the negotiation process. Second-generation. The location and negotiation of cloud resources occur at design time. Depending on business needs, however, cloud resources are provisioned either at design time or runtime, and instantiated at runtime. This approach would support, for example, a cloud-bursting strategy in which developers design a system for an average load, but the system can balance its load to a cloud provider when it reaches its full capacity. Third generation. In the third generation of cloud-based systems, the location, negotiation, provisioning, and instantiation of cloud resources occur at runtime. Reaching this third generation of cloud-based systems will most likely be the focus of future research. This work will require cloud consumers, cloud providers, and software vendor groups to work together to define standardized, self-descriptive, machine-readable representations of basic resource characteristics such as size, platform, and API (application programming interface); and  more advanced resource characteristics such as pricing and quality attributes values, negotiation protocols and processes; and billing protocols and processes. For now, standardization efforts should focus on the basic use cases of user authentication, workload migration, data migration, and workload management. Those efforts can then be used as a starting point for the more dynamic use cases of the future. Recommendations Even if vendor lock-in is mitigated, it is important for organizations to know that any migration effort comes at a cost—whether between cloud providers or local servers, databases, or applications.  Cloud standardization efforts should therefore focus on finding common representations of user identity, workload (virtual-machine images), cloud-storage APIs, and cloud management APIs. Vendors influence many standards committees, and it is unrealistic to assume that each of these elements will have a single standard. Agreement on a small number of standards, however, can reduce migration efforts by enabling the creation of transformers, importers, exporters, or abstract APIs. Such an effort could enable the dynamic third-generation of cloud-based systems.  What are your thoughts on cloud computing interoperability? Do we need new standards? Or can we live with the ones we have? Will we ever get to the third generation? Is it necessary? Additional Resources The technical report describing this research, The Role of Standards in Cloud Computing Interoperability, may be downloaded at http://www.sei.cmu.edu/library/abstracts/reports/12tn012.cfm To read all of Grace Lewis’ posts on her research in cloud computing, please visithttp://blog.sei.cmu.edu/archives.cfm/author/grace-lewis

By Julien Delange Senior Member of the Technical Staff Research Technology & System Solutions When a system fails, engineers too often focus on the physical components, but pay scant attention to the software. In software-reliant systems ignoring or deemphasizing the importance of software failures can be a recipe for disaster.  This blog post is the first in a series on recent developments with the Architecture Analysis Design Language (AADL) standard. Future posts will explore recent tools and projects associated with AADL, which provides formal modeling concepts for the description and analysis of application systems architecture in terms of distinct components and their interactions. As this series will demonstrate, the use of AADL helps alleviate mismatched assumptions between the hardware, software, and their interactions that can lead to system failures. There are many well-documented examples of problems in software-reliant systems: In December, Ford Motor Company announced that it would be making software updates to the cooling systems of its hybrid 2013 Ford Escape and Ford Fusion because problems with the original cooling system design caused the vehicle to catch fire while the engine was running. A recent study of a popular automatic external defibrillator (AED) found security flaws in the embedded software and the commercial off-the-shelf (COTS) update mechanism. A cluster of ships attached to the Ariane 5 rocket were lost when the rocket failed to achieve orbit on its maiden voyage. The failure was attributed to an "error in software design" and resulted in a loss of more than $370 million. These types of incidents demonstrate the need to make software more safe and reliable, which motivates our focus on AADL in this series of blog posts. Addressing Software (as well as Hardware) Mismatched assumptions between hardware, software, and their interactions often result in system problems that are caught too late, which is an expensive and potentially dangerous situation to developers and users of mission- and safety-critical technologies.  To address this problem, the Society of Automotive Engineers (SAE) released the aerospace standard AS5506, named the Architecture Analysis & DesignLanguage (AADL). The AADL standard, which was authored by my colleague, Peter Feiler, defines a modeling notation based on a textual and graphic representation that is used by development organizations to conduct lightweight, rigorous—yet comparatively inexpensive—analyses of critical real-time factors, such as performance, dependability, security, and data integrity. AADL models capture both software and hardware components, as well as their interactions, including the association of a software process on a processor or the deployment of connection on a bus. The AADL standard includes abstractions of software, computational hardware, and system components for specifying real-time, embedded and high dependability systems with their software/software concerns and their specific requirements (such as scheduling, bus latency or jitter) systems validating system and ensuring that stakeholders requirements can be achieved While initial AADL efforts were experimental, organizations have been using the AADL standard for nearly a decade. An early adopter was the European Space Agency, which leads the ASSERT project for the design and implementation of safety-critical systems. This project relied on AADL since its inception and project members continue to use it to model, validate, and produce software. From this early use of the system other projects and communities have shown a strong interest in adopting and applying the language. The AADL committee and its members tracked users experiences and, in response, published a new version of the AADL standard in 2009, with a minor revision in September 2012. A Global Proof of Concept In 2008, the Aerospace Vehicle Systems Institute (AVSI), a global cooperative of aerospace companies, government organizations, and academic institutions, launched an international, industry-wide program called System Architecture Virtual Integration (SAVI) to reduce cost/cycle time and rework risk by using early (and frequent) virtual integrations. In addition to the SEI, major players of the SAVI project include Boeing, Airbus, Lockheed Martin, BAE Systems, Rockwell Collins, GE Aviation, federal aviation administration, the U. S. Department of Defense (DoD), Honeywell, Goodrich, United Technologies, and NASA. SAVI was created in response to the realization by aircraft manufacturers that the source lines of code (SLOC) in software-reliant systems was increasing exponentially. Over the past 20 years  the size of aircraft software, measured in SLOC, has doubled every four years. For the next decade, the projected 27.5 million lines of code required are estimated to cost in excess of $10 billion. In addition to the increased software cost, the latest generation of aircraft, such as the Boeing 787s, will be highly interconnected and are projected to produce half of terabyte of data per flight. Coupled with that increasing complexity is a growing reliance on embedded software to meet requirements for key features, such as greater range and comfort on fewer, more economical flights. The intent of SAVI was to pilot a new technology and a new acquisition process based on architectural models rather than paper documentation with multiple dimensions of analysis used throughout the lifecycle. A key tenet of virtual integration is the use of an annotated architecture model as the single source for analysis. For the SAVI proof of concept, an aircraft system architecture was modeled using the AADL standard. This process allowed AVSI to capture integration faults earlier in the development process, and meet increasing demands for safety and economy. As indicated in the graphic below, the National Institute of Standards & Technology estimated in the report The Economic Impacts of Inadequate Infrastructure for Software Testing that more than 70 percent of all system defects are introduced prior to code development, while only a small fraction are detected during that time.  The remaining errors are detected in the testing phase when it’s most expensive to fix. The ability to detect errors and defects prior to implementation efforts therefore reduced development costs and efforts, while enhancing overall system robustness and reliability. The SEI technical report, System Architecture Virtual Integration: An Industrial Case Study, details the SAVI proof of concept and results, which include multi-tier modeling and analysis of an aircraft and its subsystems support for the needs of both system engineers and embedded software system engineers propagation of changes to the model across multiple analysis dimensions maintenance of multiple model representations in a model repository auto-generation of analytical models interfacing of multiple tools distributed team development via a model repository New Directions for AADL A number of organizations actively use AADL in their modeling and analysis efforts. While AADL was originally developed for aerospace and avionics, it is now being used in many other fields, including automotive and medicine. In particular, as more medical devices are adopted and connected together, there is a growing interest in analyzing and verifying them, especially because issues may have a catastrophic impact on patients. The Food & Drug Administration (FDA), in partnership with the Penn Research in Embedded Computing and Integrated Systems Engineering program, plans to use AADL to support the architecture of modeling concerns for the Generic Infusion Pump. One example of a generic infusion pump would be a model for a Patient-Controlled Analgesia Pump. AADL is used to specify system devices, the underlying software artifacts (in terms of tasks, subprograms), their interaction in terms of events or data and the inherent deployment over the execution platform (association with processors, buses, etc.). The language was also used to capture and analyze system behavior (dependencies among events, conditional execution of system functions according to received data, etc.) We are also working to connect the AADL technology to other technologies being developed at the SEI. For example, SEI researchers are creating cost assessments of different iterations of system architectures. This work will explore questions including When you have a defined and established architecture, what is the cost for a new iteration? What is the cost and impact involved in changing an architecture? By describing the runtime aspects of the system’s deployment, can AADL help evaluate the potential impact of a modification of the execution runtime (for example, when upgrading a device)? Future posts in this series on AADL will highlight real-world applications of AADL, which are presented by users during the AADL standard meetings. At every meeting we include user days where users reflect on their experiences with AADL and tool sets that they have developed.  Please let us know your experiences applying ADDL in the comment section below. Additional Resources In September 2012, Peter Feiler co-authored with David P. Gluch the book, Model-Based Engineering with AADL: An Introduction to the SAE Architecture Analysis & Design Language. For more information, please visithttp://www.sei.cmu.edu/library/abstracts/books/978032188894-5.cfm With our user community, the SEI maintains a wiki with all of our research on AADL that is accessible to the public. To access the wiki, please visithttp://www.aadl.info/aadl/currentsite/ To access a site that maintains source code for AADL, where people can come and freely contribute to the OSATE tool set that is supporting AADL, please visithttps://github.com/osate To read the SEI technical report, System Architectural Virtual Integration: An Industrial Case Study, please visithttp://www.sei.cmu.edu/library/abstracts/reports/09tr017.cfm Julien Delange and Peter Feiler will present Design and Analysis of Cyber-Physical Systems: AADL and Avionics Systems, at SATURN 2013. For more information, please visit http://www.sei.cmu.edu/saturn/2013/program/abstracts.cfm

Second in a Series on Readiness Fit Analysis for Adoption of Agile Methods By Suzanne Miller Senior Member of the Technical StaffAcquisition Support Program The adoption of new practices, such as agile or any new practice for that matter, is a task that is best undertaken with both eyes open. There are often disconnects between the adopting organization’s current practice and culture and the new practices being adopted. This posting is the second installment in a series on Readiness & Fit Analysis (RFA), which is a model and method for understanding risks when contemplating or embarking on the adoption of new practices, in this case agile methods. The RFA method helps organizations understand the barriers and enablers to successful adoption that are present when an analysis is performed. The first post in this series outlined the principles of RFA and described the Acquisition Support Program’s work in extending RFA to support profiling and adoption risk identification to organizations that are adopting agile methods. This blog post continues the discussion with a more in-depth dive into one more of the six RFA categories that we have identified. In our work with the Department of Defense (DoD), we often encounter organizations that have been asked by their government program office to adopt agile methods. These are organizations that have traditionally performed an engineering "V" lifecycle and are accustomed to being managed via a series of document-centric technical reviews, which focus on the evolution of the documents that describe the requirements and design of the system rather than focusing on its evolving implementation, as is more common with agile methods. After the program office and contractor are trained, they realize that there is more to an agile approach than frequent, small iterations and daily standup meetings.  Hence, they struggle to adopt agile practices. For example, contractor personnel often aren’t accustomed to working shoulder-to-shoulder with government counterparts, and the transparency of daily task management is uncomfortable. After a short period of time, programs often revert back to a more traditional approach, even though end users gave positive feedback about what was produced during the timeframe that agile methods were used. This common scenario highlights a disconnect in business strategy, values, and management style that often occurs in organizations trying to adopt new practices. These aren’t the only disconnects, but they are representative of what we often see when working with DoD or other regulated organizations trying to adopt agile methods. Adapting the Readiness Fit Analysis (RFA) to Agile The method for using RFA and the profile that supports CMMI for Development adoption is found in Chapter 12 of CMMI Survival Guide: Just Enough Process Improvement. As part of the SEI’s research in the adoption of agile methods in U.S. DoD settings, we have adapted the RFA profiling technique to accommodate typical factors used in RFA and other factors more uniquely associated with the DoD acquisition environment. To date, we have characterized six categories to profile for readiness and fit: business and acquisition (details discussed in the first post in this series) organizational climate system attributes project and customer environment technology environment practices Determining Readiness & Fit Each category of readiness and fit has a set of attributes that can be characterized by a statement representing your expectation if you were observing a successful agile project or organization operating in relation to that attribute. As a reminder, an attribute of business and acquisition is stated as follows:Oversight mechanisms are aligned with agile principles.Oversight is an aspect of acquisition that can either support or disable an agile project. So alignment of the oversight with agile principles provides less risk of oversight being counterproductive.From the title of this method - Readiness and Fit - it would be easy to assume that the only time you could productively use this method would be in the early stages of adoption, when you’re trying to decide if you’re ready to adopt and if the practices you’re considering are a fit for your organization. Actually, we have used models like this at multiple points in the adoption of a new technology or practice. Your certainty regarding the state of the organization in relation to factors represented in the RFA changes from early use of the method (before initial pilots) to later use (after you’ve run two or three releases using an agile method). Your certainty also changes with respect to the importance of a specific factor to your organization’s success. At the beginning of an agile adoption project, we are often uncertain about the state of the organization or the importance of individual factors (such as alignment of oversight practices with our agile practices) to our adoption success. Later in the adoption process, performing an RFA highlights adoption risk areas overlooked during early phases of adoption and areas where we now have more data to help guide us in developing adoption risk mitigation strategies. For example, we may not initially understand that our approach to estimation in a larger organization doesn’t easily accommodate certain agile practices, such as relative estimation. After we have been through one or two pilots, we are likely to understand the effect relative estimation has on our results, and we can develop strategies to help connect the agile estimation practices to those of the larger program. So don’t hesitate to apply RFA principles and techniques at multiple points in your adoption journey. Organizational Climate This blog entry focuses on a portion of a single RFA category: Organizational Climate. This category is one of the longer categories in the set, because misfits in organizational climate— including culture, values, and working principles—are particularly troublesome areas for many organizations pursuing agile adoption. This category covers the internal operations and culture of an organization, which are extremely important in determining fit. The alignment of the organization’s inherent operations to agile concepts and principles will help determine the ease or difficulty of the transition. The organization’s culture is one of most difficult issues to assess for agile adoption readiness. Culture encompasses assumptions about appropriate or inappropriate behavior and values ingrained in the members of an organization. For instance, in DoD organizations the culture is typically plan-driven and hierarchical, with strict command-and-control structures for communication and leadership. An organization adopting agile methods is generally more empirical, collaborative, self-organizing, and cross-functional. The following list specifies a subset of the factors within the Organizational Climate category that must be considered when an organization performs an RFA. This category includes leadership, sponsorship, incentives, values, structures, and organizational-change history. We’ll deal first with leadership, sponsorship, and incentives, which can either be aligned to an agile perspective, or misaligned, which is often a source of behaviors that may appear as resistance to the new practices. Understanding misaligned factors is important in addressing their symptoms proactively. The list includes a tag (a short title that summarizes the statement) and a statement that provides a condition or behavior that would be expected in an organization successfully using engineering and management methods consistent with agile principles as published in the Agile Manifesto. When applying an RFA, you look at the statement, and then consider the "fit" of that statement to your own organization’s behaviors or attitudes. Cascading sponsorship. Sponsor support for the use of agile or lean methods use is explicit and cascading. In particular, the sponsorship doesn’t just emanate from the program manager, it cascades throughout the acquisition chain. In most organizations, a move to agile methods involves new behaviors and different values. This paradigm is a major change in how an organization operates, and it will affect the overall climate. For DoD organizations, the entire acquisition eco-system includes not just the program but outside organizations, such as Certification and Accreditation and Operational Test & Evaluation. Due to policies and regulations, it can be hard to include these parts of the acquisition chain when adopting agile or lean methods. Cascading sponsorship helps alleviate these problems by having sponsors in multiple places within the organization who can model the new values and behaviors, instilling confidence in the people who are actively trying to adopt the new practices. Aligned incentives. Incentives among stakeholders are aligned to reflect agile principles. The fifth agile principle related to the Agile Manifesto states, "Build projects around motivated individuals. Give them the environment and support they need, and trust them to get the job done." Part of this environment and support is the incentive to work in an agile environment. Most traditional organizations have incentives aligned to individuals but not many incentives for teams. For agile or lean environments, the incentives must be aligned to enable the team to succeed, not to make an individual hero. External policy support. Adherence to agile or lean principles is supported by external policies. For example, the National Defense Authorization Act (NDAA) Section 804 promotes an iterative, incremental style of acquiring information systems that includes software-intensive systems. This legislation aids in the adoption of agile/lean methods and provides guidance in policy and regulation. In addition, Section 933 provides a strategy for acquisition and oversight of DoD cyber warfare capabilities, which also points back to section 804 of the 2010 NDAA. These are high-level examples of policy support of agile principles. Within a particular organization, however, there may be an opportunity for guidance that can push individuals and groups into adopting agile methods. For example, the CIO of a particular agency may mandate adoption of agile principles for their organizations. Caution: Using a policy or mandate to FORCE adherence to agile principles is not productive for healthy adoption of new practices. Putting policies in place too early, before the appropriate transition mechanisms are in place, often leads to malicious compliance. Malicious compliance occurs when individuals adhere to the letter of the law so rigidly that the practices can be adopted in an unproductive way. Sponsors understand agile. Sponsors understand and support the difference in roles and business rhythm when using agile approaches. The roles and responsibilities in a traditional acquisition are well documented in DoD policies, regulations, and training documents; in an agile environment they are different and not as easily understood. Sponsors must understand the four tenets of the Agile Manifesto and the 12 underlying principles to enable the necessary business rhythm for an agile development effort. They also must understand the chosen agile practices well enough to understand the role and responsibility implications of the particular practices that have been selected. Reward system supports agile. The organizational reward system supports the team-based reward focus of agile methods. The reward system is closely related to the incentives used on a program. As stated above, agile and lean organizations focus their rewards on team behavior, whereas traditional organizations reward individual behavior. Within the DoD environment, the reward system (via the performance-management system) is structured and not easily changed. This structure may actually interfere with the support and reinforcement of a more agile environment. There are ancillary reward system mechanisms, however, such as public praise, high evaluations, access to training, and certificate programs that may supplement individual-focused performance rewards. One of the key aspects of a successful reward system is understanding the kinds of rewards that individuals actually value (many teams would value an extra day off more than a gift certificate, for example). Senior support for agile. Senior stakeholders openly and explicitly support the use of agile or lean methods in the program. Successful agile implementations have consistently had a champion. The champion may or may not be a senior stakeholder, but it is someone who has the respect of adopters and the support of senior leadership in the organization. This status will help protect fledgling agile projects from being derailed by those who do not understand the new methods or are uncomfortable with change. Open and explicit support by the senior stakeholders also means that old behaviors are no longer rewarded. This factor is often one of the hardest for senior stakeholders to consistently practice when sponsoring change.  Looking Ahead As you can see, the list of factors in the Organizational Climate category is lengthy, but these factors often need the most attention in promoting a successful adoption of agile methods and principles. As stated at the beginning, however, this is just one category out of six, and only the first five factors within that category. Each category offers insight into the adoption risks that a particular organization will face when adoption agile methods. Identifying these risks is an important first step toward planning and executing mitigation strategies to address them. This series of posts describes how RFA has been used for multiple technologies and sets of practices (most notably CMMI) to help organizations in the DoD and other regulated environments mitigate agile adoption risks. In the first post in this series, we addressed Business & Acquisition, and in this post we introduced Organizational Climate. The next post in will address the remaining factors in Organizational Climate. Subsequent posts will address one or more of the remaining RFA categories: System Attributes, Technology Environment, Project and Customer Environment, and Practices. Additional Resources A 2011 SEI technical note, Agile Methods: Selected DoD Management and Acquisition Concerns, outlines many of the organization climate issues that arise in agile adoption in the DoD. To download the report, please visithttp://www.sei.cmu.edu/library/abstracts/reports/11tn002.cfm  

By David SvobodaCERT Secure Coding Team This blog post describes a research initiative aimed at eliminating vulnerabilities resulting from memory management problems in C and C++.  Memory problems in C and C++ can lead to serious software vulnerabilities including difficulty fixing bugs, performance impediments, program crashes (including null pointer deference and out-of-memory errors), and remote code execution. The exploitability of memory-management problems in C and C++ (including double-free vulnerabilities) was first demonstrated in 2000 by the security specialist Solar Designer.  Even before Solar Designer wrote a blog post demonstrating this exploit technique, the problems of memory management had been widely acknowledged throughout the C programming community since the initial standardization of C in 1989. This elimination can be achieved by designing and implementing an ownership model for dynamically allocated memory.  The ProblemWhen a program begins execution, it is provided with a region of memory called the runtime stack. This stack grows and shrinks during the program’s lifetime; it contains memory that the compiler determines will be required by the program. But some programs require memory that the compiler cannot measure, usually because the amount of memory required will be known only to the program when it runs. This type of memory is typically managed as a heap, which stores memory dynamically allocated and deallocated by the program.When software developers need to write code requesting more memory, they use a pointer to refer to the new memory allocated from a heap. This pointer becomes the developers’ link to that memory, and developers can use it to copy its contents or accomplish any task they would do with other data. Afterward, the developer can free the memory (i.e., return it to the platform’s heap). After memory has been freed, the program no longer has use of the memory and must not try to read or write to it.This exchange of memory between platform and program works well as long as the program stays within the bounds of the memory blocks provided by the platform. Unfortunately, programs often request hundreds of blocks of memory. Tracking those memory blocks can become a major effort with disastrous consequences should the developer make a mistake.The buffer overflow is a common programming error in which a program writes to an object (usually a string), but the program exceeds the bounds of memory allocated to the object and overwrites adjacent memory. Buffer overflows that occur in the heap are typically not as easy to exploit as buffer overflows in the stack. They can be commonly exploited, however, as Solar Designer showed in his blog post.Some programmers make the mistake of freeing memory twice (double-free errors) or never freeing it at all. Programs that free memory twice have been exploited both in the lab and in the wild. Failing to free memory may not be particularly harmful, but long-running programs that fail to free memory will often continue to request memory until the available memory is exhausted. Memory exhaustion can cause these programs, as well as any other programs on the same platform, to crash when their subsequent requests for more memory are denied.Most modern programming languages (e.g., Java, C#, and Python) prevent these problems by limiting pointer usage. While this can achieve memory safety, it also eliminates a major source of power and flexibility from these programming languages. Developers who require strong control of memory management must use C and C++, despite the memory management errors that often plague programmers who use these languages without being extremely careful about properly freeing their pointers.  The Pointer Ownership ModelThe C language has a fairly simple set of system calls for managing memory. If you want memory from a platform, you would use the malloc() system call. Conversely, if you want to release a block of memory back to the system, you would use the free() system call. C++ has similar mechanisms for allocating and freeing dynamic memory via the new and delete operators. Every program has the difficult task of keeping track of allocated pointers and freeing each pointer exactly once. In deference to the issues with memory management, some developers write their own application programming interface (API) to manage memory—usually in the form of wrappers around malloc() and free(). The evolution of these APIs has produced several well-known memory-management techniques, such as reference counting or garbage collection, to keep track of dynamic memory usage. Many larger software programs, such as Firefox, use one or more such systems to track the pointers that must be freed.Many other programs, however, provide no systematic scheme for tracking allocated pointers, and therefore trust that the programmer "got it right." Our approach, called the Pointer Ownership Model, applies to programs that do not have their own memory management scheme. The Pointer Ownership Model aims to help software developers distinguish pointers that need to be freed from pointers that don’t. This can be accomplished by using static analysis and developer annotations to produce strong safety guarantees. The Pointer Ownership Model partitions pointers in a program into "responsible" pointers and "irresponsible" ones. A responsible pointer must be freed eventually, and an irresponsible pointer must not be freed. The theory behind our research is that within a software program, if all the pointers are divided into responsible pointers and irresponsible pointers, that program can be automatically checked to ensure that every responsible pointer gets freed and every irresponsible pointer never gets freed. This check can be done at compile-time so that the program does not need to be run.Our approach involves designing and implementing an ownership model for dynamically allocated memory performing sound static analysis of the consistency of preexisting C source code and annotations evaluating efficacy by analyzing existing open source programs with known and seeded errors measuring annotations required per thousand source lines of code (KSLOC) To implement our Pointer Ownership Model, we are building two programs: an advisor and a verifier. The advisor takes a file of C source code and builds an ownership model of the pointers used by the code. In other words, the advisor examines the code and determines which pointers are responsible and which are not. If the advisor cannot determine the responsibility status of a pointer, it will ask the user. Once the model is complete, the verifier can then take the model and the source code and indicate if the code complies with the model. If the code does not comply with the model, an error message would be produced. This could happen, for example, if a responsible pointer goes out of scope while never being freed.Related WorkMemory management is an old problem, and there are many tools, free or proprietary, to help programmers tackle it. Most memory debugger tools employ dynamic analysis. They monitor a program as it runs while tracking its use of the heap, and they generate error messages if they detect memory bugs. Valgrind is one popular dynamic memory debugger for Linux systems. These tools accurately report errors; however, they impose a performance hit on program execution and so cannot be used on production code. A few debuggers use static analysis, which does not require the program to be run. Instead they analyze the program’s source code. Because the debuggers do not observe a running program, they can only theorize about what errors may occur, so they can suffer from false positives (reporting an error in a bit of code where there is no error) and false negatives (failing to report a true error in the code). A false positive requires a developer to inspect the code and verify that it needs no change, but a false negative can cause the analyzer to declare the code to be bug-free when it is not. False negatives are considered a far worse problem than false positives, and many static analysis tools err on the side of minimizing false negatives while issuing lots of false positives. Coverity and Fortify both produce commercial static analysis tools, and Splint is a free static analysis tool.The Pointer Ownership Model employs static analysis, but unlike traditional static analysis tools, it requires extra semantic information not provided by traditional C source code. The Pointer Ownership Model simply needs to know which pointers are responsible and which are irresponsible. The programmer must supply this information, although the advisor program should do most of the work. This semantic information means that the Pointer Ownership Model will not be subject to false positives or false negatives. The error messages it generates will be sound, and it will not overlook violations of its model. And, because it does not employ dynamic analysis, it will impose no runtime penalty.CollaborationsTo create the verifier program, I am collaborating with Lutz Wrage, a researcher in the SEI’s Research, Technology, & System Solutions Program. Lutz is building a prototype of the model using the C Intermediate Language (CIL) Framework. The program that we are creating is an open source project, and it needs to contain a C language parser that can build an intermediate representation of a C program. This is essentially the first step performed by a compiler. But our goal is not to build a compiler; it is instead to conduct an in-depth analysis of the C program to enforce our model. Impact to the DoD The verifier program that we are developing will enable the Department of Defense (DoD) to expand its capabilities with the proof-of-concept secure C compiler. It will also influence C language standards and commercial compiler development. We also hope the verifier will provide the DoD and DoD contractors with a secure compiler technology. Accounting for Corner Cases & Other ChallengesThis approach will face several challenges. One challenge centers on developing a sufficiently timely approach whose benefits will outweigh programmer effort. The research must also demonstrate a low annotation effort and low runtime overhead. One of the most difficult challenges in building any program is corner cases, which in this instance would be valid code that we did not account for. The C language is complex and contains constructs that will derail our model. For example, our current model does not handle arrays of responsible pointers, so we disallow them and require that all pointers in an array of pointers be irresponsible. While forbidding responsible pointer arrays keeps the model consistent and simple, it greatly limits its usability. Therefore a future goal of this research is to extend our model to handle arrays of responsible pointers.We have several small code examples that we can run through our model verifier once we have built it. Some of these code examples comply with a consistent model, and others intentionally violate the model. The verifier should produce an error message when handed one of these noncompliant examples. We eventually want to test the verifier on a larger software program so that we can understand the true impact of corner cases on our research. Another challenge will be transitioning our implementation from operating on toy code examples to real-world code. Consequently, one of our research goals is to survey open source software to find out which projects use their own memory management model and which do not. An example of the latter would be a suitable test case for the Pointer Ownership Model.Future Research Our current research posits that our model of pointer ownership can be applied to C programs that do not already provide their own ownership model. We can extend our model in several ways, including handling arrays of responsible pointers handling C++ programs These will be subjects for future research.Additional ResourcesFor more information about the work of the CERT Secure Coding Team, please visit http://www.cert.org/secure-coding/.

By Douglas C. Schmidt Principal Researcher In launching the SEI blog two years ago, one of our top priorities was to advance the scope and impact of SEI research and development projects, while increasing the visibility of the work by SEI technologists who staff these projects. After 114 posts, and 72,608 visits from readers of our blog, this post reflects on some highlights from the last two years and gives our readers a preview of posts to come. First, the numbers. Since the blog was launched in early 2011 the top 10 posts, in terms of visits to the site, are Improving Security in the Latest C Programming Language Standard What is Agile? Strategic Planning with Critical Success Factors and Future Scenarios Fuzzy Hashing Techniques in Applied Malware Analysis The CERT Perl Secure Coding Standard The Importance of Safety- and Security-related Requirements, First of a Three-Part Series Writing Effective YARA Signatures to Identify Malware Improving Testing Outcomes Through Software Architecture Cloud Computing for the Battlefield The Growing Importance of Sustaining Software for the DoD One observation about these top posts is the wide range of topics covered by the blogs, including secure coding, malware analysis, organizational planning, agile software methods, quality assurance, cloud computing at the tactical edge, and software sustainment across the lifecycle.  These topic areas reflect the diversity scope, and impact of the work being done at the SEI. Numbers aren’t the only metric. One goal of the blog when we launched was to provide immediate and accessible insights into the broad spectrum of work we do at the SEI via a two-way "read-write" medium that allows our audience to interact with SEI technologists, rapidly and effectively.  Several posts that sparked dialogue between researchers and readers have highlighted the success of this model, including Robert Nord’s post on Rapid Lifecycle Development in an Agile Context and David Svoboda’s post on the CERT Perl Secure Coding Standard, which drew substantial feedback from coders. Another goal of the blog was to give our audience immediate and accessible insights into our work. One success story in this arena stemmed from a post on fuzzy hashing in applied malware analysis by CERT malware researcher David French. Within a day of its publication, the post was referenced in the blog, Technology Review, which is maintained by the Massachusetts Institute of Technology. A few days after that, a writer for the Tech Republic blog interviewed David French about his research into fuzzy hashing.   While the SEI remains committed to publishing our work in traditional venues, such as peer-refereed journals and conferences, it’s also clear that social media dissemination vehicles like blogs can have tremendous impact in a short time period. Securing the Cyber Infrastructure Many of our posts have focused on securing the cyber infrastructure. The CERT Secure Coding Initiative is conducting research to reduce the number of software vulnerabilities to a level that can be mitigated in DoD operational environments. This work focuses on static and dynamic analysis tools, secure coding patterns, and scalable conformance testing techniques that help prevent coding errors or discover and eliminate security flaws during implementation and testing. The post that brought in the most visitors during the past two years, Improving Security in the C Programming Language Standard by David Keaton, explored two of the changes—bounds-checking interfaces and analyzability—from the December 2011 revision of the C programming language standard, which is known informally as C11 (each revision of the standard cancels and replaces the previous one, so there is only one C standard at a time). Other popular posts in this area highlighted work by SEI researchers who are developing tools to analyze obfuscated malware code to enable analysts to more quickly derive the insights required to protect and respond to intrusions of DoD and other government systems. Their approach, as described in a post by Sagar Chaki, uses semantic code analysis to de-obfuscate binary malware to a simple intermediate representation and then convert the intermediate representation back to readable binary that can be inspected by existing malware tools. A Growing Importance in Software Sustainment The high costs of software sustainment (which account  for 60 to 90 percent of the total software lifecycle effort) are receiving increased attention as the DoD wrestles with the ramifications of sequestration. Over the last two years, we’ve dedicated a substantial portion of this blog space to the importance of highlighting our efforts to help the DoD sustain software more effectively. Mike Phillips wrote a series of posts on efficient and effective software sustainment. The first post highlighted specific examples of the importance of software sustainment in the DoD, where software upgrade cycles need to refresh capabilities every 18 to 24 months on weapon systems that have been out of production for many years, but are expected to maintain defense capability for decades. The second post described effective sustainment engineering efforts in the Air Force, using examples from across the service’s Air Logistics Centers (ALCs). In June 2012, Bill Scherlis also penned a post on software sustainment stemming from a research effort that he led that studied defense software producibility, with the purpose of identifying the principal challenges and developing recommendations regarding both improvement to practice and priorities for research. The post highlighted key findings of the report Critical Code: Software Producibility for Defense, a summary of the results of the three-year research effort conducted under the auspices of the National Research Council (NRC).  I also authored a two-part series on this topic based on my involvement in an Air Force Scientific Advisory Board study on sustaining aging aircraft. The first post in the series, The Growing Importance of Sustaining Software for the DoD, summarized key software sustainment challenges faced by DoD; the subsequent post describes R&D activities conducted by the SEI to address some of these challenges. The second post in the series described key R&D activities conducted by the SEI to address these challenges including work in sustainment R&D, software product lines, Team Software Process, and software architecture. An Interest in Agile While agile methods have become popular in commercial software development organizations, the engineering disciplines needed to apply agility to mission-critical, software-reliant systems are not as well defined or practiced. Perhaps it’s no surprise, therefore, that the category visitors to the blog site have clicked most on is "agile," and one of the most popular posts in that series is Stephany Bellomo’s "What is Agile," which has drawn the second highest number of visitors to the site. A strong reader interest in agile also spurred a series of posts that highlighted presentations made during the SEI Agile Research Forum. This forum brought together researchers and practitioners from around the world to discuss when and how to best apply agile methods in mission-critical environments found in government and many industries.  I wrote a series of posts recapping presentations made at the Agile Research Forum by Anita Carleton, director of the SEI's Software Engineering Process Management Program, and Teresa M. Takai, chief information officer at the Department of Defense, who discussed the SEI's research efforts in Applying Agile at Scale for Mission-Critical Software Systems Mary Ann Lapham, a senior member of the technical staff, who discussed the importance of collaboration with end users, as well as among cross-functional teams, to facilitate the adoption of agile approaches into DoD acquisition programs Ipek Ozkaya, a senior member of the technical staff, who discussed the use of agile practices in strategic management of architectural technical debt  Jim Over, manager of the SEI's Team Software Process initiative, who advocated the building of self-managed teams, planning and measuring project process, designing before building, and making quality the top priority, among other principles associated with applying agile methods at scale My presentation on applying agile methods to common operating platform environments (COPEs) that have become increasingly important for the DoD.  Looking Ahead In December, we published a post on the State of the Practice of Cyber Intelligence from the SEI Innovation Center. The post describes a research initiative aimed at helping organizations bolster their cyber security posture by leveraging best practices in methodologies and technologies that provide a greater understanding of potential risks and threats in the cyber domain.  We will continue to work with the SEI Innovation Center to cover their work on data-intensive scalable computing, which includes the following technical areas: heterogeneous high-performance cloud computing cyber intelligence (tradecraft, capabilities, and prototyping new analysis methodologies) adaptive and autonomous systems analytics/applied machine learning prototype application development data architectures human-information interaction In the coming months we will also be continuing our series on the Architecture Analysis & Design Language (AADL) standard, which provides formal modeling concepts for the description and analysis of application systems architecture in terms of distinct components and their interactions. Future posts in this series on AADL will cover tools and real-world applications highlighting experiences from organizations in the medical device and space domains.  On another front, we will continue writing about exploratory research efforts at the SEI, the outcomes of which will determine future directions at the SEI. Most importantly, we’d like to thank our readers for their feedback and insight over the last two years. We value your insights and would welcome your feedback below on ways we can improve the SEI Blog to better serve our audience. Additional Resources To download the latest SEI technical reports and notes, please visitwww.sei.cmu.edu/library/reportspapers.cfm

By Grace LewisTechnical Lead Edge-Enabled Tactical Systems Research In 2009, a popular blogger published a post entitled "SOA is Dead," which generated extensive commentary among those who work in the field of service-oriented architecture (SOA). Many practitioners in this field completely misinterpreted the post; some read the title and just assumed that the content referenced the demise of SOA. Quite the opposite, the post was inviting people to stop thinking about SOA as a set of technologies and start embracing SOA as an approach for designing, developing, and managing distributed systems that goes beyond just the technology. Unfortunately, even though SOA is still alive and widely adopted, a belief still persists that SOA can be purchased off the shelf. This post highlights recent research aimed at clarifying this misperception for architects, as well as identifying the elements that constitute a service-oriented system and the relationships between these elements. Service Orientation in a Nutshell Service orientation is a way of designing, developing, and deploying software systems. The architecture of a service-oriented system can typically be characterized by three high-level components: services that represent a reusable business functionality exposed via standard service interfaces service consumers that use the functionality provided by these services as part of the processes they implement an SOA infrastructure that connects service consumers to services Rather than being viewed as an off-the-shelf product, SOA should be viewed as an architectural style or paradigm for designing systems. From this paradigm, an infinite number of configurations can be derived that either benefit or harm a system. Even organizations that acquire products to build a service-oriented system from a vendor must contend with numerous architectural and design decisions. These decisions include deciding what services to expose and whether the functionality exposed comes from legacy systems, is built from scratch, or is acquired from a third party.  Other critical considerations include deciding how information about the system will be accessed and shared, as well as how to promote quality attributes that are important for a particular organization. For example, service-oriented systems typically promote interoperability and modifiability at the expense of optimal performance and stringent security. Implementing a Service-Oriented System Many architectural and design decisions must be made when implementing a service-oriented system. When an organization acquires an SOA infrastructure product, such as an enterprise service bus (ESB), from a vendor it is buying only one piece of the service-oriented system. The architecting of a service-oriented system, however, begins when the system stakeholders define the quality attributes that must be met by the system, such as easy and flexible integration with legacy systems (interoperability) streamlined business processes (maintainability) reduced costs (modifiability) agility to handle rapidly changing business processes (extensibility) Common components of an SOA infrastructure that can help meet some of these quality attributes include a service registry and repository, which can be custom-built or, more commonly, provided by a product in the SOA infrastructure. Typical functionality includes dependency management, discovery, versioning, event notification, access control, policy management, and federation capabilities. a messaging system, also known as message-oriented middleware, which is often a part of execution environment of enterprise applications. A messaging system allows distributed components to exchange asynchronous messages. Today messaging systems are commonly used in service-oriented systems for transactions that involve background processing because they provide high levels of scalability and reliability. a business process engine, which is a software component responsible for processing incoming requests by performing the steps—defined according to rules—of business processes that correspond to that request. A key element of the business process engine is a business process modeling (BPM) tool that enables the description—and often visualization—of the business processes. The BPM tool then generates a business process that is deployed to and executed by the business process engine when processing requests. monitoring and management tools, which enable organizations to detect, diagnose, and react to potential problems in service-oriented systems. These tools are used for runtime monitoring to provide information that can be used to maintain system quality of service and to inform tactics and patterns, such as dynamic load balancing, failover and recovery procedures, and service removal and replacement.  SOA Architecture Principles Service-oriented systems, like any system, have business mission goals and a corresponding set of desired quality attributes that drive the architecture of the system. SOA, as an architectural style, has a set of service-oriented principles that influence the architecture of a system and impact a system’s quality attributes. When these two sets of quality attributes collide, conflicts arise, and architects must apply each principle in the context of the business goals of the system to make necessary tradeoffs and to make architecture decisions to meet the system’s business goals. Some of the key principles are described belowLoose Coupling Loose coupling can play a great role in impacting quality attributes (either positively or negatively) in SOA adoption. Architects can enable loose coupling by making several key decisions, including ensuring that service providers and service consumers can make independent decisions about technology creating services that are independent, self-contained capabilities that can be used in isolation from those capabilities provided by other services enabling the actual binding between a service consumer and a service at runtime  Reusability Reusability mostly refers to service reusability, which includes services that are reusable because they represent self-contained functionality that can be reapplied in multiple business processes. If service reusability is a goal, architects should identify services that perform utility operations, such as logging and data manipulation, and design them so they are independent of the business processes that use them design (when possible) stateless services that do not maintain a conversational state provide an infrastructure that abstracts or mediates differences between service consumers and service interfaces, such as different versions of standards (e.g., SOAP 1.1 vs. SOAP 1.2) or data formats (e.g., XML vs. CSV). Composability Composability enables organizations to rapidly change elements of a business process in response to changes in the business environment, without impacting consumers of the composite service that implements the business process.  Service composability relies on many of the same characteristics as service reusability including self-contained functionality standardized interfaces availability in a service registry Composability introduces architectural risks that require strategies for mitigation, including compositions with services that span multiple organizational boundaries or transactions that contain services that belong to different systems. Discoverability Discoverability in a service-oriented environment refers to the creation and publication of services in a location (such as a service registry, web page, or directory) that is accessible to and can be queried by service consumers. Metadata associated with a service includes its interface specification or contract in addition to any other data that can help a developer make decisions about whether or not to use that service.StandardizationStandardization is perhaps the service-oriented principle that has contributed the most to widespread SOA adoption. Especially for Web Services, standardization has driven the creation of a large amount of tools and third-party components that can be leveraged by developers and can effectively lead to shorter development and deployment times. In addition, standardization enables the leverage of legacy systems because these can remain in their native platforms while exposing their functionality via standard interfaces. A caveat when using Web Services is that while the basic WS-* stack (HTTP, XML, WSDL, SOAP) has remained fairly stable over the years, other standards such as, WS-Security, WS-Transaction, or WS-Addressing continue evolving and contain areas that are subject to multiple interpretations. Aware of this shortcoming, the WS-I organization was created to promote interoperability by providing clarifications, refinements, interpretations, and amplifications in areas of specific standards that are indeed subject to multiple interpretations. Applying Our Research on Service-Oriented SystemsI authored, with fellow researchers, a technical report that described the effect that service-oriented principles have on system quality attributes. As an architectural style, SOA may be an appropriate solution in some situations, but there will be other situations where it is not appropriate, or it has to be used in conjunction with other technologies to achieve the desired system qualities. SOA architects are often at a conflict because on one hand there are business and mission goals that dictate the quality attributes that are important for the system. On the other hand, SOA architects using service-orientation want to adhere to service-orientation principles and leverage those to the advantage of the system. Our intent in documenting our research is to help architects of service-oriented systems design better systems and understand the tradeoffs. Later this month, at the SEI Architecture Technology User Network (SATURN) Conference (SATURN 2013), my SEI colleague, Stephany Bellomo, will be teaching a two-day course, Advanced Topics in Service-Oriented Architecture, that expands on the concepts presented in this posting. Additional Resources To learn more about the course, Advanced Topics in Service-Oriented Architecture, which will be offered April 29 and 30 at SATURN 2013, please visit http://www.sei.cmu.edu/saturn/2013/courses/index.cfm#advsoaTo read the SEI technical report, Architecting Service-Oriented Systems, please visithttp://www.sei.cmu.edu/library/abstracts/reports/11tn008.cfmTo view the first installment of a virtual tutorial that Grace presented on Architecture and Design of Service-Oriented Systems, please visit http://www.sei.cmu.edu/library/abstracts/webinars/Architecture-and-Design-of-Service-Oriented-System.cfm

By Julien DelangeSenior Member of the Technical Staff     Research, Technology, & System Solutions Software and systems architects face many challenges when designing life- and safety-critical systems, such as the altitude and control systems of a satellite, the auto pilot system of a car, or the injection system of a medical infusion pump. Architects in software and systems answer to an expanding group of stakeholders and often must balance the need to design a stable system with time-to-market constraints. Moreover, no matter what programming language architects choose, they cannot design a complete system without an appropriate tool environment that targets user requirements. A promising tool environment is the Architecture Analysis and Design Language (AADL), which is a modeling notation that employs both textual and graphical representations. This post, the second in a series on AADL, provides an overview of existing AADL tools and highlights the experience of researchers and practitioners who are developing and applying AADL tools to production projects. Improving Software Development with Tools According to the IEEE Software article, The Impact of Tools on Software Productivity, "Tools have long been recognized as an effective way to improve software development variables such as productivity and product quality." System engineers use different notations and tools that are hard to integrate due to different representation methods and semantics gaps between notations. For that reason, tools must interoperate to help engineers bridge the gap between different specifications.  A number of AADL tools have been developed by organizations, such as Rockwell Collins, the European Space Agency, Honeywell, and Ellidiss Software. These tools provide modeling concepts that describe the runtime architecture of application systems in terms of concurrent tasks, their interactions, and their mapping onto an execution platform.  To support AADL for the design of life-/safety-critical systems, SEI researchers—in conjunction with members of the AADL ecosystem—have developed various tools that help define specifications with modeling capabilities, implementations with automatic code generation, and validation and/or verification with analysis plug-ins that check architecture issues from corresponding models. Examples of AADL Tools Popular AADL tools include the Open-Source AADL Toolset Environment (OSATE), which supports AADL within the Eclipse environment, and Ocarina, which is a command-line tool for expert AADL users.OSATE. The SEI developed OSATE to support the AADL language within Eclipse, which is popular in the software development community and offers built-in support for many languages including Java, C, and C++. Adding support for AADL thus also facilitates connections with existing tools and avoids dedicated training on Eclipse. OSATE supports the core AADL language, as well as several annex languages (such as the error or behavior annex language for designing specific aspects of the system) and several plug-ins for validating system requirements (such as latency or dependability).OSATE is released under the open-source Eclipse Public License (EPL) and available for most platforms (including Windows, Mac OS X, and Linux). Two versions of OSATE are available: The stable version is updated every two to three months after each AADL committee meeting. This version has a set of supported functionalities that are considered "stable," but it does not include all the latest functionality. The testing version is evolving constantly and rebuilt every night with the latest patches and bug fixes. This version is considered "unstable," however, because it is continually under development. The OSATE source code is available on github, which is a social platform for code-sharing that allows users to contribute, potentially learn, and reuse OSATE-based functions for their own projects. As with most active, open-source projects, a dedicated bugtracker lists existing bugs and planned improvements. OSATE extensions allow users to develop their own AADL tool on top of OSATE without having to re-design everything from scratch. Many users created OSATE extensions for research projects, such as the System Architecture Virtual Integration (SAVI) program, which was undertaken by the Aerospace Vehicle Systems Institute, or the Refinement of AADL Models for Synthesis of Embedded Systems (RAMSES), a code generation framework that targets avionics and automotive systems. The most widely-used plug-ins for OSATE include The Requirements Definition and Analysis Language Tool Environment (RDALTE), which connects AADL models with requirements documents. The Instance Model Viewer (IMV), which provides a graphical visualization of AADL models. RC Meta, an Eclipse-based toolset built by Rockwell Collins, which assists with the design and validation of cyber-physical systems. RC Meta includes an AADL /SysML translator and helps engineers track the consistency of models when using both notations. The LUTE constraint language also developed by Rockwell Collins, which queries and validates a model by associating constraints (such as properties to be declared, required composition, etc.) with model components. OCARINA. Jerome Hugues from the Institut Superieur de l’Aeronautique et de l’Espace (ISAE) developed Ocarina to provide the following capabilities for command-line driven use of AADL: Model parsing and analysis, which allows users to check that AADL models are syntactically and semantically correct. Model validation and verification, which allows users to check model properties and consistency. For example, Ocarina can check whether a model contains a given amount of memory component or that the processor associated with some software components has enough processing capacity to process data on time. Code generation, which targets embedded platforms, including the generation of code that implements the system described by the model. The code-generation features of Ocarina target embedded platforms and operating systems, such as Real Time Operating Systems (RTEMS), which are common in the aerospace industry; ARINC653-compliant systems, which are common in avionics; and other mainstream platforms, such as Linux and its real-time variants. Ocarina is now part of the TASTE framework, which is supported by the European Space Agency to automate the design and implementation of their life-/safety-critical projects.  For example, the European IST-ASSERT project enabled the automated production of system implementation from models—a major achievement. In particular, it generates and manages the configuration code of embedded systems, which is challenging because life-/safety-critical systems have specific configuration and deployment requirements. For example, they often need to set the polling period for obtaining sensor data and the need to ensure a task’s completion at a specific pace. Leveraging the AADL Ecosystem to Extend Its Power and Reach One of the strengths of AADL is the expertise and passion of technologists in its ecosystem. To glean insights from some of them, we interviewed the following AADL researchers, developers, and users at the January 2013 AADL Committee Meeting: Steven Miller and Darren Cofer work at the Rockwell Collins Advanced Technology Center, which develops innovative solutions for both its commercial and military business units.  They work in the Trusted Systems group where they develop and apply advanced analysis tools, including formal methods that can be used by the center’s business units to build and verify software components in high-integrity systems, such as avionics flight and mission computers.  Jerome Hugues is an associate professor at ISAE, the reference engineering school in Aeronautics and Space Engineering. He is also co-head of the Advanced Master on Embedded Systems (MS EMS) program and the last-year computer science curriculum. Hugues specializes in the teaching of software architecture. His research is focused on embedded systems architecture with a strong focus on the AADL, as evidenced by his work on the Ocarina command-line-driven AADL tool described earlier. He has been part of the AADL standardization committee since 2006, and a member of its steering committee since 2011. Miller and Cofer have been applying AADL and OSATE for modeling distributed real-time and embedded systems that are of interest to their company. As they described it We began serious investigation of AADL as part of our DARPA-funded META project in 2010.  We used AADL to model portions of an avionics system architecture.  Our objective was to use AADL to support compositional reasoning about the system-level behavior of an architectural model in which the behaviors of individual components are specified by means of assume-guarantee contracts. Under the META project, Rockwell Collins implemented a new constraint language, Lute, on top of OSATE to verify the model characteristics according to design requirements and/or contracts.  For example, Lute can be used to check that a collection of tasks, such as those computing statistics from a sensor, are all executing at the required rate. Engineers use these tools to check integration consistency. Hugues began investigating the use of AADL in 2006 while working on the IST ASSERT Project, which was led by the European Space Agency to study the design of on-board software that is integrated in launchers or satellites. In our discussion, Hugues described his development of Ocarina: Relying on complex, object-oriented frameworks was a blocking factor, as those cannot be easily qualified. Switching to AADL was appealing: I could separate the modeling of the configuration of my system from the actual logic, and have a tool that generates automatically for me the application-specific middleware, instead of using object-orientation. This was one of the first contributions I made to AADL through the Ocarina toolset. The AADL ecosystem integrates well with and offers many connections to other tools used for designing life-/safety-critical systems, as Hugues reported: During my last projects, and as an academic, I tested (or reviewed) most AADL tools: OSATE, STOOD, but also gateways to other analysis tools (formal methods such as timed automata, petri nets; scheduling analysis; model transformation). It is fascinating to see the diversity of analysis supported from one single description language. I also contribute to the implementation of some of them through the Ocarina toolset. In that context, AADL is a good candidate to connect different notations. Likewise, Miller and Cofer highlight how AADL’s well-defined semantics and description of both hardware and software concerns help bridge the gap between different notations and avoid usual issues such as semantics inconsistency and notational differences: Some of our engineers are using SysML to model system architectures and requirements and we expect SysML to be the dominant modeling language in our product divisions.  However, the SysML language does not provide sufficiently formal semantics to support the kinds of analysis that we want to do.  We have tried to resolve this problem by developing translators from SysML to/from AADL. Having used AADL as an underlying language to connect different representations, Miller and Cofer reported promising feedback: There are a number of tools and efforts aimed at system-level modeling of complex embedded systems.  We would like to see AADL used as a common underlying language for these systems that could be used for both synthesis and analysis. This feedback is also shared by Hugues, who described using AADL with other languages on his projects: In parallel, I investigated the use of other formalisms (such as Simulink, SCADE or SysML) in conjunction with the AADL. My main motivation to keep using AADL for my future projects and research is that AADL acts truly as a "backbone" for your system: it nicely federates all the blocks of your systems, but also lets you use external models when needed. Such adaptability makes AADL appealing: I can focus on the architecture using one formalism, and bind it to another one for requirements engineering, control/command, etc. The Future of AADL Tools SEI researchers have been developing new tools and functions in OSATE for many years. We are now developing features to support the specification of faults in the system architecture and provide analysis tools so that users may document their impact. For example, in the medical domain, a fault with a pump infusion system on the patient device should not propagate to the injection subsystem nor alter the quantity of drugs being injected. On the other hand, new functions motivate new needs, as Miller and Cofer highlighted: The lack of a stable graphical environment has been a limitation both in communication (demonstrating results to others) and in quickly/accurately modeling new systems.  It is often difficult to figure out how to use some of the more advanced features of AADL when we need them. The correct syntax is difficult to deduce from the reference manual and the examples available often do not illustrate what we want to do. To address some of these needs, the Instance Model Viewer (IMV), a graphical viewer, has been added to OSATE. The graphical edition is more convenient than the textual version and will make OSATE more accessible to the user community. Convenience is also one of the reasons that engineers use other modeling frameworks that support a user-friendly graphical notation, even if they do not support analysis or validation capabilities. As Hugues reported: The main challenge I’ve seen is human-related: convincing people to invest on a new language instead of relying on the (presumably) known UML derivatives. When people caught on to AADL, they understood how much they can benefit from it. The existing IMV modeling framework motivates the need for new research projects and opens new possibilities. Users want to experiment with new features and additions in the latest version of AADL, including the new components and language annexes (such as ARINC653 or the Error-Model). Miller and Cofer described a similar experience with respect to the DARPA META program: We are continuing the work on compositional verification that we started in META in DARPA’s HACMS project, as well as in projects funded by AFRL and NASA.  In the HACMS project, we are using AADL to model a UAV system and developing new tools to verify that it is robust against cyber attacks. This blog post presents an overview of existing AADL tools, highlighting two key tools developed and used by the community. Many other tools exist, however, that are either open-source or commercial. Users who are interested in learning more about these tools can visit the AADL community wiki, which provides an extensive list of the tools that support AADL. The next post in this series will highlight the use of AADL in the medical domain, including reports from researchers that use AADL for specifying and validating medical devices/domains. Additional Resources OSATE https://wiki.sei.cmu.edu/aadl/index.php/Osate_2 OSATE2 download: https://wiki.sei.cmu.edu/aadl/index.php/OSATE_2_download_page OSATE2 source code: https://github.com/osate/ IST-ASSERT: http://www.assert-project.net/ META project: http://cps-vo.org/node/2675 AADL latency analysis http://www.sei.cmu.edu/library/abstracts/reports/07tn010.cfm AADL dependability http://www.sei.cmu.edu/library/abstracts/reports/07tn043.cfm AADL wiki: https://wiki.sei.cmu.edu/aadl/index.php/Main_Page