By Bill PollakTransition ManagerResearch Technology & System Solutions It’s undeniable that the field of software architecture has grown during the past 20 years. In 2010, CNN/Money magazine identified "software architect" as the most desirable job in the U.S. Since 2004, the SEI has trained people from more than 900 organizations in the principles and practices of software architecture, and more than 1,800 people have earned the SEI Software Architecture Professional certificate. It is widely recognized today that architecture serves as the blueprint for both the system and the project developing it, defining the work assignments that must be performed by design and implementation teams. Architecture is the primary purveyor of system quality attributes, which are hard to achieve without a unifying architecture; it’s also the conceptual glue that holds every phase of projects together for their many stakeholders. This blog posting—the final installment in a series—provides lightly edited transcriptions of presentations by Jeromy Carriere and Ian Gorton at a SATURN 2012 roundtable, "Reflections on 20 Years of Software Architecture." Jeromy Carrierehttp://www.sei.cmu.edu/library/abstracts/presentations/carriere-panel-saturn2012.cfm In 1999 and 2000, I had just left the SEI to join a startup company that I co-founded. I came out of the SEI with great respect for software architecture. This [Please view Slide 2 from Carriere’s presentation.] was the top-level, logical view of the back end of my startup company’s software. Not many startup companies then—and certainly not now—do architecture models of this sort. This was an early model-driven design; it actually drilled down into mechanisms between components, and it generated code. That was cool, I had a little experimental testbed, and it was fun. All of these factors led me to develop a different working definition of architecture: A system’s architecture codifies a set of decisions that are both hardest to change and that have the most significant impact on the way the system manifests its quality attributes. This diagram [Please view Slide 6 from Carriere’s presentation.] helped us sell the company for $200 million to AOL. This is completely informal, but it reflects what’s important about architecture; that it was quality-attribute driven. We were making decisions as a small startup company based on what was important. What was paramount for us was flexibility. Above all else, we knew that we didn’t know where the business was going. In the very short life of the company, the business model changed substantially four times; the software architecture did not change once. Because the software architecture of the system was designed specifically to adapt—from a physical-device company to a voice-based thing, B2B, B2C… But this [refers to architecture diagram in slide] was maintained throughout. And that was enough to get us the deal. We sold the company to AOL, and then it failed. To put it bluntly, it was designed to be sold. We built our first product—AOL By Phone—in 90 days. From the day that we did the deal to the day that we deployed the system, including deploying hardware, it was 90 days. Later in my career, I still loved diagrams. This [Please view Slide 7 from Carriere’s presentation.] is a fantastic diagram labeled "critical systems interrelations overview," but it is useless. What good is that [refers to slide 7] besides being decorative? I wanted to be able to find a reason to use it in my day job, but it didn’t have any use. This was a period in my architectural career where I came to this conclusion and started to take on a different perspective. Last year at SATURN, I talked about low-ceremony architecture, which is how do you architecture like a startup. This is based on my Quack.com experience. [For more information about Carriere and his experience co-founding Quack.com, read the article "SEI Architecture Practices Propel Successful Startup."] If you want to get a job at Facebook, Twitter, Google, or Netflix, do not put "architect" on your resume. You will not get hired. Call yourself a lead engineer or something else, but not architect. In Silicon Valley, there’s a lingering allergic reaction to "Big A" Architecture, which is characterized by setting standards and making rules and drawing diagrams, those kinds of activities that we have all experienced. That disconnection from the actual engineering practice of developing software and what became architecture in some companies that were predecessors in the net space is deeply rooted in Silicon Valley. What I find most ironic is that these kinds of companies need architecture practice the most. This goes back to the Quack [Quack.com] story; we needed a perspective on quality attributes, prioritizing for flexibility, thinking about how you’re prioritizing architectural decision making because you’re living in a very uncertain world. Ian Gorton, Pacific Northwest National Laboratoryhttp://www.sei.cmu.edu/library/abstracts/presentations/gorton-panel-saturn2012.cfm We've seen our profession mature over the past 20 years. We've made advances in methods, processes and tools, but the scale of systems has grown enormously. But has there been anything truly innovative in the past five years in practice or research? While we apply the methods and tools that we know and love, the software world is getting more complex, and our solutions are very much based on qualitative engineering approaches. In the future, maybe we need to become more quantitative, applying more objectivity in our assessments and designs. There has been some initial work in many areas. I've done work in combining decision theory with optimization techniques to enable architects to quantitatively assess complex, multidimensional design alternatives. There are tools for modeling performance with multiple modeling approaches built into the tools, which makes them easy to use. So I think there's a lot of work we can leverage and extend here if we actually look toward future problems. It's not that we have to invent any mathematics or statistics; there's a rich body of math and stats that we can exploit. We just need to know when and how quantitative methods can be usefully applied. Quantification is in some ways harder, but it doesn't stop others from addressing really hard problems. Climate modelers, for example, build simulations that are being used by governments around the world to predict future climate and mitigating policies. These climate models are tested and validated against historical data. When they don't produce results that exactly match the historical data, they are calibrated to provide reasonable outputs. But there's no guarantee that these calibrated models are going to be able to predict the future. So if climate scientists can do this, we can too. Our current and emerging design and quantification models and methods may not (or may never) produce 100 percent accurate predictions of software architecture characteristics, but they will be much better than our current qualitative approaches. We have to move toward quantification because there are challenges in the exponential growth of system scale. There are huge software systems being employed in clouds that are so complex that our current architecture design methods are inadequate. To more rigorously address these systems of massive scale, we need concerted R&D [research and development] to move software architecture to more quantitative foundations. This is hard because, as you get to larger scales, things we know and understand well (such as design approaches, technologies, platforms) must be reexamined to address scale. This community is perfectly positioned to discover underlying principles and make these available in systematic approaches that can be broadly adapted to build the highly scalable systems of the future. Additional Resources Carriere’s presentation may be viewed at http://www.sei.cmu.edu/library/abstracts/presentations/carriere-panel-saturn2012.cfm Gorton’s presentation may be viewed athttp://www.sei.cmu.edu/library/abstracts/presentations/gorton-panel-saturn2012.cfm For more information about Carriere and his experience co-founding Quack.com, read the article "SEI Architecture Practices Propel Successful Startup" athttp://www.sei.cmu.edu/library/abstracts/news-at-sei/feature11q02.cfm

By James Edmondson,Senior Member of the Technical StaffResearch, Technology, & System Solutions An autonomous system is a computational system that performs a desired task, often without human guidance. We use varying degrees of autonomy in robotic systems for manufacturing, exploration of planets and space debris, water treatment, ambient sensing, and even cleaning floors. This blog post discusses practical autonomous systems that we are actively developing at the SEI. Specifically, this post focuses on a new research effort at the SEI called Self-governing Mobile Adhocs with Sensors and Handhelds (SMASH) that is forging collaborations with researchers, professors, and students with the goal of enabling more effective search-and-rescue crews. Motivating Scenario: An Earthquake Hits a Populated Area Recent earthquakes and earthquake-induced events like tsunamis in Haiti, Japan, and Indonesia have shown both the destructive power of nature and the compassion of communities to lend aid to people in need.  Compassion alone, however, won’t help emergency responders locate and help all of the potential survivors. Search-and-rescue crews thus need technology to help them utilize the often meager resources they have in the best way possible. Figure 1. Ground crews search for survivors after an earthquake hits a metropolitan area. Automated aerial robots aid crews with thermal cameras that can penetrate rubble to find the trapped and injured. Tackling a Massive Problem Search-and-rescue is a massive undertaking in every conceivable way.  There are issues of scale in geographic area coverage and number of people, devices, and sensors involved. Despite the numbers of people in need, emergency responders must deal with limited mobility and availability of ground crews and a complete loss of most telecommunications infrastructure, which is common after an earthquake. There are only so many people available to help, and it can take days to completely search a single building. Even in a small town, a crew of thousands of emergency responders could take weeks to find survivors, and by then it may be too late.To assist in this type of important mission, the SMASH project is exploring a combination of effective software design practices, artificial intelligence, and inexpensive hardware. These capabilities will allow one responder to control a fleet of quadcopters  or other robotic vehicles that search the rubble (for example, using thermal sensors, ground-penetrating radar, and other types of sensor payloads) and present the results to a smartphone or other human-computer device in a manner that’s useful and efficient to human operators, as shown in Figure 2. Figure 2. Automated aerial robots send information back to a smartphone, tablet, or other visual interface to provide ground crews with the thermal context of the area around the ground crews. We want to extend the capabilities of emergency responders by equipping them with dozens of robotic vehicles per operator, which would allow them to infer context from thermal images and locate survivors in the rubble. A Focus on Human-in-the-loop Autonomy Our central idea is to extend—not replace—the capabilities of human operators, most obviously by providing them with additional coverage. We also provide ground operators the ability to penetrate the rubble and look into dangerous or out-of-reach areas and locate survivors. Enhanced vision is useless, however, if robotic vehicles are not intelligent enough to search locations collaboratively with other agents.  Likewise, the vehicles need to use maps of building locations to proactively search likely survivor zones quickly and take ground operator feedback and guidance on the likely location of survivors. These are central components to our SMASH autonomous design philosophy. Figure 3. Aerial robots can land to form sensor networks for routing information from other personnel or sensors. Robust telecommunications infrastructure is another capability SMASH aims to provide. The robotic vehicles we are working on form their own wireless access points and also have a long-range radio (2 km) that may be useful in communicating with other ground crews or even by survivors with cellphones who need to send important messages, e.g., that they are trapped in a car of a parking garage. Part of the autonomy we are developing focuses on landing the robotic vehicles or using throwable wireless access points to form sensor networks that route information based on the priority of information. Figure 4. Routing and bandwidth allocation are dictated by contextual priority. In this image, two aerial robots detect thermal targets while the other three detect nothing. The two aerial robots have higher priority information and are given preferential bandwidth treatment. Our research on SMASH will also address issues of timing and scale. In a search-and-rescue operation, timing is critical, and data volumes from thousands of ground crews and new sensors can cause an ad hoc telecommunication infrastructure to fail. Consequently, autonomy is only useful if it is designed to work within the constraints of the available bandwidth of existing networking infrastructure.  Solving this problem depends on focusing on prioritization, quality-of-service, and effective bandwidth utilization. For example, in Figure 4, only two of the aerial robot quadcopters are detecting thermal targets. The other quadcopters are not detecting anything of use to the ground crews. Our solution approach focuses on only sending information that is relevant (such as thermal targets) and not constantly broadcasting the full thermal or video images unless specifically requested by the human operators. Moreover, the quadcopters that are perceiving potential human beings have higher priority and the ability to send more data to the operator within the networking bandwidth available. This type of autonomous self-interest is important to provide a scalable distributed infrastructure. Open-source dissemination Open architectures, i.e., open standards and open source initiatives that allow end-users to extend products and services,  are also important for critical infrastructures—not only because transparency helps in creating a better product, but also to reduce the cost of providing solutions to government agencies and non-profits. Consequently, SMASH is built on top of open-source initiatives including the Multi-Agent Distributed Adaptive Resource Allocation (MADARA) project, which provides real-time knowledge and reasoning services for distributed systems, and the Drone-RK project, which provides a programming interface for controlling and processing data from the Parrot AR.Drone 2. We are augmenting these open-source software projects with additional tools that the software community can use to enable human-in-the-loop autonomous system development and deployment. Collaborations To create these new autonomous techniques and tools, we are partnering with researchers and students from around the world. Within Carnegie Mellon University’s department of Electrical & Computer Engineering (ECE) within the College of Engineering, we have formed a core group of professors and students including Anthony Rowe, Larry Pileggi, and Kenneth Mai of the ECE departments, who will be working on extending the Parrot AR.Drone 2 and developing a new type of thermal, wireless-enabled throwable sensor. In the fall of 2013, Tony Lattanze of CMU’s Master of Software Engineering Program will work with students to investigate area coverage problems with collaborative robots and group artificial intelligence. Students from CMU will be joined by graduate students from research universities around the world for summer programs that investigate the search-and-rescue problem space in hopes of extending the state-of-the-art and effecting real world solutions to hard research problems. Contacts and Conclusion The SMASH project is driving the state-of-the-art in human-aided autonomy with an expected software payload that will be deployed to real-world and often mission-critical scenarios. If you have questions about the effort or would like to discuss potential collaboration possibilities, please contact  info@sei.cmu.edu or leave a comment below. Additional Resources For more information about the open-source knowledge and reasoning engine that we are using for distributed artificial intelligence in real-time systems, please visit the Madara project site at https://code.google.com/p/madara/ For more information on the software development kit used  for manipulating and reading images and data from the Parrot AR.Drone 2,please  visit the Drone-RK project site at http://drone-rk.org/ For a cool robotics emulation site that includes models of the Parrot AR.Drone, please try out the V-REP emulator (trial version lasts for 1 to 2 months). For more information please visithttp://www.v-rep.eu/

By Bill ScherlisSEI Principal Researcher and Director, Institute for Software Research The Department of Defense (DoD) has become deeply reliant on software. As a federally funded research and development center (FFRDC), the SEI is chartered to work with the DoD to meet the challenges of designing, producing, assuring, and evolving software-reliant systems in an affordable and dependable manner. This blog post is the second in a multi-part series that describes key elements of our forthcoming Strategic Research Plan that address these challenges through research, acquisition support, and collaboration with the DoD, other federal agencies, industry, and academia.  The first post in this series focused on Architecture-Led Incremental Iterative Development.  This part focuses on the remaining three elements of our strategic plan: (1) designed-in security and quality (evidence-based software assurance), (2) a set of DoD critical component capabilities relating to cyber-physical systems (CPS), autonomous systems, and big data analytics, and (3) cybersecurity tradecraft and analytics. Evidence-based Software Assurance and Certification The goal of the second element in the SEI Strategy Research Plan is a dramatic reduction in the cost and difficulty of making assurance judgments related to quality and security attributes. Achieving this goal is particularly important as systems become more complex and evolve more rapidly. Current approaches for certification and accreditation are largely based on an after-the-fact evaluation of a snapshot of a system. While after-the-fact approaches are effective for certain well-defined categories of components and systems, they tend to break down as systems increase in complexity, scale, and dynamism. They also tend to hinder ongoing evolution, rapid reconfiguration, dynamic loading of components, autonomy, and composition and interlinking of systems-of-systems. Put simply, these established techniques do not scale up, and they do not work well for the emerging software framework-based systems now prevalent in commercial and infrastructural applications. The industry folklore has long asserted that quality-related activities, including security-related assurance, can consume half of total development costs for larger systems. For example, the IBM Systems Journal states that, in a typical commercial development organization, "the cost of providing [the assurance that the program will perform satisfactorily in terms of its functional and nonfunctional specifications within the expected deployment environments] via appropriate debugging, testing, and verification activities can easily range from 50 to 75 percent of the total development cost." Additionally, after-the-fact evaluation practices can add a year or more to the elapsed time required to develop and deploy software-reliant systems. Commercial systems, including products and software as a service (SaaS), cloud-based systems, tend to undergo a relatively rapid and continual evolution. For many of our DoD and infrastructural systems, we similarly need to support a continuous evolution. Some areas of particular technical emphasis include Architecture and composition principles that enable separate evaluation of individual components, with the possibility of combining results to achieve aggregate assurance judgments. These principles are motivated by the reality of modern software supply chains, which are rich and diverse in sourcing and geography. Modeling and analytics to support the diversity of quality attributes significant to DoD and infrastructural systems. These include modeling, simulation, and design techniques that support critical security attributes. Exploration of development approaches that incorporate creation of evidence in support of assurance claims into the process of development. This evidence-based assurance can harmonize incentives to create designs and implementations that can more readily support evaluation. Evaluation and other techniques to support the use of more opaque components in systems, including binary components and potentially dangerous components from unknown sources. Part of the SEI Strategy Research Plan addresses incentives for developers to adapt their architectural structures, development process, and tooling to better accommodate the idea of amassing evidence—during the development process—that can support an eventual assurance claim with respect to quality and security attributes critical to the operation of the particular system. Our work benefits from the fact that the development process can be naturally incremental through the composition of components and the incremental validation of assurance claims. This development process is supported by the partial accumulation of engineered artifacts and evidence. These ideas of "designed-in security" build on the fourth theme of the Networking Information Technology Research & Development (NITRD) Program plan, "Trustworthy Cyberspace: Strategic Plan for the Federal Cybersecurity Research and Development Program." Critical Component Capabilities The goal of the third element in the SEI Strategic Research Plan is to enhance DoD software capability in several areas that have critical and pervasive roles in DoD software-reliant systems. These areas include composable, cyber-physical systems (CPS), autonomous and distributed systems, and high-performance, data-intensive computing.Each of the areas presents challenges: Composable cyber-physical systems (CPS). "Cyber-physical" refers to the fact that embedded software operates in the context of physical sensors and affectors. CPS thus includes control systems, real-time systems, and many other categories of systems pervasive in DoD and critical infrastructure. These systems tend to have greater complexity due, for example, to higher coupling and the need to model and manage associated physical system components. They also tend to need to assure that real-time deadlines are met. A consequence is that they typically manifest greater internal coupling in their design, thwarting higher levels of capability, composition, and flexibility.There are diverse technical challenges related to design and evaluation, which include improving modeling and analysis for cyber-physical and control systems, modeling and managing hardware reliability issues, making safe use of concurrency and effective scheduling to assure that performance goals can be met, providing support for analysis and testing, and advancing the overall stack architectures beyond legacy concepts. Many mobile systems fall in the category of cyber-physical, including ad hoc networks connecting with large numbers of sensors and enhanced mobile devices deployed to the field. Several recent blog posts outline efforts by SEI researchers to address these challenges.  Autonomous systems. Autonomous systems are cyber-physical systems that can accept sensor data and mission guidance, and, with very limited (or no) human interaction, arrive at mission decisions and enact outcomes. These systems are increasingly critical to the mission, and yet they pose particular challenges for verification and validation, since they rely so much less on ongoing human interaction.  Indeed, the capability and complexity of these systems are often limited for this reason. Systems must both behave as expected and, additionally, not manifest unwanted behaviors that can be dangerous or threaten mission success. From a technical perspective, autonomy can be particularly challenging because of the vast state space, the number of possibilities of combinations of inputs, and challenges of error tolerance, and the difficulty of fully modeling the environmental circumstances of their operation. High-Performance Computing (HPC) for analytics and for modeling and simulation. Advances in sensor fidelity, rapid growth in network capacity, increasing convergence in data-center and high performance computing architectures, advances in large-scale data storage, and emerging frameworks for scalable distributed computing (such as MapReduce and GraphLab) have all resulted in the growing phenomenon of "big data." There are many significant applications of big-data techniques in DoD and infrastructural systems—and in the development of those systems as well. Indeed, many of the other features of the SEI Strategic Research Plan build on progress in big data. Cybersecurity Tradecraft and Analytics The goal of the fourth strategic element is to advance analytic capability in support of diverse aspects of the cybersecurity mission. These aspects include analytics and situational awareness for malware, vulnerability categorization and assessment, vulnerability information management, network activity analysis, threat characterization and assessment, organizational security, and many other dimensions of operational response, remediation, and recovery. This capability builds on a range of data assets related to adversarial tradecraft, malware, vulnerabilities, insider threats, and other results of experience with large numbers of cybersecurity-related incidents. There are diverse purposes of this strategic element, including: Improvement in our understanding and communication of threats and risks Development of better preventive approaches in the engineering of systems and in managing secure operations including considerations for security and assurance "at scale", improved indications and warning, and near-real-time data analysis Support for forensic and corpus analysis Support for hypothesis generation using machine learning, near-real-time analysis, and other advanced capabilities What’s Next The next blog posting in this series focuses on our approach for evaluating and validating SEI research projects. The SEI Strategic Research Plan is designed to ensure that the SEI conducts high-quality and high-impact work that benefits the DoD by identifying and solving key technical challenges facing current and future DoD software-reliant systems. This strategy itself undergoes continual evolution and improvement; the broad range of SEI engagements enable us to continually refine the strategy as the technology advances, the mission evolves, and our understanding improves. We welcome engagement from our partners and stakeholders in the improvement and refinement of this strategy. Please leave comments below or contact us directly at info@sei.cmu.edu. Additional Resources To download the report Critical Code: Software Producibility for Defense, please see www.nap.edu/catalog.php?record_id=12979 To download the Report of the Defense Science Board Task Force on Defense Software (2000), please visit http://oai.dtic.mil/oai/oai?verb=getRecord&metadataPrefix=html&identifier=ADA385923 To view on-demand presentations from the SEI Agile Research Forum, please visitwww.sei.cmu.edu/go/agile-research-forum For more information about the Networking Information & Technology Research & Development Program, please visit www.nitrd.gov/about/about_nitrd.aspx

By Austin WhisnantMember of the Technical StaffThe CERT Network Situational Awareness Team Knowing what assets are on a network, particularly which assets are visible to outsiders, is an important step in achieving network situational awareness. This awareness is particularly important for large, enterprise-class networks, such as those of telephone, mobile, and internet providers. These providers find it hard to track hosts, servers, data sets, and other vulnerable assets in the network. Exposed vulnerable assets make a network a target of opportunity, or "low-hanging fruit" for attackers. According to the 2012 Data Breach Investigations Report, of the 855 incidents of corporate data theft reported in 2012, 174 million records were compromised. Of that figure, 79 percent of victims were targets of opportunity because they had an easily exploitable weakness, according to the report. This blog post highlights recent research in how a network administrator can use network flow data to create a profile of externally-facing assets on mid- to large-sized networks. Network flow data, which is an aggregation of the header information contained in datagrams (packets), can be used to create profiles of network traffic, detect malicious activity, and determine appropriate traffic prioritization settings. Network flow data includes information about communicating pairs of IP addresses, and the ports and protocols on which they communicate, as well as aggregated byte counts and flags used. Network administrators can use network profiling to consider how decisions about configuration changes will affect the rest of the assets on their network. Security administrators can evaluate the profiles to identify assets that violate policy and suspicious activity, while business administrators can use the profiles to help guide long-term decisions regarding network security. The intent of this research by the CERT Network Situational Awareness Team was to create a step-by-step guide for using network flow to inventory or profile a network that includes thorough explanations of why certain steps were chosen so that administrators could understand the process and tailor the steps for their environments. We on the research team focused our analysis on creating a profile of externally facing assets on mid- to large-sized networks that serve thousands to hundreds of thousands of users. We used data from a medium-sized enterprise network that allowed us to access its typical data usage. By focusing on network flow data, we had much less data to deal with than if we collected all traffic on a network (full packet capture). Focusing on headers also allowed us to avoid issues with confidentiality and privacy because we were not actually collecting payload information. We then parsed the data we collected using the System for Internet-Level Knowledge (SiLK).  SiLK is an open-source tool developed by the CERT Network Situational Awareness Team that is an efficient network flow collection and storage infrastructure that will accept flow data from a variety of sensors. To produce relevant results, the process we developed for network profiling must complete within a fixed amount of time. For networks with relatively stable assets, this process could take place over one or two months. For fast-changing networks, the process could take place in as little as one to two weeks. By following these steps, a network administrator profiling a network will obtain a list of public-facing assets, information about the ports through which each asset is communicating, and other pertinent information, such as the external IP addresses to which the asset is connecting. Our approach can be broken down into the following steps: Gather available network information. It is important to gather available information about a network prior to beginning the profile. This information helps to define the scope for the rest of the process. Baseline information can include address space, network maps, lists of servers and proxies, and policies governing network design. Even if this information is incomplete or out of date, it still provides a reference baseline.Not everything about the network can be known. Consider conducting a quick assessment or penetration test to develop a network map and a list of exposed services on various machines. Doing so provides a background for the profile. Later, the network maps and lists of servers can be updated for a cycling-through process. Select an initial data set. This step shapes the entire analysis. A representative sample must be large enough to represent typical traffic, but small enough to support an iterative processing of queries. The general guidelines for selecting a sample data set include •    Duration. Start with at least an hour’s worth of data. Add more data up to a day’s worth, until the query time starts to slow.•    Timing. Select the busiest time of day to carve out the most representative network traffic.•    Direction. If the traffic is bidirectional, start by looking at outbound traffic.•    Sampling. Avoid starting with sample data because it may mask important routine behaviors.•    Network size. Consider separating a large IP-bound network into a few independent profiles and merging them after analysis is complete. Identify the active address space. Issues involved in monitoring the address space include whether sensors cover private address space, what traffic is expected on failover circuits during normal operations, and whether a business unit has connected a system without administrator knowledge. The steps involved in identifying and monitoring the active address space include1. Identify hosts that have active Transmission Control Protocol (TCP) connections and those that don’t.2. Identify hosts that generate a non-trivial amount of traffic on protocols other than TCP.3. Aggregate individual hosts into populated network blocks. 4. Examine additional information gathered in step one to confirm the list of active IP address blocks. Catalog common services. After the active hosts on the network have been identified, inventory the services that comprise the majority of bandwidth use and business operations, such as web traffic and email. Once these protocols are inventoried, start working on other services likely to run on the network and visible to instrumentation, including Virtual Private Network (VPN), Domain Name System (DNS), and File Transfer Protocol (FTP). Catalog remaining active assets. The list of assets in the profile thus far should cover almost all network traffic, as well as services that were profiled based on the most frequent services in the network. Prior to profiling any remaining hosts, expand the time frame for the sample data set to see if there are other active hosts that were not represented in the smaller data set. The expanded data set should include at least one month’s worth of data for the most accurate results. While profiling leftovers, note the findings in the profile, even if they are hard to verify or if they seem incorrect. This information can always be reviewed in more detail during further investigation. Maintain the profile. Six months after you create a profile, it may no longer be accurate. A majority of these steps can be automated to address this problem. For example, network flow analysis software may allow for scheduling filters to run weekly or monthly during this process. Automated tools can only do so much, so you need to consistently validate potential assets for a particular service before adding them to the profile. To update the profile at least once a month, either run through the profiling process again or examine trends over time. Report on findings. To increase the impact of the profile, consider adding more data points that pertain to security, including•    machine administrator•    update schedule•    intended purpose Challenges in this Approach Relying only on network flow data to create a network profile is inherently more inaccurate than using full packet capture. As long as administrators are aware of the limitations of using flow data (as explained in the report documenting the guide), useful results can be produced by following the steps in the guide. Future Research Our team is currently examining how administrators customize and use the results of network profiles. We are investigating how to automate these steps and implement a continual update process. We are also considering development of a new step-by-step guide using this same approach with a different tool. For example, ARGUS is another network flow analysis tool with a slightly different approach than SiLK. Please feel free to suggest other tools for investigation in the comments section below. Additional Resources The technical report describing this research, Network Profiling Using Flow, may be downloaded at http://www.sei.cmu.edu/library/abstracts/reports/12tr006.cfm Our approach is based on System for Internet-Level Knowledge (SiLK), an open-source tool developed by the CERT Network Situational Awareness Team. You can download SiLK at http://tools.netsa.cert.org/silk/. The Network Situational Awareness (NetSA) group in the CERT Program at the SEI developed and maintains a suite of open-source tools for monitoring large-scale networks using flow data. These tools have grown out of our work on the AirCERT project, the SiLK project, and the effort to integrate this work into a unified, standards-compliant flow collection and analysis platform. You can view that suite of tools at http://tools.netsa.cert.org/index.html

By Bill ScherlisSEI Principal Researcher and Director, Institute for Software Research Some of the principal challenges faced by developers, managers, and researchers in software engineering and cybersecurity involve measurement and evaluation.  In two previous blog posts, I summarized some features of the overall SEI Technology Strategy. This post focuses on how the SEI measures and evaluates its research program to help ensure these activities address the most significant and pervasive problems for the Department of Defense (DoD). Our goal is to conduct projects that are technically challenging and whose solution will make a significant difference in the development and operation of software-reliant systems. In this post we’ll describe the process used to measure and evaluate the progress of initiated projects at the SEI to help maximum their potential for success. The Importance of Measurement and Evaluation in Software and Cybersecurity Certain characteristics of software and cybersecurity are easy to measure—lines of code produced, errors found, errors corrected, port scan events, and time consumed. But much of what really matters in practice offers an even greater challenge to our measurement and evaluation capabilities. For example How far are we from completion of the project? Are the specifications consistent and feasibly implementable? How much engineering risk is associated with this requirements specification? How resilient is this architectural design with respect to flooding attacks? What kinds of robustness are offered by this application programming interface (API )definition? How many errors reside in the code base, and how many of them are vulnerabilities? How many of these vulnerabilities are easy coding errors, and how many are intrinsic to the architectural design? Is the component or system implementation safe for deployment in a secure environment? Will the control system meet deadlines? Over the years, much of the research activity at the SEI has been focused on enhancing our ability to measure and evaluate. Indeed, the genesis of the Capability Maturity Model (CMM) and its successors in the late 1980’s is based on a need—unmet at the time—to evaluate the capabilities of software development organizations to deliver on their promises. Models such as the Resilience Maturity Model (RMM) and architecture evaluation methodologies serve similar purposes. Much of the body of research work at the SEI is focused on these challenges. And so there is a natural question: How can we evaluation the potential significance of the research, development, testing, and evaluation (RDT&E) work we (and others) undertake to improve technology and practice in software engineering and cybersecurity?  This is a significant issue in all research, development, test, & evaluation (RDT&E) programs. We need to take risks, but we want these to be appropriate risks that will yield major rewards, and with a maximum likelihood. An overly conservative approach to RDT&E management can yield small increments of improvement to existing practices, when what is really needed, in many cases, are game-changing advances. Indeed, in software and cybersecurity, despite the admonition of Lord Kelvin ("If you cannot measure it, you cannot improve it"), purely "numbers-based" approaches can be dangerous (viz. the famous "light under the lamppost" story). Naïve application of management-by-the-numbers is especially damaging in our disciplines, where so many important measures are lacking. On the other hand, bold "open loop" approaches can yield risks with little likelihood of benefits. We must therefore employ a combination of measures, expert judgment, and careful identification of evaluation criteria—and we must structure the process of research to obtain early indicators and develop them when necessary (i.e., build new lampposts). This multidimensional evaluation process is essential because, in research, it is not just a matter of risk versus reward—thoughtful management in the definitional phases of a project can enable us to "push the curve out," simultaneously mitigating risk, identifying or developing meaningful measures in support of evaluation, and increasing potential for success and impact. A more general discussion on research management practice is the subject of chapter 4 of the 2002 National Academy report on Information Technology Research, Innovation, and E-Government. Heilmeier’s Catechism In this vein, more than two decades ago, George Heilmeier, former director of the Defense Advanced Research Projects Agency (DARPA), developed a set of questions that he posed to prospective leaders of research projects. If these questions can be well addressed, then the proposed projects were more likely to be both successful and significant. These questions have been so widely adopted in research management that they have become known as the "Heilmeier Catechism." Experience at DARPA and other innovative research organizations has shown that compelling answers to the following questions are likely to predict successful outcomes: What are you trying to do? Explain objectives using no jargon. How is it done today? What are the limits of current practice? What's new in your approach? Why do you think it will be successful? If you're successful, what difference will it make? To whom? What are the risks and the payoffs? How much will it cost? How long will it take? What are the midterm and final "exams" to assess progress? (A more detailed presentation of these questions appears in the National Academy Critical Code report, pages 114 - 115.) Our work at the SEI spans a spectrum ranging from deep, technically-focused research on the hard problems of software engineering and cybersecurity—the technology, tools, practices, and data analyses—to more operationally-focused efforts supporting the application of diverse technical concepts, methods, and tools to the challenges of practice, including development and operations for complex software-reliant systems. To ensure we help the DoD and other government and industry sponsors identify and solve key technical and development challenges facing their current and future software-reliant systems, we have adapted Heilmeier’s Catechism into an evaluation process that is more adapted to the particular measurement challenges in software and cybersecurity.  This evaluation process—which we summarize below—involves a combination of four factors: Mission relevance and potential significance of a proposed scope of work to the mission effectiveness of DoD, its supply chain, and other critical sectors Field significance, or capability to evaluate mission effectiveness of emerging solutions to DoD and other stakeholders, including the development of early indicators of mission significance Technical soundness of the research undertaken, based on accepted scientific approaches Technical significance and innovation of the work, for example according to the standards of quality and significance evident in relevant top publication venues Ensuring Validity The wide spectrum of activities in the SEI body of work motivates us to contemplate four different dimensions of validity in assessing proposals for research projects from SEI technical staff: Mission relevance. Are we working on the right problem? Means to enhance validity related to mission relevance include challenge problem workshops, along with early and ongoing collaboration with knowledgeable, mission-savvy stakeholders. This is analogous to the agile practice of involving the customer early and often; it dramatically lessens risk that we are solving the wrong problem. The SEI also benefits from having technical staff members with extensive operational experience in the development, sustainment, and modernization of systems, as well as in a wide range of cybersecurity mission activities. This experience, along with direct engagement with operational stakeholders, enhances validity related to mission relevance. Field significance. Will our solution have the intended impact when it is matured and fielded? Early indicators of field significance include the use of models and simulation, surrogate trials, and field exercises, as well as identification of metrics and validated evaluation criteria and perhaps, most importantly, collaboration with partners who can assist in appraisals, including test and evaluation organizations. Development of such early indicators can be very challenging and can require explicit investment in trials, exercises, and, development of surrogates. Technical soundness. Does our approach embody a sound scientific process that will lead to scientifically valid results? This dimension of validity forces attention to experiment design, mathematical soundness, measures and metrics, statistical significance, and other technical factors that influence the scientific acceptability of outcomes. Technical soundness of published results is generally validated through peer review. The best early predictors of sound outcomes relate to the caliber and experience of the technical staff and their research collaborators. Technical significance. Is our technical approach well situated in the literature or practice of the discipline, and will results be recognized as significant and scientifically impactful? The best evidence of technical significance derives from direct evaluation by recognized peer expert. For publishable results, this can come in the form of peer review and publication in the top venues, best-paper awards, keynote invitations, recognition from professional technical societies, and other types of peer recognition. Finally, we consider the use and development of metrics, which contribute generally both to our ability to evaluate research and to the advancement of practice generally in software and cybersecurity. Much of the research we do is focused on developing and validating new metrics. But, additionally, when we develop metrics useful in research, we find they can also have broad value in practice. In the management of research projects, there are four dimensions we consider: Success criteria.  How can we test for overall soundness, significance, and impact when a project reaches completion?  What are the observable features of final project results, and who must be involved in making an assessment?  Which of the observables are indisputably objective, and which require expert judgment? It is important to note that there is nothing wrong with the use of expert judgment to make evaluations of success and impact, especially when the alternative is a set of inaccurate and incomplete objective measures. Performance criteria.  What are early indicators and tracking mechanisms to monitor progress (the so-called "mid-term exams")? What are the various dimensions of performance, and how are they assessed? As with the success criteria, these can involve a combination of objective measures and expert judgment.   User inputs and feedback. Who are the early collaborative stakeholders, and how are they selected? What are the mechanisms to get feedback from them? This is an explicit criterion, because of the importance of expert judgment and stakeholder acceptance of much of the new work in software and cybersecurity. Metric development. Where are there dark shadows where the light of measurement is needed?  How will the project advance measurement capability as part of its work? At the SEI, this is a first-class activity for two reasons. First, the development of new metrics can have broad and deep impact on practice. For example, metric development (or, more generally, development of evaluation capability) is an explicit feature of some of the challenges mentioned in previous blog posts related to software assurance, cost-re-estimation, and architectural resilience. Advances in Metrics As noted at the outset, measurement is, in some ways, the deepest challenge in both software engineering and cybersecurity. For example, how can we measure the security of a system or the flexibility of a software framework? Likewise, how can we develop useful subsidiary measures that can help us understand the flow of data and information within a large system, or the extent of coupling among a set of federated software components?  This challenge is not only technically deep but profoundly significant to the advancement of practice. Particularly when earned value must be evaluated, the Architecture-Led Incremental Iterative Development (ALIID) approach depends on advances in metrics. Described in the first post in this series, ALLID (or, "agile at scale") enables iterative and incremental development of highly capable and innovative systems with acceptable levels of programmatic risk. Earned value based purely on accumulation of lines of code is unlikely to lead to project success, when much of the risk mitigation (accomplished through prototyping and modeling-and-simulation, for example) leads to reduced variances in estimates, though not necessarily reduced mean values. That is, the estimates become more predictive and reliable as a result of these kinds of activities, and thus greater levels of innovation and engineering-risk taking can be supported without a necessary threat to the potential for success of the overall effort. This is the essence of iteration and incrementality. Effective metrics can have a profound impact on the industry—witness the CMM and subsequent CMMI families of measures, just successfully transitioned out of the SEI into a separate activity at Carnegie Mellon. This is why we take as a principle that the development of new measures must be a first-class activity in the research portfolio. Unlike the work done decades ago, our disciplines are now data rich, and we can build on data collection, models and simulation, field tests, and other analytic approaches that, just a few years ago, were relatively less feasible. We see this reflected in the research portfolio at the SEI, which is moving more aggressively than ever to address these challenges and create rich possibilities for the development and operation of software-reliant systems for the DoD, other mission agencies, and their supply chains. In this series on the SEI Technical Strategic Plan, we have outlined how we are meeting the challenges of designing, producing, assuring, and evolving software-reliant systems in an affordable and dependable manner. The first post in this series outlined our strategy for ALLID or agile-at-scale, combining work on process, measurement, and architecture. The second post in the series detailed our strategies for to achieve game-changing approaches to designed-in security and quality (evidence-based software assurance). It also identified a set of DoD-critical component capabilities relating to cyber-physical systems (CPS), autonomous systems, and big data analytics. Finally, it outlined our approach to cybersecurity tradecraft and analytics. I am also very excited to take this opportunity to introduce you to Kevin Fall, who joined the SEI late last month as its new chief technology officer. In this role, Dr. Fall will take over responsibility from me for the evolution and execution of the SEI Technical Strategic Plan, as well as direction of the research and development portfolio of the SEI’s technical programs in cybersecurity, software architecture, process improvement, measurement and estimating, and unique technical support to sponsors. I will continue to take an active role in supporting the SEI and its essential mission—even including occasional future blog posts! Kevin will join the blogger team with a blog post in the near future summarizing his vision for research and transition activities at the SEI. Additional Resources We expect that the SEI Strategic Research Plan will be soon be available for download.

By Grace Lewis Technical Lead, Edge-Enabled Tactical Systems Research In 2011, Col. Timothy Hill, director of the Futures Directorate within the Army Intelligence and Security Command, urged industry to take a more open-standards approach to cloud computing. "Interoperability between clouds, as well as the portability of files from one cloud to another, has been a sticking point in general adoption of cloud computing," Hill said during a panel at the AFCEA International 2011 Joint Warfighting Conference. Hill’s view has been echoed by many in the cloud computing community, who believe that the absence of interoperability has become a barrier to adoption.  This posting reports on recent research exploring the role of standards in cloud computing and offers recommendations for future standardization efforts. Avoiding Vendor Lock-In Since the inception of the cloud, organizations have been transferring data and workloads to pooled, configurable computing resources. These resources include networks, servers, storage, applications, and services. One concern voiced by many organizations that use cloud-based services is vendor lock-in, which stems from the inability to move resources from one cloud provider to another. Users want to have the freedom to move between cloud providers for many reasons. For example, a relationship with a vendor may not be working, service-level agreements may not be met, other providers offer better prices, or a provider goes out of business. In an environment without common standards, there is little or no freedom to move between vendors. The cloud computing community has already developed numerous standards (some argue there are too many) by various forums, standards organizations, and nonprofit organizations including OpenStack, the Standards Acceleration to Jumpstart Adoption of Cloud Computing, and The Open Group Cloud Work Group, to name a few. One issue explored in my research is whether we should create new standards or just leverage existing standards. Some standardization efforts focus on codifying parts of a cloud-computing solution, such as workloads, authentication, and data access. Other standards focus on unifying disparate efforts to work together on a solution. In addition, due to their large market share in this space, interfaces used by Amazon have emerged as de facto standards. The technical report describing my research, The Role of Standards in Cloud Computing Interoperability, explains how answers to questions about how standards can enable interoperability depend on several factors.  Key factors include the type of service model that a cloud provider uses and the level of interoperability that an organization expects.  Note that the cloud community typically uses the term interoperability to refer to portability, i.e., the ability to move a system from one platform to another. Use Cases Initially, my research identified four typical cloud-computing interoperability use cases that are supported by standards: Workload migration. A workload that executes in one cloud provider can be uploaded to another cloud provider.  Some standardization efforts that support this use case are Amazon Machine Image (AMI), Open Virtualization Framework (OVF), and Virtual Hard Disk (VHD). Data migration.  Data that resides in one cloud provider can be moved to another cloud provider. A standardization effort that supports this use case is Cloud Data Management Interface (CDMI). In addition, even though SOAP and REST are not data-specific standards, multiple cloud-storage providers support data- and storage-management interfaces that use SOAP and REST. User authentication.  A user who has established an identity with a cloud provider can use the same identity with another cloud provider.  Standardization efforts that support this use case are Amazon Web Services Identity Access Management (AWS IAM), OAuth, OpenID, and WS-Security. Workload management. Custom tools developed for cloud workload management can be used to manage multiple cloud resources from different vendors.  Even though most environments provide a form of management console or command-line tools, they also provide APIs based on REST or SOAP. We found that workload migration and data migration can benefit the most from standardization.  For example, standardization of VM (virtual machine) image file formats would allow organizations to move workloads from one provider to another or from private to public clouds. Standardized APIs for cloud storage would do the same for data. Provider Service Models In examining the issue of standardization through the provider lens, we looked at the three main service models: Infrastructure-as-a-service (IaaS). IaaS stands to benefit the most from standardization because the main building blocks are workloads that are represented as VM images and storage units, whether type data or raw data. This finding also ties back to the first two use cases identified earlier, which were workload migration and data migration. Platform-as-a-service (PaaS). Organizations that buy into PaaS, do so for the perceived advantages of the development platform. The platform provides many capabilities out of the box such as managed application environments, user authentication, data storage, reliable messaging, and other functionality in the form of libraries that can be integrated into applications. Organizations that adopt PaaS are not thinking only of extending their IT resources, but are seeking value-added features (such as libraries and platforms) that can help them develop and deploy applications more quickly. Software-as-a-service (SaaS). SaaS stands to benefit the least from standardization. SaaS is different from IaaS and PaaS in that it represents a licensing agreement to third-party software instead of a different deployment model for existing resources that range from data storage to applications. Organizations that adopt SaaS are acquiring complete software solutions or services that can be integrated into applications. Organizations select PaaS and SaaS specifically for these value-added features, and end up in a commitment similar what one experiences when purchasing software. Expecting PaaS and SaaS providers to standardize these features would be equivalent to asking an enterprise resource-planning software vendor to standardize all of its features; it’s not going to happen because it’s not in their best interests. Future Research One challenge among standardization organizations is determining what areas of cloud computing to standardize first. In 2005, researchers from the European Union defined three generations of service-oriented systems. The development of cloud-based systems over time is analogous to the following classification of the way that service-oriented systems have evolved First-generation. The location and negotiation of cloud resources occur at design time. Cloud resources are provisioned and instantiated following the negotiation process. Second-generation. The location and negotiation of cloud resources occur at design time. Depending on business needs, however, cloud resources are provisioned either at design time or runtime, and instantiated at runtime. This approach would support, for example, a cloud-bursting strategy in which developers design a system for an average load, but the system can balance its load to a cloud provider when it reaches its full capacity. Third generation. In the third generation of cloud-based systems, the location, negotiation, provisioning, and instantiation of cloud resources occur at runtime. Reaching this third generation of cloud-based systems will most likely be the focus of future research. This work will require cloud consumers, cloud providers, and software vendor groups to work together to define standardized, self-descriptive, machine-readable representations of basic resource characteristics such as size, platform, and API (application programming interface); and  more advanced resource characteristics such as pricing and quality attributes values, negotiation protocols and processes; and billing protocols and processes. For now, standardization efforts should focus on the basic use cases of user authentication, workload migration, data migration, and workload management. Those efforts can then be used as a starting point for the more dynamic use cases of the future. Recommendations Even if vendor lock-in is mitigated, it is important for organizations to know that any migration effort comes at a cost—whether between cloud providers or local servers, databases, or applications.  Cloud standardization efforts should therefore focus on finding common representations of user identity, workload (virtual-machine images), cloud-storage APIs, and cloud management APIs. Vendors influence many standards committees, and it is unrealistic to assume that each of these elements will have a single standard. Agreement on a small number of standards, however, can reduce migration efforts by enabling the creation of transformers, importers, exporters, or abstract APIs. Such an effort could enable the dynamic third-generation of cloud-based systems.  What are your thoughts on cloud computing interoperability? Do we need new standards? Or can we live with the ones we have? Will we ever get to the third generation? Is it necessary? Additional Resources The technical report describing this research, The Role of Standards in Cloud Computing Interoperability, may be downloaded at http://www.sei.cmu.edu/library/abstracts/reports/12tn012.cfm To read all of Grace Lewis’ posts on her research in cloud computing, please visithttp://blog.sei.cmu.edu/archives.cfm/author/grace-lewis

By Julien Delange Senior Member of the Technical Staff Research Technology & System Solutions When a system fails, engineers too often focus on the physical components, but pay scant attention to the software. In software-reliant systems ignoring or deemphasizing the importance of software failures can be a recipe for disaster.  This blog post is the first in a series on recent developments with the Architecture Analysis Design Language (AADL) standard. Future posts will explore recent tools and projects associated with AADL, which provides formal modeling concepts for the description and analysis of application systems architecture in terms of distinct components and their interactions. As this series will demonstrate, the use of AADL helps alleviate mismatched assumptions between the hardware, software, and their interactions that can lead to system failures. There are many well-documented examples of problems in software-reliant systems: In December, Ford Motor Company announced that it would be making software updates to the cooling systems of its hybrid 2013 Ford Escape and Ford Fusion because problems with the original cooling system design caused the vehicle to catch fire while the engine was running. A recent study of a popular automatic external defibrillator (AED) found security flaws in the embedded software and the commercial off-the-shelf (COTS) update mechanism. A cluster of ships attached to the Ariane 5 rocket were lost when the rocket failed to achieve orbit on its maiden voyage. The failure was attributed to an "error in software design" and resulted in a loss of more than $370 million. These types of incidents demonstrate the need to make software more safe and reliable, which motivates our focus on AADL in this series of blog posts. Addressing Software (as well as Hardware) Mismatched assumptions between hardware, software, and their interactions often result in system problems that are caught too late, which is an expensive and potentially dangerous situation to developers and users of mission- and safety-critical technologies.  To address this problem, the Society of Automotive Engineers (SAE) released the aerospace standard AS5506, named the Architecture Analysis & DesignLanguage (AADL). The AADL standard, which was authored by my colleague, Peter Feiler, defines a modeling notation based on a textual and graphic representation that is used by development organizations to conduct lightweight, rigorous—yet comparatively inexpensive—analyses of critical real-time factors, such as performance, dependability, security, and data integrity. AADL models capture both software and hardware components, as well as their interactions, including the association of a software process on a processor or the deployment of connection on a bus. The AADL standard includes abstractions of software, computational hardware, and system components for specifying real-time, embedded and high dependability systems with their software/software concerns and their specific requirements (such as scheduling, bus latency or jitter) systems validating system and ensuring that stakeholders requirements can be achieved While initial AADL efforts were experimental, organizations have been using the AADL standard for nearly a decade. An early adopter was the European Space Agency, which leads the ASSERT project for the design and implementation of safety-critical systems. This project relied on AADL since its inception and project members continue to use it to model, validate, and produce software. From this early use of the system other projects and communities have shown a strong interest in adopting and applying the language. The AADL committee and its members tracked users experiences and, in response, published a new version of the AADL standard in 2009, with a minor revision in September 2012. A Global Proof of Concept In 2008, the Aerospace Vehicle Systems Institute (AVSI), a global cooperative of aerospace companies, government organizations, and academic institutions, launched an international, industry-wide program called System Architecture Virtual Integration (SAVI) to reduce cost/cycle time and rework risk by using early (and frequent) virtual integrations. In addition to the SEI, major players of the SAVI project include Boeing, Airbus, Lockheed Martin, BAE Systems, Rockwell Collins, GE Aviation, federal aviation administration, the U. S. Department of Defense (DoD), Honeywell, Goodrich, United Technologies, and NASA. SAVI was created in response to the realization by aircraft manufacturers that the source lines of code (SLOC) in software-reliant systems was increasing exponentially. Over the past 20 years  the size of aircraft software, measured in SLOC, has doubled every four years. For the next decade, the projected 27.5 million lines of code required are estimated to cost in excess of $10 billion. In addition to the increased software cost, the latest generation of aircraft, such as the Boeing 787s, will be highly interconnected and are projected to produce half of terabyte of data per flight. Coupled with that increasing complexity is a growing reliance on embedded software to meet requirements for key features, such as greater range and comfort on fewer, more economical flights. The intent of SAVI was to pilot a new technology and a new acquisition process based on architectural models rather than paper documentation with multiple dimensions of analysis used throughout the lifecycle. A key tenet of virtual integration is the use of an annotated architecture model as the single source for analysis. For the SAVI proof of concept, an aircraft system architecture was modeled using the AADL standard. This process allowed AVSI to capture integration faults earlier in the development process, and meet increasing demands for safety and economy. As indicated in the graphic below, the National Institute of Standards & Technology estimated in the report The Economic Impacts of Inadequate Infrastructure for Software Testing that more than 70 percent of all system defects are introduced prior to code development, while only a small fraction are detected during that time.  The remaining errors are detected in the testing phase when it’s most expensive to fix. The ability to detect errors and defects prior to implementation efforts therefore reduced development costs and efforts, while enhancing overall system robustness and reliability. The SEI technical report, System Architecture Virtual Integration: An Industrial Case Study, details the SAVI proof of concept and results, which include multi-tier modeling and analysis of an aircraft and its subsystems support for the needs of both system engineers and embedded software system engineers propagation of changes to the model across multiple analysis dimensions maintenance of multiple model representations in a model repository auto-generation of analytical models interfacing of multiple tools distributed team development via a model repository New Directions for AADL A number of organizations actively use AADL in their modeling and analysis efforts. While AADL was originally developed for aerospace and avionics, it is now being used in many other fields, including automotive and medicine. In particular, as more medical devices are adopted and connected together, there is a growing interest in analyzing and verifying them, especially because issues may have a catastrophic impact on patients. The Food & Drug Administration (FDA), in partnership with the Penn Research in Embedded Computing and Integrated Systems Engineering program, plans to use AADL to support the architecture of modeling concerns for the Generic Infusion Pump. One example of a generic infusion pump would be a model for a Patient-Controlled Analgesia Pump. AADL is used to specify system devices, the underlying software artifacts (in terms of tasks, subprograms), their interaction in terms of events or data and the inherent deployment over the execution platform (association with processors, buses, etc.). The language was also used to capture and analyze system behavior (dependencies among events, conditional execution of system functions according to received data, etc.) We are also working to connect the AADL technology to other technologies being developed at the SEI. For example, SEI researchers are creating cost assessments of different iterations of system architectures. This work will explore questions including When you have a defined and established architecture, what is the cost for a new iteration? What is the cost and impact involved in changing an architecture? By describing the runtime aspects of the system’s deployment, can AADL help evaluate the potential impact of a modification of the execution runtime (for example, when upgrading a device)? Future posts in this series on AADL will highlight real-world applications of AADL, which are presented by users during the AADL standard meetings. At every meeting we include user days where users reflect on their experiences with AADL and tool sets that they have developed.  Please let us know your experiences applying ADDL in the comment section below. Additional Resources In September 2012, Peter Feiler co-authored with David P. Gluch the book, Model-Based Engineering with AADL: An Introduction to the SAE Architecture Analysis & Design Language. For more information, please visithttp://www.sei.cmu.edu/library/abstracts/books/978032188894-5.cfm With our user community, the SEI maintains a wiki with all of our research on AADL that is accessible to the public. To access the wiki, please visithttp://www.aadl.info/aadl/currentsite/ To access a site that maintains source code for AADL, where people can come and freely contribute to the OSATE tool set that is supporting AADL, please visithttps://github.com/osate To read the SEI technical report, System Architectural Virtual Integration: An Industrial Case Study, please visithttp://www.sei.cmu.edu/library/abstracts/reports/09tr017.cfm Julien Delange and Peter Feiler will present Design and Analysis of Cyber-Physical Systems: AADL and Avionics Systems, at SATURN 2013. For more information, please visit http://www.sei.cmu.edu/saturn/2013/program/abstracts.cfm

Second in a Series on Readiness Fit Analysis for Adoption of Agile Methods By Suzanne Miller Senior Member of the Technical StaffAcquisition Support Program The adoption of new practices, such as agile or any new practice for that matter, is a task that is best undertaken with both eyes open. There are often disconnects between the adopting organization’s current practice and culture and the new practices being adopted. This posting is the second installment in a series on Readiness & Fit Analysis (RFA), which is a model and method for understanding risks when contemplating or embarking on the adoption of new practices, in this case agile methods. The RFA method helps organizations understand the barriers and enablers to successful adoption that are present when an analysis is performed. The first post in this series outlined the principles of RFA and described the Acquisition Support Program’s work in extending RFA to support profiling and adoption risk identification to organizations that are adopting agile methods. This blog post continues the discussion with a more in-depth dive into one more of the six RFA categories that we have identified. In our work with the Department of Defense (DoD), we often encounter organizations that have been asked by their government program office to adopt agile methods. These are organizations that have traditionally performed an engineering "V" lifecycle and are accustomed to being managed via a series of document-centric technical reviews, which focus on the evolution of the documents that describe the requirements and design of the system rather than focusing on its evolving implementation, as is more common with agile methods. After the program office and contractor are trained, they realize that there is more to an agile approach than frequent, small iterations and daily standup meetings.  Hence, they struggle to adopt agile practices. For example, contractor personnel often aren’t accustomed to working shoulder-to-shoulder with government counterparts, and the transparency of daily task management is uncomfortable. After a short period of time, programs often revert back to a more traditional approach, even though end users gave positive feedback about what was produced during the timeframe that agile methods were used. This common scenario highlights a disconnect in business strategy, values, and management style that often occurs in organizations trying to adopt new practices. These aren’t the only disconnects, but they are representative of what we often see when working with DoD or other regulated organizations trying to adopt agile methods. Adapting the Readiness Fit Analysis (RFA) to Agile The method for using RFA and the profile that supports CMMI for Development adoption is found in Chapter 12 of CMMI Survival Guide: Just Enough Process Improvement. As part of the SEI’s research in the adoption of agile methods in U.S. DoD settings, we have adapted the RFA profiling technique to accommodate typical factors used in RFA and other factors more uniquely associated with the DoD acquisition environment. To date, we have characterized six categories to profile for readiness and fit: business and acquisition (details discussed in the first post in this series) organizational climate system attributes project and customer environment technology environment practices Determining Readiness & Fit Each category of readiness and fit has a set of attributes that can be characterized by a statement representing your expectation if you were observing a successful agile project or organization operating in relation to that attribute. As a reminder, an attribute of business and acquisition is stated as follows:Oversight mechanisms are aligned with agile principles.Oversight is an aspect of acquisition that can either support or disable an agile project. So alignment of the oversight with agile principles provides less risk of oversight being counterproductive.From the title of this method - Readiness and Fit - it would be easy to assume that the only time you could productively use this method would be in the early stages of adoption, when you’re trying to decide if you’re ready to adopt and if the practices you’re considering are a fit for your organization. Actually, we have used models like this at multiple points in the adoption of a new technology or practice. Your certainty regarding the state of the organization in relation to factors represented in the RFA changes from early use of the method (before initial pilots) to later use (after you’ve run two or three releases using an agile method). Your certainty also changes with respect to the importance of a specific factor to your organization’s success. At the beginning of an agile adoption project, we are often uncertain about the state of the organization or the importance of individual factors (such as alignment of oversight practices with our agile practices) to our adoption success. Later in the adoption process, performing an RFA highlights adoption risk areas overlooked during early phases of adoption and areas where we now have more data to help guide us in developing adoption risk mitigation strategies. For example, we may not initially understand that our approach to estimation in a larger organization doesn’t easily accommodate certain agile practices, such as relative estimation. After we have been through one or two pilots, we are likely to understand the effect relative estimation has on our results, and we can develop strategies to help connect the agile estimation practices to those of the larger program. So don’t hesitate to apply RFA principles and techniques at multiple points in your adoption journey. Organizational Climate This blog entry focuses on a portion of a single RFA category: Organizational Climate. This category is one of the longer categories in the set, because misfits in organizational climate— including culture, values, and working principles—are particularly troublesome areas for many organizations pursuing agile adoption. This category covers the internal operations and culture of an organization, which are extremely important in determining fit. The alignment of the organization’s inherent operations to agile concepts and principles will help determine the ease or difficulty of the transition. The organization’s culture is one of most difficult issues to assess for agile adoption readiness. Culture encompasses assumptions about appropriate or inappropriate behavior and values ingrained in the members of an organization. For instance, in DoD organizations the culture is typically plan-driven and hierarchical, with strict command-and-control structures for communication and leadership. An organization adopting agile methods is generally more empirical, collaborative, self-organizing, and cross-functional. The following list specifies a subset of the factors within the Organizational Climate category that must be considered when an organization performs an RFA. This category includes leadership, sponsorship, incentives, values, structures, and organizational-change history. We’ll deal first with leadership, sponsorship, and incentives, which can either be aligned to an agile perspective, or misaligned, which is often a source of behaviors that may appear as resistance to the new practices. Understanding misaligned factors is important in addressing their symptoms proactively. The list includes a tag (a short title that summarizes the statement) and a statement that provides a condition or behavior that would be expected in an organization successfully using engineering and management methods consistent with agile principles as published in the Agile Manifesto. When applying an RFA, you look at the statement, and then consider the "fit" of that statement to your own organization’s behaviors or attitudes. Cascading sponsorship. Sponsor support for the use of agile or lean methods use is explicit and cascading. In particular, the sponsorship doesn’t just emanate from the program manager, it cascades throughout the acquisition chain. In most organizations, a move to agile methods involves new behaviors and different values. This paradigm is a major change in how an organization operates, and it will affect the overall climate. For DoD organizations, the entire acquisition eco-system includes not just the program but outside organizations, such as Certification and Accreditation and Operational Test & Evaluation. Due to policies and regulations, it can be hard to include these parts of the acquisition chain when adopting agile or lean methods. Cascading sponsorship helps alleviate these problems by having sponsors in multiple places within the organization who can model the new values and behaviors, instilling confidence in the people who are actively trying to adopt the new practices. Aligned incentives. Incentives among stakeholders are aligned to reflect agile principles. The fifth agile principle related to the Agile Manifesto states, "Build projects around motivated individuals. Give them the environment and support they need, and trust them to get the job done." Part of this environment and support is the incentive to work in an agile environment. Most traditional organizations have incentives aligned to individuals but not many incentives for teams. For agile or lean environments, the incentives must be aligned to enable the team to succeed, not to make an individual hero. External policy support. Adherence to agile or lean principles is supported by external policies. For example, the National Defense Authorization Act (NDAA) Section 804 promotes an iterative, incremental style of acquiring information systems that includes software-intensive systems. This legislation aids in the adoption of agile/lean methods and provides guidance in policy and regulation. In addition, Section 933 provides a strategy for acquisition and oversight of DoD cyber warfare capabilities, which also points back to section 804 of the 2010 NDAA. These are high-level examples of policy support of agile principles. Within a particular organization, however, there may be an opportunity for guidance that can push individuals and groups into adopting agile methods. For example, the CIO of a particular agency may mandate adoption of agile principles for their organizations. Caution: Using a policy or mandate to FORCE adherence to agile principles is not productive for healthy adoption of new practices. Putting policies in place too early, before the appropriate transition mechanisms are in place, often leads to malicious compliance. Malicious compliance occurs when individuals adhere to the letter of the law so rigidly that the practices can be adopted in an unproductive way. Sponsors understand agile. Sponsors understand and support the difference in roles and business rhythm when using agile approaches. The roles and responsibilities in a traditional acquisition are well documented in DoD policies, regulations, and training documents; in an agile environment they are different and not as easily understood. Sponsors must understand the four tenets of the Agile Manifesto and the 12 underlying principles to enable the necessary business rhythm for an agile development effort. They also must understand the chosen agile practices well enough to understand the role and responsibility implications of the particular practices that have been selected. Reward system supports agile. The organizational reward system supports the team-based reward focus of agile methods. The reward system is closely related to the incentives used on a program. As stated above, agile and lean organizations focus their rewards on team behavior, whereas traditional organizations reward individual behavior. Within the DoD environment, the reward system (via the performance-management system) is structured and not easily changed. This structure may actually interfere with the support and reinforcement of a more agile environment. There are ancillary reward system mechanisms, however, such as public praise, high evaluations, access to training, and certificate programs that may supplement individual-focused performance rewards. One of the key aspects of a successful reward system is understanding the kinds of rewards that individuals actually value (many teams would value an extra day off more than a gift certificate, for example). Senior support for agile. Senior stakeholders openly and explicitly support the use of agile or lean methods in the program. Successful agile implementations have consistently had a champion. The champion may or may not be a senior stakeholder, but it is someone who has the respect of adopters and the support of senior leadership in the organization. This status will help protect fledgling agile projects from being derailed by those who do not understand the new methods or are uncomfortable with change. Open and explicit support by the senior stakeholders also means that old behaviors are no longer rewarded. This factor is often one of the hardest for senior stakeholders to consistently practice when sponsoring change.  Looking Ahead As you can see, the list of factors in the Organizational Climate category is lengthy, but these factors often need the most attention in promoting a successful adoption of agile methods and principles. As stated at the beginning, however, this is just one category out of six, and only the first five factors within that category. Each category offers insight into the adoption risks that a particular organization will face when adoption agile methods. Identifying these risks is an important first step toward planning and executing mitigation strategies to address them. This series of posts describes how RFA has been used for multiple technologies and sets of practices (most notably CMMI) to help organizations in the DoD and other regulated environments mitigate agile adoption risks. In the first post in this series, we addressed Business & Acquisition, and in this post we introduced Organizational Climate. The next post in will address the remaining factors in Organizational Climate. Subsequent posts will address one or more of the remaining RFA categories: System Attributes, Technology Environment, Project and Customer Environment, and Practices. Additional Resources A 2011 SEI technical note, Agile Methods: Selected DoD Management and Acquisition Concerns, outlines many of the organization climate issues that arise in agile adoption in the DoD. To download the report, please visithttp://www.sei.cmu.edu/library/abstracts/reports/11tn002.cfm  

By David SvobodaCERT Secure Coding Team This blog post describes a research initiative aimed at eliminating vulnerabilities resulting from memory management problems in C and C++.  Memory problems in C and C++ can lead to serious software vulnerabilities including difficulty fixing bugs, performance impediments, program crashes (including null pointer deference and out-of-memory errors), and remote code execution. The exploitability of memory-management problems in C and C++ (including double-free vulnerabilities) was first demonstrated in 2000 by the security specialist Solar Designer.  Even before Solar Designer wrote a blog post demonstrating this exploit technique, the problems of memory management had been widely acknowledged throughout the C programming community since the initial standardization of C in 1989. This elimination can be achieved by designing and implementing an ownership model for dynamically allocated memory.  The ProblemWhen a program begins execution, it is provided with a region of memory called the runtime stack. This stack grows and shrinks during the program’s lifetime; it contains memory that the compiler determines will be required by the program. But some programs require memory that the compiler cannot measure, usually because the amount of memory required will be known only to the program when it runs. This type of memory is typically managed as a heap, which stores memory dynamically allocated and deallocated by the program.When software developers need to write code requesting more memory, they use a pointer to refer to the new memory allocated from a heap. This pointer becomes the developers’ link to that memory, and developers can use it to copy its contents or accomplish any task they would do with other data. Afterward, the developer can free the memory (i.e., return it to the platform’s heap). After memory has been freed, the program no longer has use of the memory and must not try to read or write to it.This exchange of memory between platform and program works well as long as the program stays within the bounds of the memory blocks provided by the platform. Unfortunately, programs often request hundreds of blocks of memory. Tracking those memory blocks can become a major effort with disastrous consequences should the developer make a mistake.The buffer overflow is a common programming error in which a program writes to an object (usually a string), but the program exceeds the bounds of memory allocated to the object and overwrites adjacent memory. Buffer overflows that occur in the heap are typically not as easy to exploit as buffer overflows in the stack. They can be commonly exploited, however, as Solar Designer showed in his blog post.Some programmers make the mistake of freeing memory twice (double-free errors) or never freeing it at all. Programs that free memory twice have been exploited both in the lab and in the wild. Failing to free memory may not be particularly harmful, but long-running programs that fail to free memory will often continue to request memory until the available memory is exhausted. Memory exhaustion can cause these programs, as well as any other programs on the same platform, to crash when their subsequent requests for more memory are denied.Most modern programming languages (e.g., Java, C#, and Python) prevent these problems by limiting pointer usage. While this can achieve memory safety, it also eliminates a major source of power and flexibility from these programming languages. Developers who require strong control of memory management must use C and C++, despite the memory management errors that often plague programmers who use these languages without being extremely careful about properly freeing their pointers.  The Pointer Ownership ModelThe C language has a fairly simple set of system calls for managing memory. If you want memory from a platform, you would use the malloc() system call. Conversely, if you want to release a block of memory back to the system, you would use the free() system call. C++ has similar mechanisms for allocating and freeing dynamic memory via the new and delete operators. Every program has the difficult task of keeping track of allocated pointers and freeing each pointer exactly once. In deference to the issues with memory management, some developers write their own application programming interface (API) to manage memory—usually in the form of wrappers around malloc() and free(). The evolution of these APIs has produced several well-known memory-management techniques, such as reference counting or garbage collection, to keep track of dynamic memory usage. Many larger software programs, such as Firefox, use one or more such systems to track the pointers that must be freed.Many other programs, however, provide no systematic scheme for tracking allocated pointers, and therefore trust that the programmer "got it right." Our approach, called the Pointer Ownership Model, applies to programs that do not have their own memory management scheme. The Pointer Ownership Model aims to help software developers distinguish pointers that need to be freed from pointers that don’t. This can be accomplished by using static analysis and developer annotations to produce strong safety guarantees. The Pointer Ownership Model partitions pointers in a program into "responsible" pointers and "irresponsible" ones. A responsible pointer must be freed eventually, and an irresponsible pointer must not be freed. The theory behind our research is that within a software program, if all the pointers are divided into responsible pointers and irresponsible pointers, that program can be automatically checked to ensure that every responsible pointer gets freed and every irresponsible pointer never gets freed. This check can be done at compile-time so that the program does not need to be run.Our approach involves designing and implementing an ownership model for dynamically allocated memory performing sound static analysis of the consistency of preexisting C source code and annotations evaluating efficacy by analyzing existing open source programs with known and seeded errors measuring annotations required per thousand source lines of code (KSLOC) To implement our Pointer Ownership Model, we are building two programs: an advisor and a verifier. The advisor takes a file of C source code and builds an ownership model of the pointers used by the code. In other words, the advisor examines the code and determines which pointers are responsible and which are not. If the advisor cannot determine the responsibility status of a pointer, it will ask the user. Once the model is complete, the verifier can then take the model and the source code and indicate if the code complies with the model. If the code does not comply with the model, an error message would be produced. This could happen, for example, if a responsible pointer goes out of scope while never being freed.Related WorkMemory management is an old problem, and there are many tools, free or proprietary, to help programmers tackle it. Most memory debugger tools employ dynamic analysis. They monitor a program as it runs while tracking its use of the heap, and they generate error messages if they detect memory bugs. Valgrind is one popular dynamic memory debugger for Linux systems. These tools accurately report errors; however, they impose a performance hit on program execution and so cannot be used on production code. A few debuggers use static analysis, which does not require the program to be run. Instead they analyze the program’s source code. Because the debuggers do not observe a running program, they can only theorize about what errors may occur, so they can suffer from false positives (reporting an error in a bit of code where there is no error) and false negatives (failing to report a true error in the code). A false positive requires a developer to inspect the code and verify that it needs no change, but a false negative can cause the analyzer to declare the code to be bug-free when it is not. False negatives are considered a far worse problem than false positives, and many static analysis tools err on the side of minimizing false negatives while issuing lots of false positives. Coverity and Fortify both produce commercial static analysis tools, and Splint is a free static analysis tool.The Pointer Ownership Model employs static analysis, but unlike traditional static analysis tools, it requires extra semantic information not provided by traditional C source code. The Pointer Ownership Model simply needs to know which pointers are responsible and which are irresponsible. The programmer must supply this information, although the advisor program should do most of the work. This semantic information means that the Pointer Ownership Model will not be subject to false positives or false negatives. The error messages it generates will be sound, and it will not overlook violations of its model. And, because it does not employ dynamic analysis, it will impose no runtime penalty.CollaborationsTo create the verifier program, I am collaborating with Lutz Wrage, a researcher in the SEI’s Research, Technology, & System Solutions Program. Lutz is building a prototype of the model using the C Intermediate Language (CIL) Framework. The program that we are creating is an open source project, and it needs to contain a C language parser that can build an intermediate representation of a C program. This is essentially the first step performed by a compiler. But our goal is not to build a compiler; it is instead to conduct an in-depth analysis of the C program to enforce our model. Impact to the DoD The verifier program that we are developing will enable the Department of Defense (DoD) to expand its capabilities with the proof-of-concept secure C compiler. It will also influence C language standards and commercial compiler development. We also hope the verifier will provide the DoD and DoD contractors with a secure compiler technology. Accounting for Corner Cases & Other ChallengesThis approach will face several challenges. One challenge centers on developing a sufficiently timely approach whose benefits will outweigh programmer effort. The research must also demonstrate a low annotation effort and low runtime overhead. One of the most difficult challenges in building any program is corner cases, which in this instance would be valid code that we did not account for. The C language is complex and contains constructs that will derail our model. For example, our current model does not handle arrays of responsible pointers, so we disallow them and require that all pointers in an array of pointers be irresponsible. While forbidding responsible pointer arrays keeps the model consistent and simple, it greatly limits its usability. Therefore a future goal of this research is to extend our model to handle arrays of responsible pointers.We have several small code examples that we can run through our model verifier once we have built it. Some of these code examples comply with a consistent model, and others intentionally violate the model. The verifier should produce an error message when handed one of these noncompliant examples. We eventually want to test the verifier on a larger software program so that we can understand the true impact of corner cases on our research. Another challenge will be transitioning our implementation from operating on toy code examples to real-world code. Consequently, one of our research goals is to survey open source software to find out which projects use their own memory management model and which do not. An example of the latter would be a suitable test case for the Pointer Ownership Model.Future Research Our current research posits that our model of pointer ownership can be applied to C programs that do not already provide their own ownership model. We can extend our model in several ways, including handling arrays of responsible pointers handling C++ programs These will be subjects for future research.Additional ResourcesFor more information about the work of the CERT Secure Coding Team, please visit http://www.cert.org/secure-coding/.

By Douglas C. Schmidt Principal Researcher In launching the SEI blog two years ago, one of our top priorities was to advance the scope and impact of SEI research and development projects, while increasing the visibility of the work by SEI technologists who staff these projects. After 114 posts, and 72,608 visits from readers of our blog, this post reflects on some highlights from the last two years and gives our readers a preview of posts to come. First, the numbers. Since the blog was launched in early 2011 the top 10 posts, in terms of visits to the site, are Improving Security in the Latest C Programming Language Standard What is Agile? Strategic Planning with Critical Success Factors and Future Scenarios Fuzzy Hashing Techniques in Applied Malware Analysis The CERT Perl Secure Coding Standard The Importance of Safety- and Security-related Requirements, First of a Three-Part Series Writing Effective YARA Signatures to Identify Malware Improving Testing Outcomes Through Software Architecture Cloud Computing for the Battlefield The Growing Importance of Sustaining Software for the DoD One observation about these top posts is the wide range of topics covered by the blogs, including secure coding, malware analysis, organizational planning, agile software methods, quality assurance, cloud computing at the tactical edge, and software sustainment across the lifecycle.  These topic areas reflect the diversity scope, and impact of the work being done at the SEI. Numbers aren’t the only metric. One goal of the blog when we launched was to provide immediate and accessible insights into the broad spectrum of work we do at the SEI via a two-way "read-write" medium that allows our audience to interact with SEI technologists, rapidly and effectively.  Several posts that sparked dialogue between researchers and readers have highlighted the success of this model, including Robert Nord’s post on Rapid Lifecycle Development in an Agile Context and David Svoboda’s post on the CERT Perl Secure Coding Standard, which drew substantial feedback from coders. Another goal of the blog was to give our audience immediate and accessible insights into our work. One success story in this arena stemmed from a post on fuzzy hashing in applied malware analysis by CERT malware researcher David French. Within a day of its publication, the post was referenced in the blog, Technology Review, which is maintained by the Massachusetts Institute of Technology. A few days after that, a writer for the Tech Republic blog interviewed David French about his research into fuzzy hashing.   While the SEI remains committed to publishing our work in traditional venues, such as peer-refereed journals and conferences, it’s also clear that social media dissemination vehicles like blogs can have tremendous impact in a short time period. Securing the Cyber Infrastructure Many of our posts have focused on securing the cyber infrastructure. The CERT Secure Coding Initiative is conducting research to reduce the number of software vulnerabilities to a level that can be mitigated in DoD operational environments. This work focuses on static and dynamic analysis tools, secure coding patterns, and scalable conformance testing techniques that help prevent coding errors or discover and eliminate security flaws during implementation and testing. The post that brought in the most visitors during the past two years, Improving Security in the C Programming Language Standard by David Keaton, explored two of the changes—bounds-checking interfaces and analyzability—from the December 2011 revision of the C programming language standard, which is known informally as C11 (each revision of the standard cancels and replaces the previous one, so there is only one C standard at a time). Other popular posts in this area highlighted work by SEI researchers who are developing tools to analyze obfuscated malware code to enable analysts to more quickly derive the insights required to protect and respond to intrusions of DoD and other government systems. Their approach, as described in a post by Sagar Chaki, uses semantic code analysis to de-obfuscate binary malware to a simple intermediate representation and then convert the intermediate representation back to readable binary that can be inspected by existing malware tools. A Growing Importance in Software Sustainment The high costs of software sustainment (which account  for 60 to 90 percent of the total software lifecycle effort) are receiving increased attention as the DoD wrestles with the ramifications of sequestration. Over the last two years, we’ve dedicated a substantial portion of this blog space to the importance of highlighting our efforts to help the DoD sustain software more effectively. Mike Phillips wrote a series of posts on efficient and effective software sustainment. The first post highlighted specific examples of the importance of software sustainment in the DoD, where software upgrade cycles need to refresh capabilities every 18 to 24 months on weapon systems that have been out of production for many years, but are expected to maintain defense capability for decades. The second post described effective sustainment engineering efforts in the Air Force, using examples from across the service’s Air Logistics Centers (ALCs). In June 2012, Bill Scherlis also penned a post on software sustainment stemming from a research effort that he led that studied defense software producibility, with the purpose of identifying the principal challenges and developing recommendations regarding both improvement to practice and priorities for research. The post highlighted key findings of the report Critical Code: Software Producibility for Defense, a summary of the results of the three-year research effort conducted under the auspices of the National Research Council (NRC).  I also authored a two-part series on this topic based on my involvement in an Air Force Scientific Advisory Board study on sustaining aging aircraft. The first post in the series, The Growing Importance of Sustaining Software for the DoD, summarized key software sustainment challenges faced by DoD; the subsequent post describes R&D activities conducted by the SEI to address some of these challenges. The second post in the series described key R&D activities conducted by the SEI to address these challenges including work in sustainment R&D, software product lines, Team Software Process, and software architecture. An Interest in Agile While agile methods have become popular in commercial software development organizations, the engineering disciplines needed to apply agility to mission-critical, software-reliant systems are not as well defined or practiced. Perhaps it’s no surprise, therefore, that the category visitors to the blog site have clicked most on is "agile," and one of the most popular posts in that series is Stephany Bellomo’s "What is Agile," which has drawn the second highest number of visitors to the site. A strong reader interest in agile also spurred a series of posts that highlighted presentations made during the SEI Agile Research Forum. This forum brought together researchers and practitioners from around the world to discuss when and how to best apply agile methods in mission-critical environments found in government and many industries.  I wrote a series of posts recapping presentations made at the Agile Research Forum by Anita Carleton, director of the SEI's Software Engineering Process Management Program, and Teresa M. Takai, chief information officer at the Department of Defense, who discussed the SEI's research efforts in Applying Agile at Scale for Mission-Critical Software Systems Mary Ann Lapham, a senior member of the technical staff, who discussed the importance of collaboration with end users, as well as among cross-functional teams, to facilitate the adoption of agile approaches into DoD acquisition programs Ipek Ozkaya, a senior member of the technical staff, who discussed the use of agile practices in strategic management of architectural technical debt  Jim Over, manager of the SEI's Team Software Process initiative, who advocated the building of self-managed teams, planning and measuring project process, designing before building, and making quality the top priority, among other principles associated with applying agile methods at scale My presentation on applying agile methods to common operating platform environments (COPEs) that have become increasingly important for the DoD.  Looking Ahead In December, we published a post on the State of the Practice of Cyber Intelligence from the SEI Innovation Center. The post describes a research initiative aimed at helping organizations bolster their cyber security posture by leveraging best practices in methodologies and technologies that provide a greater understanding of potential risks and threats in the cyber domain.  We will continue to work with the SEI Innovation Center to cover their work on data-intensive scalable computing, which includes the following technical areas: heterogeneous high-performance cloud computing cyber intelligence (tradecraft, capabilities, and prototyping new analysis methodologies) adaptive and autonomous systems analytics/applied machine learning prototype application development data architectures human-information interaction In the coming months we will also be continuing our series on the Architecture Analysis & Design Language (AADL) standard, which provides formal modeling concepts for the description and analysis of application systems architecture in terms of distinct components and their interactions. Future posts in this series on AADL will cover tools and real-world applications highlighting experiences from organizations in the medical device and space domains.  On another front, we will continue writing about exploratory research efforts at the SEI, the outcomes of which will determine future directions at the SEI. Most importantly, we’d like to thank our readers for their feedback and insight over the last two years. We value your insights and would welcome your feedback below on ways we can improve the SEI Blog to better serve our audience. Additional Resources To download the latest SEI technical reports and notes, please visitwww.sei.cmu.edu/library/reportspapers.cfm