By Randy Trzeciak, Senior Member of the Technical Staff The CERT Program According to the 2011 CyberSecurity Watch Survey, approximately 21 percent of cyber crimes against organizations are committed by insiders. Of the 607 organizations participating in the survey, 46 percent stated that the damage caused by insiders was more significant than the damage caused by outsiders. Over the past 11 years, CERT Insider Threat researchers have collected incidents related to malicious activity by insiders obtained from a number of sources, including media reports, the courts, the United States Secret Service, victim organizations, and interviews with convicted felons. From these cases, four patterns of insider threat behavior have been identified: (1) information technology (IT) sabotage, (2) fraud, (3) national security/espionage, and (4) theft of intellectual property (IP). From those patterns, our researchers developed controls that combine technological tools with behavioral indicators to identify employees at risk for committing cyber crimes. These tools and indicators provide those who monitor networks a better warning of potential anomalous behavior. This blog posting—the first in a series highlighting controls developed by the CERT Insider Threat Center—explores controls developed to prevent, identify, or detect IP theft. Motives and Behaviors By analyzing more than 700 insider threat cases, CERT researchers have identified a series of patterns and behaviors based on the motive of the perpetrator and the impact to an organization. For example, of the documented insider threat cases that we analyzed, 84 incidents are categorized as theft of IP in which employees take information with them as they leave to go to work for a competing organization, use the information to get a job with a competitor, or start their own competing company. In approximately one third of the theft of IP cases in the CERT database, the insider took the information and gave it to a foreign organization or government. An interesting finding emerged when the researcher analyzed these cases: the majority of the insiders (approximately 70 percent) who steal IP do so within 30 days of announcing their resignation. This window gives an organization an opportunity to detect potential malicious activity.  Many organizations do not have the resources to alert and investigate everytime a document is sent off of the network, so this window may allow for focused attention during higher risk periods, thereby reducing the high volume of false positives that may be returned via continual data leakage identification. That finding is used when developing the theft-of-IP technical control outlined in this blog. Based on these observations, we constructed a model of employees who steal information. The model takes into account technological variables, social variables, and the relationships between them. By studying the patterns in various cases, we observed how the crimes tend to evolve over time, and the trends we noticed provided the foundation for our model. After our researchers established the model, they narrowed their focus to portions of it where controls may be applied to prevent or detect information leaving the organization’s network. For example, they configured a tool alert on suspicious activity possibly indicating that a departing employee may be stealing intellectual property. An organization can then use an open source, log-aggregation tool to write rules to alert when potential suspicious activity is observed. For example, analysts can write a query in a log-aggregation tool, such as SPLUNK, to flag employees who meet these criteria: Their system accounts were disabled or are scheduled to be disabled in the next 30 days. They are sending email through the organization’s network. Those emails include attachments. Analysts can further refine the SPLUNK tool to focus on employees in that group who have resigned within the last 30 days and are sending emails with attachments from personal email accounts. (Much of the activity will probably be authorized, but the approach allows organizations investigating suspicious activity to gain a better idea of what activity warrants additional investigation.) Our aim with this research is not to create new tools, but rather to allow organizations to configure their existing arsenal so it is more effective at preventing or detecting malicious insider activity. The controls developed by our researchers should be used in addition to existing tools that many organizations already own, including Data loss prevention (DLP) tools. These tools allow organizations to prioritize critical assets, and observe when employees are accessing data and when that data is being sent through the network. Digital rights management (DRM) tools. These tools allow organizations to require that critical data be validated or authenticated against data on its network. For example, if information was taken off a network, it could not be used on another network, and no one would be able to open it up and view it. Security information and event management (SIEM) tools. These tools allow organizations to detect anomalies on networks and networked systems. One example of such an anomaly would be an employee’s login outside of normal working hours using a remote connection.   Telltale Signs In the October 2011 SEI technical note titled Insider Threat Control: Using Centralized Logging to Detect Data Exfiltration Near Insider Termination, CERT researchers Michael Hanley and Joji Montelibano described the controls developed to prevent IP theft. They reported that the primary vehicles for data exfiltration over the network are corporate email systems or web-based personal email services. They therefore concluded that organizations should consider doing the following when trying to prevent, detect, or deter IP theft: Monitor for misuse of web-based personal email services. This mode of exfiltration will be addressed in detail in future research. Monitor for email to the organization’s competitors or the insider’s personal account. Corporate email accounts running on an enterprise-class service contain robust auditing and logging functionality available for use in an investigation or, in this case, a query to detect suspicious behavior. Taking these factors into account, an organization can proceed on an implementation strategy for these conditions on a logging engine. Hanley and Montelibano defined the following implementation outline:     If the mail is from the departing insider    and the message was sent in the last 30 days     and the recipient is not in the organization’s domain     and the total bytes summed by day are more than a specified threshold     then send an alert to the security operator With the time element serving as the root of a query, any of the following could be used to verify the query: an active directory a lightweight directory access protocol (LDAP) directory service partial human resources records badge access status After the query has narrowed the field to all mail sent within a certain timeframe (the 30-day window), the query will next identify mail traffic that has left the local domain namespace of the organization. This constraint flags email messages to recipients in a namespace that the organization has no control over. The next constraint examines the email byte count to identify exfiltrated data. Hanley and Montelibano set a reasonable per-day byte threshold of between 20 and 50 kilobytes to identify whether several attachments or large volumes of text pasted into the bodies of email messages have left an organization’s network on a given day. Future Research Our future research is focusing on verifying that control models are still applicable and on developing new controls to address other modes of insider crime. The next blog post in this series will examine research that developed controls to prevent, detect, or mitigate IT sabotage by insiders. Additional Resources To read the SEI technical note, Insider Threat Control: Using Centralized Logging to Detect Data Exfiltration Near Insider Termination, please visit www.cert.org/archive/pdf/11tn024.pdf. To read the CERT Insider Threat blog, please visit www.cert.org/blogs/insider_threat/.

Part 2: Understanding Success Drivers By Douglas C. Schmidt, Principal Researcher Common operating platform environments (COPEs) are reusable software infrastructures that incorporate open standards; define portable interfaces, interoperable protocols, and data models; offer complete design disclosure; and have a modular, loosely coupled, and well-articulated software architecture that provides applications and end users with many shared capabilities. COPEs can help reduce recurring engineering costs, as well as enable developers to build better and more powerful applications atop a COPE, rather than wrestling repeatedly with tedious and error-prone infrastructure concerns. Despite technical advances during the past decade, however, building affordable and dependable COPE-based solutions for the DoD remains elusive. This blog posting—the second in a three-part series—builds upon the first posting to describe key success drivers for COPEs that proactively and intentionally exploit commonality across multiple DoD acquisition programs. This blog is based on work by researchers at (and associated with) the SEI—including myself, Adam Porter, John Robert, and Mike McLendon—who are investigating how to create and govern COPEs successfully. We have identified the following three classes of drivers that DoD acquisition programs and system integrators must master to improve the odds of succeeding with COPEs: Business drivers: Achieving effective governance and broad acceptance of the economic aspects of COPEs. When the DoD had the resources to acquire and sustain many redundant solutions, it was often hard to motivate the adoption of common software services and capabilities within the acquisition community. Adopting these services and capabilities were perceived as introducing program and technical risks and potentially impeding the ability of system integrators to offer more unique, custom-based solutions that were more expensive and perhaps perceived as more effective. Now that the status quo is no longer economically viable (which has been the case for many years and is now exacerbated in the shadow of sequestration), government and defense industry leadership has renewed interest in COPE initiatives. While this top-down support for COPEs is welcome and necessary, it is insufficient if program managers and system integrators do not fully accept the need to adopt new business models.Because both government and industry are significantly affected by new business models, they must devise collaborative, socio-technical ecosystems where participants share the risks and rewards of COPEs. One promising approach is managed consortia, which provide a solid commercial and legal foundation for forming and coordinating COPE government and industry stakeholders. These consortia are even more effective when they yield interoperable, open standards that are implemented by multiple open- and closed-source suppliers.Two often overlooked COPE strategy components are policy and governance, which are essential to success of collaborative approaches. These components are often not addressed explicitly because they are often not perceived as being important and are "organizational messy." DoD acquisition policy and guidance must emphasize COPE as an acquisition business and technical strategy at all levels within the acquisition and sustainment community. The collaborative environment also demands that concepts, structures, and processes for governance be an integral component of the overall COPE strategy. DoD program offices also need to work closely with system integrators to ensure that proper contracting models are adopted to incentivize cost-effective, on-time delivery of innovative and integrated solutions. While COPE based technical solutions may be feasible and desirable, these solutions cannot be achieved unless the government first puts in place the proper contract models to appropriately incentivize technical and program behaviors consistent with the government’s business goals. Effective contract models can also be the means to enable rapid-delivery order execution can help streamline technology insertion. In addition, the government must negotiate necessary licenses and data rights, technical data on verification and validation facilities, and procedures to decrease total ownership costs over program life cycles by retaining access to key software and documentation artifacts throughout the development and sustainment phases.   Management drivers: Ensuring effective leadership and guidance of COPE initiatives. While it has become fashionable to pay lip service to the goals and benefits of COPEs, it is much harder to find program managers and senior acquisition executives who can successfully sell and defend the near-term investments in time and effort needed to achieve the long-term payoffs of COPEs. These leaders must not only recognize the strategic role of software in DoD systems, but they must also articulate this role in ways that resonate with congressional appropriators, authorizers, and their staffs.Technical and acquisition leaders should also be savvy and avoid placing their bets on technological "silver bullets" and fads. They should also manage the application of modern iterative and incremental methods at scale, as opposed to traditional waterfall methods. COPEs are most effective when they are developed with strong feedback loops between developers of the reusable COPE infrastructure and developers of applications that use the COPE. Since COPE efforts rarely have the time or resources to please all customers, it is important for managers to be goal-directed—rather than exhaustive—when determining which common assets to develop and sustain. Without continual interactions with application developers, therefore, software artifacts produced by COPE developers rarely address core business problems and thus will not be reused effectively.COPE technical managers must also know when to build and when to buy reusable software platforms and tools. Managers who cling tenaciously to particular platforms or tools, and who ignore all other options, typically trade off short-term progress for long-term pain. A more effective, long-term approach involves working with open standards and establishing affiliations with industry standard groups to ensure continuity across the COPE life cycle.   Technical drivers: The foundations of COPE development. The operational and programmatic success drivers for COPEs often garner the most attention because they fundamentally depend on people, who represent the most complicated and demanding part of socio-technical ecosystems. Even if we could magically solve these vexing challenges, there are still many technical drivers that influence the success or failure of COPEs.To start with, developing a successful COPE requires a clear architectural vision. This vision should be codified and documented by experienced software and system architects who possess a deep understanding of the canonical patterns and architectural styles of the domain(s) associated with a COPE. Other key elements associated with achieving an architectural vision for COPEs include developing open reference implementations for key parts of a COPE infrastructure to help DoD programs avoid getting locked into proprietary solutions adopting effective licensing models to ensure broad adoption and commercialization of COPE components ensuring a strong connection with R&D communities in software engineering and systems engineering to help mitigate technical risks Having a strategy for mitigating technical risks is particularly important for new and planned systems. Although the DoD and software R&D communities have some knowledge about foundational patterns and architectures for legacy DoD systems, they are less aware of key patterns and architectural styles for emerging systems, particularly net-centric systems-of-systems. Unfortunately, there are too few designated software and system architect positions in the acquisition community and program offices to insure an architecture focused vision is a driving foundational and life cycle technical imperative.DoD COPEs necessarily comprise a wide range of network, hardware, and software configurations; different algorithms; and different security profiles. This variation is a key driver of total ownership costs because it affects the time and effort required to assure, optimize, and manage unique system deployments and their unique and often multi-configurations throughout the life cycle. To manage this variation effectively, the SEI helped pioneer software product lines (SPLs). SPLs have been applied in COPEs to manage software variation while reusing large amounts of code that implement common features within a particular domain. SPL-based COPEs help reduce software development and sustainment costs by maintaining and validating reusable components in a common repositoryOther technical drivers associated with successful COPEs include (but are by no means limited to) the following: domain-engineering and use-case analysis methods that elicit and document COPE commonality and variability requirements and software architectures iterative and incremental life cycle methods, processes, and toolkits that help developers better plan, measure, and improve software producibility so they have better confidence in COPE quality and cost estimates software frameworks that codify the expertise needed to implement COPEs in the form of reusable algorithms and extensible and/or reusable component implementations software patterns that codify expertise needed to design COPEs in the form of reusable architecture themes and styles, which can be reused even when algorithms, components implementations, or frameworks cannot commercial-off-the-shelf component-based and service-oriented middleware that codifies expertise needed to develop COPEs in the form of portable open standard interfaces, interoperability protocols, and reusable building blocks COPE-specific middleware components and services that provide APIs and data models via a simpler facade that shields applications from the powerful (and complex) capabilities of the underlying domain-specific middleware frameworks higher-level languages, analysis tools, model-driven engineering technologies that enhance the productivity of COPE application developers and support "correct-by-construction" generation of software artifacts automated verification and validation methods, standards conformance test suites, and system execution modeling tools that leverage the powerful, commoditized computing resources in a distributed, continuous manner to improve persistent quality attributes of COPEs, ensure portability and interoperability, and assure key functional and performance attributes Succeeding with COPEs This blog posting just scratched the surface of the technical and non-technical issues associated with developing and sustaining COPEs for the DoD. In our experience working with many COPE initiatives over the past two decades, achieving success requires a multi-dimensional perspective to foster effective COPE ecosystems and leverage key linkages between the success drivers identified above. Organizations implementing COPE initiatives that address these drivers in a thorough and holistic manner thus have a fighting chance to succeed. Many challenges persist, however, as evidenced by the relatively few COPE success stories to date for the DoD. History shows that organizations that do not understand (or do not execute) these drivers properly will fail, often at great expense and great detriment to the warfighter. My next blog posting describes our work at the SEI on a COPE maturity model to help military and commercial organizations assess and improve their progress in developing and adopting systematic software reuse approaches for DoD acquisition programs. We welcome your feedback in the comments section below with suggestions on how the DoD can improve the technologies and ecosystems needed to develop COPEs more effectively. Additional Resource: To read the SEI technical report, A Framework for Evaluating Common Operating Environments: Piloting, Lessons Learned, and Opportunities, please visitwww.sei.cmu.edu/library/abstracts/reports/10sr025.cfm?DCSext.abstractsource=SearchResults

By Randy Trzeciak Senior Member of the Technical StaffThe CERT Program According to the 2011 CyberSecurity Watch Survey, approximately 21 percent of cyber crimes against organizations are committed by insiders. Of the 607 organizations participating in the survey, 46 percent stated that the damage caused by insiders was more significant than the damage caused by outsiders. Over the past 11 years, researchers at the CERT Insider Threat Center have documented incidents related to malicious insider activity. Their sources include media reports, the courts, the United States Secret Service, victim organizations, and interviews with convicted felons. From these cases, CERT researchers have identified four models of insider threat behavior: (1) information technology (IT) sabotage, (2) fraud, (3) national security/espionage, and (4) theft of intellectual property (IP). Using those patterns, our researchers have developed network monitoring controls that combine technological tools with behavioral indicators to warn network traffic analysts of potential malicious behavior. While these controls do not necessarily identify ongoing cyber crimes, they may identify behaviors of at-risk insiders that an organization should consider for further investigation. This blog posting, the second in a series highlighting controls developed by the CERT Insider Threat Center, explores controls developed to prevent, identify, or detect IT sabotage. Existing technical tools can be better configured to prevent instances of IT sabotage. Many organizations deploy Data loss prevention (DLP) tools and digital rights management (DRM) tools to try to stop theft of IP, or security information event management (SIEM) tools to mitigate IT sabotage. These tools are able to detect and examine network traffic, but determining the difference between anomalous and normal behavior remains hard. Behavioral Indicators Prior to IT Sabotage The CERT Program’s research has shown that employees who commit IT sabotage typically exhibit certain behavioral indicators prior to the crime. These usually begin with an employee’s unmet expectations of the organization, precipitated by a negative workplace event, such as being passed over for a promotion, failure to receive a raise or bonus, or demotion.  Next, the employee becomes disgruntled and seeks revenge against the organization for a perceived injustice. Some behavioral indicators that may be observable in IT sabotage cases are performance problems, conflicts with coworkers or supervisors, outbursts in the workplace, and tardiness. The situation escalates to a point where the disgruntled employee sets up an attack using technical means. If such insiders have been denied access to the organization’s network, they often find ways to regain access (such as exploiting an unknown access path) to deploy their malicious code and then leave the organization or are terminated. The impact to the organization tends to become visible only after the insider’s departure. Using the Security Information and Event Management (SIEM) Signature The following SIEM signature can be used to determine the identity of individuals engaging in behaviors that an organization should consider investigating further, what remote connection protocol they are using, and whether this activity is occurring outside normal working hours. The signature is based on the following key fields: username, VPN account name, hostname of the attacker, and whether the attacker is using SSH, Telnet, or RDP. The characteristics of insider attacks include remote access to the organization’s information systems, outside normal working hours. Given these characteristics, we developed following signature: Detect <username> and/or <VPN account name> and/or <hostname> using <ssh> and/or <telnet> and/or <RDP> from <5:00 PM> to <9:00 AM> Note: This signature should only be applied to individuals who warrant increased scrutiny. This signature should not be applied to all privileged users because it will generate inordinate false positives. Two standards were used to create the SEIM signature: the Common Event Format (CEF) and the Common Event Expression (CEE): The Common Event Format (CEF) is an event interoperability standard developed by ArcSight. The purpose of this standard is to improve the interoperability of infrastructure devices by instituting a common log output format for different technology vendors. It assures that an event and its semantics contain all necessary information. Using this standard and the key indicators identified during the database analysis, we developed two CEF-based SIEM signatures, for Microsoft and Snort products, to identify suspected attackers. The Common Event Expression (CEE) architecture defines an open and practical event log standard developed by MITRE. Like CEF, the purpose of CEE is to improve the audit process and users’ ability to effectively interpret and analyze event log and audit data. It standardizes the event-log relationship by normalizing the way events are recorded, shared, and interpreted. Using the CEE format, we developed a signature based on the key indicators of insider IT sabotage. The signature identifies a suspected attacker who is using a remote connection to log onto the organization’s internal system outside normal working hours, and it also logs the time the event was recorded. Recognizing these behaviors has allowed us to create rules for when to apply a SIEM signature to detect insiders at risk of committing IT sabotage. By applying a SIEM signature, network traffic analysts can detect changes in configuration and changes in timing of network connections and specifically look at people who log in to the network outside of normal working hours. Using nontechnical indicators with the signature also helps to minimize the number of false positives. By combining behavioral and technical aspects, the SIEM signature can be used to help organizations act proactively, not reactively, to protect themselves. Future Research We are not advocating that organizations "advertise" the controls in an attempt to dissuade disgruntled employees from harming the organization.  Instead, we want to persuade organizations to improve the communication between human resources, managers, and co-workers to identify potential disgruntled employees and apply additional IT Controls (including the SEIM signature) to identify potential suspicious changes to critical files. Our future work includes enhancing the CERT insider threat database by collecting incidents, verifying that the behavioral model is still current and applicable, and customizing the model to create more controls. We will continue to use our insider threat lab to test tools, develop controls, and make better recommendations for existing or new configurations of tools to prevent, detect, or respond to malicious attacks on a network. Additional Resources To read the technical report Insider Threat Control: Using a SIEM Signature to Detect Potential Precursors to IT Sabotage, please visit www.cert.org/archive/pdf/SIEM-Control.pdf. To read the technical report Using Centralized Logging to Detect Data Exfiltration Near Insider Termination, please visit www.cert.org/archive/pdf/11tn024.pdf. To read more about the CERT Program’s Insider Threat research, please visit www.cert.org/insider_threat/. To read about the new book The CERT Guide to Insider Threats by Dawn Cappelli, Andrew Moore, and Randy Trzeciak, please visit www.sei.cmu.edu/newsitems/insider-book.cfm. To read the CERT blog post Insider Threat Control: Using an SIEM signature to detect potential precursors to IT sabotage, please visit www.cert.org/blogs/insider_threat/2012/01/insider_threat_control_using_a_siem_signature_to_detect_potential_precursors_to_it_sabotage.html.

By Marc Novakouski, Member of the Technical StaffResearch, Technology & System Solutions Our modern data infrastructure has become very effective at getting the information you need, when you need it. This infrastructure has become so effective that we rely on having instant access to information in many aspects of our lives.  Unfortunately, there are still situations in which the data infrastructure cannot meet our needs due to various limitations at the tactical edge, which is a term used to describe hostile environments with limited resources, from war zones in Afghanistan to disaster relief in countries like Haiti and Japan. This blog post describes our ongoing research in the Advanced Mobile Systems initiative at the SEI on edge-enabled tactical systems to address problems at the tactical edge. At the tactical edge, the people that need the information the most—warfighters, first responders, or other emergency personnel—depend on timely and valuable information to perform their tasks, or even survive.  Unfortunately, the access to the information they need can be extremely hard to achieve, for the following reasons: information overload stemming from too much information, coupled with an inability to locate truly vital information information obscurity due to a lack of awareness of the available information, aka "you don’t know what you don’t know" resource scarcity manifested as insufficient bandwidth, central processing unit (CPU) power, battery power, or even attention span to get the needed information and continue to process, exploit, and disseminate it for as long as needed The remainder of this posting describes how we are tackling the information overload and information obscurity aspects of this problem by developing context-aware mobile applications. A Different Approach to Context-Aware Mobile Applications Context awareness in the mobile environment is not a new field of research.  Most mobile devices come preloaded with applications that use location or time to account for user context. There is certainly no shortage of similar applications available for download.  We decided, therefore, to explore alternative sources of data that would not only push the limit of what could be done with user context, but also focus on the extremely challenging environment at the tactical edge. Our "eureka" moment came when we realized that when warfighters or first responders are at the tactical edge, they are almost never operating alone.  As a result, the most important contextual information to warfighters or first responders is the context of the people in the group, and how they relate to that context.  This realization drove us to explore group context-aware mobile applications.  These applications would, if built correctly, first consider individual user context and then relate that information to the group context, thereby helping users understand both their own state, as well as the state of the group in which they participate. Group context-aware mobile applications clearly have value at the tactical edge.  For example, warfighters are well served by having access to positions of friends and foes on the battlefield (position data being a simple case). They could also benefit from supportive applications that monitor resources, such as food, ammunition, or vital signs.  With sufficient data and processing power, these applications could even use historical trends to determine dynamically if a squad is walking into a possible ambush situation. In less deadly (yet still hostile) environments, such as tsunami disaster areas, the ability to share information about resource needs, dangerous situations, or health emergencies in a structured way can also be valuable.  Such applications could tailor information to managers, construction workers, doctors, and other emergency personnel to help coordinate an effective emergency response. An extensive literature review on context awareness yielded relatively little research on the topic of group context.  Much of the prior work cites the basic context model developed by Anind Dey, but does not expand the model past the individual, choosing instead to tailor the model to a particular domain.  Our research project, called Information Security to the Edge (ISE), explores the structure, applications, and implementation of a context model that includes group information.  We have constructed a prototype application on the Android platform that implements the essential components needed by group context-aware mobile applications, as discussed next. App Architecture - Logic and Data The ISE prototype application follows the common model-view-controller (MVC) pattern, which decomposes an application into the following parts: The model is the data. This data is the information processed by the application.  For example, the words typed by the user into a word processing application are data. The view is the user interface. In the case of a word processing application, the view would be the buttons, menus, scroll bars, and other visual effects provided by the application to help a user write a document.  The controller is the logic. In the case of a word processing application, the controller would be the rules the application uses to save, present, filter, and otherwise modify the text.  The function provided by each button or menu item can also be part of the controller. Consistent with the MVC pattern, the ISE prototype has a central control mechanism (which forms the "brains" of the application) that manages data flow through the application.  In practice, this means that the central controller coordinates data flow and processing through the following primary application elements: The context engine is the central processor for all context information used by the application.  As device sensors report new data and applications on external devices send data to the local application, all data is passed through the engine so that new events are detected as they occur.  For example, if an external user sends their GPS coordinates that indicate they are within 100 feet of a warfighter, then the device can alert the warfighter to their presence.  Expanding on this concept, if a group task must be performed but everyone is working individually on their own tasks the local device can monitor task status and user position and report to the leader when all group members are ready and close by so the group task can be performed. The sensor manager accepts data from sensors that reside upon the mobile device.  A typical smart phone contains position sensors, movement sensors, and in some cases, light and proximity sensors.  The application captures data from these sensors and passes it through the sensor manager.  The sensor manager enables the sensors and controls their sample rate, so that the application can tailor usage to the situation and avoid overwhelming the system. The communications manager acts as the gateway to all external communications within the system.  This gateway currently includes Bluetooth and TCP/IP communications, but can be expanded to include other communication mechanisms that are available to the device.  Any messages to and from users on other devices are passed through the communications manager. The sensor and communications manager architecture consolidates all sensor and communication concerns into a single location.  This consolidation approach enabled us to build a standardized interface that simplifies integration an arbitrary sensor (for example, a radiation sensor) or an arbitrary communication mechanism (for example, a line-of-sight radio that communicates with UAVs) with the application.  We tested this feature through a collaboration with Joao Sousa of George Mason University. This testing resulted in the development of an alternative communication mechanism that integrates with the prototype with only a few weeks of effort, instead of months or years.  We anticipate leveraging these standardized interfaces to collaborate with a variety of external groups and organizations as new sensor technologies and communication mechanisms become available.   App Architecture - User Interface (UI) The ISE app, through the use of Android UI screens called Activities, reflects the view part of the MVC Pattern. There are currently only three supported UIs in ISE: User: Allows users to look at the people with whom they are or can be connected, as well as the context data associated with each person. Task View: Allows users to create their own tasks, receive updates about other users’ tasks, and mark their tasks complete or incomplete. We are expanding the task view to include tasks, with main tasks and subtasks under them. Ultimately, we will develop a capability that displays complex missions in an intuitive manner. Alerts View: As events occur, some will automatically appear in the alerts view along with a list of the considerations the context engine has identified as items of importance for users. The alerts presented will be tailored to the needs and context of individual users. We are upgrading the ISE architecture to support any UI that subscribes to standardized updates from the data services. Challenges One challenge we face involves accounting for the lack of network infrastructure.  In particular, limited bandwidth exists for the available communication channels.  We are building atop of communication capacities that other organizations are field testing in Afghanistan to tailor our solution to practical field situations.  A second challenge involves providing warfighter access to backend data sources.  Soldiers told us that important information is available in such sources, but they can’t readily find the relevant information. Moreover, they can’t access the database in the field.  Other Advanced Mobile Systems work is investing ways to provide access to critical data through the use of cloudlets. A third challenge involves reducing the user’s cognitive load by limiting the amount of interaction and attention required of the user. Residents in a metropolitan area can use their smart phones without undue concern for being in a distracted state, as long as they are not engaging in tasks that demand undivided attention. A soldier in Haiti, on the other hand, must be cognizant of crumbling buildings, while a warfighter on the ground in Afghanistan might need to digest information while taking enemy fire. Our goal is to use hardware that allows the warfighter to capture and process information seamlessly, without sacrificing valuable time and resources. We are also addressing the challenge of resource scarcity.  Resources are limited at the tactical edge and warfighters are typically limited to the power and bandwidth of whatever devices they can carry.  We are therefore exploring resource optimization based upon our expanded model of context. For example, if a warfighter’s assignment involves driving through a known safe area, it may not be necessary for the smartphone to activate the GPS capability.  By optimizing the system to use sensors only when needed, warfighters can save battery power, CPU cycles, and communication bandwidth that can be used to support other mission-critical needs. Finally, our work will not have the desired impact if we cannot meet the challenge of relevance. Warfighters made it clear to us that if a device or application is not directly useful to their immediate task, it will be ignored.  In any given day, a warfighter in Afghanistan may be asked to determine if a particular individual is a threat, sweep a village to establish identities of residents, deliver food to children, or check for a weapons cache. These different missions impact the type of information that interests soldiers and the type of information a software application should consider.  Solving this problem requires a deep understanding of the needs of soldiers and the missions in which they engage.  We are leveraging this domain knowledge so our ISE application can tailor information processing to a particular mission, thereby ensuring relevance to the current mission and the ability to change mission parameters as needed. Looking Ahead The ISE prototype is just one part of our strategy to address the problems of information overload, information obscurity, and resource scarcity.  The Advanced Mobile Systems initiative is also engaged in the Edge-Enabled Programming project as well as the Resource Optimization for Mobile Platforms at the Edge project.  Each project attacks the three problems of information overload, information obscurity, and resource scarcity from different perspectives. We intend to integrate each project together after they have matured, thereby providing an end-to-end solution to warfighters and first responders at the tactical edge. Additional Resources For more information on the MVC pattern, consult the Documenting Software Architectures: Views and Beyond and the Pattern-Oriented Software Architecture: Volume 1 books.

By Bill Scherlis, Chief Technology Officer (Acting)SEI The extent of software in Department of Defense (DoD) systems has increased by more than an order of magnitude every decade. This is not just because there are more systems with more software; a similar growth pattern has been exhibited within individual, long-lived military systems.  In recognition of this growing software role, the Director of Defense Research and Engineering (DDR&E, now ASD(R&E)) requested the National Research Council (NRC) to undertake a study of defense software producibility, with the purpose of identifying the principal challenges and developing recommendations regarding both improvement to practice and priorities for research. The NRC appointed a committee, which I chaired, that included many individuals well known to the SEI community, including Larry Druffel, Doug Schmidt, Robert Behler, Barry Boehm, and others. After more than three years of effort—which included an intensive review and revision process—we issued our final report, Critical Code: Software Producibility for Defense. In the year and a half since the report was published, I have been asked to brief it extensively to the DoD and the Networking and Information Technology Research and Development (NITRD) communities. This blog posting, the first in a series, highlights several of the committee’s key findings, specifically focusing on three areas of identified improvements to practice—areas where the committee judged that improvements both are feasible and could substantially help the DoD to acquire, sustain, and assure software-reliant systems of all kinds. The "help" is in the form of reduced costs, greater productivity, improved schedules, and lower risks of program failure—and also in enabling the DoD to build systems with much greater levels of capability, flexibility, interlinking, and assurance. The next blog postings will cover some of the lessons learned since the report came out.  Practice Improvement 1: Process and Measurement Success in developing software-dominated systems requires organizational processes that enable managers and developers to set achievable goals, analyze data, and guide decisions—and to succeed in these processes despite rapid change in operating context and in the technical and infrastructural environment. Advances related to process and measurement help facilitate broader and more effective use of incremental and iterative development methods, which have relatively short process feedback loops. These iterative approaches can better accommodate change and uncertainty.  As a consequence, these approaches are commonplace in commercial and enterprise development. But for the DoD, advances in incremental and iterative development methods must account for the typical "arms-length" relationships, common in acquisition programs, that exist between contractor development teams and government stakeholders. Incremental development practices enable continuous identification and mitigation of engineering risks during a system’s development process. Engineering risks pertain to the consequences of particular choices made within an engineering process—the risks are high when the outcomes of immediate project commitments are consequential, hard to predict, and apparent only well after the commitments are made. Engineering risks may relate to many different kinds of engineering decisions—most notably architecture, quality attributes, functional characteristics, and infrastructure choices. When managed properly, incremental practices can enable successful innovative engineering without increasing the overall programmatic risk related to completing engineering projects, such as managing stakeholder expectations and triaging priorities for cost, schedule, capability, quality, and other attributes. Incremental practices help identify and mitigate engineering risks earlier in system lifecycles than traditional waterfall approaches—the feedback is sooner, and so the costs and consequences are lower. These practices are enabled through the use of diverse techniques, such as modeling, simulation, prototyping, and other means for early validation—coupled with extensions to earned-value models that measure and acknowledge the accumulating body of evidence in support of program feasibility.  Incremental methods include iterative approaches (such as Agile), staged acquisition, evidence-based systems engineering, and other methods that explicitly acknowledge engineering risks and their mitigation. The committee found that incremental and iterative methods are of fundamental significance to innovative, software-reliant engineering in the DoD, and they can be managed more effectively through improvements in practices and supporting tools. The committee recommended a diverse set of improvements related to advanced incremental development practice, supporting tools, and earned-value models. Practice Improvement 2: Architecture In software-reliant DoD systems, architecture represents the earliest and often most important design decisions—those that are the hardest to change and the most critical to get right. Architecture is the principal way we address requirements related to quality attributes such as performance, security, adaptability, and the like. Architectural design also embodies expectations regarding the various dimensions of variability and change for a system. When architecture design is successful, system quality is more predictable, and change is more likely to be accommodated through smaller increments of effort rather than through wholesale restructuring of systems. Advances related to architecture practice thus contribute to our ability to build systems with demanding requirements related to quality attributes, interlinking, and planned-for flexibility. Software architecture techniques and tools model the structures of a system that comprises software components, externally visible properties of those components, and relationships among the components. Architecture thus has both structural and semantic aspects—it is not just about how components interconnect. Good architecture entails a minimum of engineering commitment that yields a maximum of business value. Architecture design is thus an engineering activity that is separate, for example, from standards-related policy setting and the certification of commercial ecosystems and components. For complex innovative DoD systems, architecture definition embodies planning for flexibility—defining and encapsulating areas (such as common operating platform environments and cyber-physical systems) where innovation, change, and competition are anticipated. Architecture definition strongly influences diverse quality attributes, ranging from availability and performance to security and isolation.  It also embodies planning for the interlinking of systems to form systems-of-systems and ultra-large-scale systems and for product line development enabling encapsulation of individual innovative elements of a system. For many innovative DoD systems it is essential to consider architecture and quality attributes before making too many specific commitments to functionality. This may seem backwards from the usual model, of putting functional requirements first. But the engineering reality is that architecture includes the earliest and typically the most important design decisions: those engineering costs that are the hardest to change later. Early architectural commitment (and validation) can therefore often yield better project outcomes with less programmatic risk. The committee found that in highly complex DoD systems with emphasis on quality attributes, architecture decisions may dominate functional capability choices in overall significance. The committee also noted that architecture practice in many areas of industry is sufficiently mature for the DoD to adopt. The committee recommended that the DoD more aggressively assert architectural leadership, with an early focus on architecture being essential for systems with innovative functional or demanding quality requirements. Practice Improvement 3: Assurance and Security A significant—and growing—challenge for DoD systems is software assurance, which encompasses diverse reliability, security, robustness, safety, and other quality-related and functional attributes.  The weights given these various attributes are often determined by modeling hazards associated with operational context, including potential threats and the penalties of system failure. Software assurance is very expensive—the process of achieving assurance judgments, regardless of sector, is generally recognized to account for approximately half the total development cost for major projects.  Advances related to assurance and security would therefore facilitate greater mission assurance for systems at greater degrees of scale and complexity. Advances in assurance and security are particularly important to the rich supply chains and architectural ecosystems that are increasingly commonplace in modern software engineering. The growing reliance on software by the DoD has increased the functional capability of all kinds of systems, as well as a growth in the interconnectedness of systems and the extent of potential for rapid adaptation of systems. With this growth has come a dependence of DoD software-reliant systems on increasingly complex, diverse, and geographically distributed supply chains. These supply chains include not only custom components developed for specific mission purposes, but also commercial and open-source ecosystems and components, such as the widely used infrastructures for web services, cloud computing environments, mobile devices, and graphical user interaction. This places emphasis on composition and on localizing points of trust within a system. In addition to managing overall costs, the DoD faces many challenges for assurance relating to technology, practices, and incentives, including: The arms-length relationship between contractor development teams and government stakeholders also complicate the creation and sharing of information necessary to make assurance judgments. This type of relationship can lead to approaches that focus excessively on post hoc acceptance evaluation, rather than on the emerging practice of building evidence in support of an overall assurance case. Modern systems draw on components from diverse sources, implying that supply-chain and configuration-related attacks must be contemplated, with "attack surfaces" existing within an overall application, and not just at its perimeter. The consequence of this trend is that evaluative and preventive approaches should ideally be integrated throughout a complex supply chain. A particular challenge is managing black box components in a system (this issue is addressed in the full report). The growing role of DoD software in warfighting, protection of national assets, and the safe guarding of human lives creates a diminishing tolerance for faulty assurance judgments.  The Defense Science Board notes that there are profound risks associated with the increasing reliance on modern software-reliant systems: "this growing dependency is a source of weakness exacerbated by the mounting size, complexity, and interconnectedness of its software programs." Losing the lead in the ability to evaluate software and prevent attacks can confer advantage to adversaries with respect to both offense and defense. It can also force the DoD to restrict functionality or performance to a level such that assurance judgments can be achieved more readily. The Defense Science Board also found "it is an essential requirement that the United States maintain advanced capability for ‘test and evaluation’ of IT products. Reputation-based or trust-based credentialing of software (‘provenance’) needs to be augmented by direct, artifact-focused means to support acceptance evaluation." Achieving this capability is a significant challenge due to the rapid advance of software technology generally, as well as the increasing pace by which potential adversaries are advancing their capabilities. This challenge—coupled with the observations above regarding software innovation—provides an important part of the rationale for the committee’s recommendation that the DoD actively and directly address its software producibility needs. The committee found that assurance is facilitated by advances in diverse aspects of software engineering practice and technology, including modeling, analysis, tools and environments, traceability and configuration management, programming languages, and process support. The committee also found that simultaneous creation of assurance-related evidence with ongoing development has high potential to improve the overall assurance of systems. The committee recommended enhancing incentives for software assurance practices and production of assurance-related evidence throughout the software lifecycle and through the software supply chain for both contractor and in-house developments. Looking Ahead The next blog posting in this series will focus on lessons learned in the many interactions subsequent to the publication of the NRC Critical Code report. I will also discuss what these lessons signify for developing software strategy for the DoD, in general, and the SEI, in particular. Additional Resources This posting is an excerpted, edited copy of an article that Bill Scherlis wrote for The Next Wave, "Critical Code: Software Producibility for Defense," which was published in Volume 19, No. 1 (2011). To request copies of the journal, please send an email to tnw@tycho.ncsc.mil. To download a PDF of the report Critical Code: Software Producibility for Defense, go towww.nap.edu/catalog.php?record_id=12979. To download a PDF of the Report of the Defense Science Board Task Force on Defense Software (2000), go to http://oai.dtic.mil/oai/oai?verb=getRecord&metadataPrefix=html&identifier=ADA385923

By Dave Zubrow, Chief ScientistSoftware Engineering Process Management Program By law, major defense acquisition programs are now required to prepare cost estimates earlier in the acquisition lifecycle, including pre-Milestone A, well before concrete technical information is available on the program being developed. Estimates are therefore often based on a desired capability—or even on an abstract concept—rather than a concrete technical solution plan to achieve the desired capability. Hence the role and modeling of assumptions becomes more challenging.  This blog posting outlines a multi-year project on Quantifying Uncertainty in Early Lifecycle Cost Estimation (QUELCE) conducted by the SEI Software Engineering Measurement and Analysis (SEMA) team. QUELCE is a method for improving pre-Milestone A software cost estimates through research designed to improve judgment regarding uncertainty in key assumptions (which we term program change drivers), the relationships among the program change drivers, and their impact on cost. Our Approach According to a February 2011 presentation by Gary Bliss, director of Program Assessment and Root Cause Analysis, to the DoD Cost Analysis Symposium, unrealistic cost or schedule estimates are a frequent causal factor for programs breaching a performance criterion.  Steve Miller, director of the Advanced Systems Cost Analysis Division of OSD Cost Analysis and Program Evaluation, noted during his DoDCAS 2012 presentation that "Measuring the range of possible cost outcomes for each option is essential …Our sense is not that the cost estimates were poorly developed [but] rather key input assumptions didn’t pan out."  For instance, an estimate might assume It is possible to mature technology A from technology readiness level 4 to level 7 in three years. The program will not experience any obsolescence of parts within the next five years. Foreign military sales will support lower production costs. An interdependent program will complete its development and deployment in time for this program to use the products. We can reuse 70 percent of the code in the missile tracking system. QUELCE addresses the challenge of getting the assumptions "right" by characterizing them as uncertain events rather than certain eventualities.  As we’ve noted previously, modeling uncertainty on the input side of the cost model is a hallmark of the QUELCE method. By better representing uncertainty, and therefore risk, in the assumptions and explicitly modeling them, DoD decision makers, such as Milestone Decision Authorities (MDAs) and Service Acquisition Executives (SAEs), can make more informed choices about funding programs and portfolio management. QUELCE is designed to ensure that DoD acquisition programs will be funded at levels consistent with the magnitude of risk to achieving program success, fewer and less severe program cost overruns will occur due to poor estimates, and there will be less rework reconciling program and OSD cost estimates. QUELCE relies on Bayesian Belief Network (BBN) modeling to quantify uncertainties among program change drivers as inputs to cost models. QUELCE then uses Monte Carlo simulation to generate a distribution (as opposed to a single point) for the cost estimate. In addition, QUELCE includes a DoD domain-specific method for improving expert judgment regarding the nature of uncertainty in program change drivers, their interrelationships, and eventual impact on program cost drivers. QUELCE is distinguished from other approaches to cost estimation by its ability to allow subjective inputs based solely on expert judgment, such as the identification of program change drivers and the probabilities of state changes for those drivers, as well as empirically grounded ones based on historical data, such as estimated system size and likely growth in that estimate visually depict influential relationships, scenarios, and outputs to aid team-based development, and explicit description and documentation of assumptions underlying an estimate use scenarios as a means to identify program change drivers, as well as the impacts of alternative acquisition strategies, and employ dependency matrix  transformation techniques to limit the combinatorial effect of multiple interacting program change drivers for more tractable modeling and analysis The QUELCE method consists of the following steps in order: Identify program change drivers: workshop and brainstorm by experts. Identify states of program change drivers. Identify cause-and-effect relationships between program change drivers, represented as a dependency matrix. Reduce the dependency matrix to a feasible number of inter-driver relationships for modeling, using matrix transformation techniques. Construct a BBN using the reduced dependency matrix. Populate BBN nodes with conditional probabilities. Define scenarios representing nominal and alternative program execution futures by altering one or more program change driver probabilities. Select a cost estimation tool and/or cost estimating relationships (CERs) for generating the cost estimate. Obtain program estimates of size and/or other cost inputs that will not be computed by the BBN. For each selected scenario, map BBN outputs to the input parameters for the cost estimation model and run a Monte Carlo simulation. Improving the Reliability of the Expert Opinion Early cost estimates rely heavily on subject matter expert (SME) judgment, and improving the reliability of these judgments represents another focus of our research. Expert judgment can be idiosyncratic, and our aim is to try to make it more reliable. QUELCE draws upon the work of Dr. Douglas Hubbard, whose book How to Measure Anything describes a technique known as "calibrating your judgment" that we are adapting for our DoD cost estimation analysis. For example, if you state you are 90 percent confident, you should be correct in your answers 90 percent of the time.  If you state you are 80 percent confident, you would be correct 8 times out of 10.  Performing in agreement with your statement of confidence is termed "being calibrated."  Hubbard’s technique operates by giving participants a series of questionnaires.  The participants are asked to provide an upper and lower bound for the answer to each question such that they believe they will be correct 90 percent of the time. Hence, a participant should get 9 out of 10 answers right.  If they answer all 10 correctly, they are being too conservative in their answers; they provided too wide of a range.  If they get fewer than 9 correct, they are over confident and providing too narrow of a range for their answers. Hubbard’s approach provides feedback so that participants are consistently correct 90 percent of the time.  Through this method of testing and feedback, they learn to calibrate their judgment. Applying that same approach to DoD cost estimation analysis would ideally mean that if two calibrated judgments are being applied to the same cost estimate, there is now a more precise idea of what those judgments mean. Hubbard, who taught a class at the SEI, demonstrated that most people start off being highly over confident in terms of their knowledge and judgment. We plan to test Hubbard’s approach of calibrating judgment with questions specific to software estimating at several universities, including Carnegie Mellon University and the University of Arizona. To develop the materials for these experiments, we are mining information from open-source repositories, such as Ohloh.net.  Our objective is to increase the consistency and repeatability of expert judgment as it is used in software cost estimation. Addressing Challenges A key challenge that our team faces in conducting our research is validating the QUELCE method.  It can literally take years for a program to reach a milestone against which we can compare its actual costs to the estimate produced by QUELCE.  We are addressing this challenge by validating pieces of the method through experiments, workshops, and retrospectives.  We are currently conducting a retrospective on an active program that provided us access to its historical records.  Key to this latter activity is the participation of team members from the SEI Acquisition Support Program (ASP).   The ASP members are playing the role of program experts as we work our way through the retrospective. Another challenge that our work on QUELCE addresses is insufficient access to DoD information and data repositories may significantly jeopardize our ability to conduct sufficient empirical analysis for the program change driver repository. To address this, we have been working with our sponsor and others in the Office of the Secretary of Defense to gain access to historical program data stored in a variety of repositories housed throughout the DOD.  We plan to use this data to develop reference points and other information that will be used by QUELCE implementers as a decision aid when developing the BBN for their program.  This data would also be included in the program change driver repository. Developing a Repository We are creating a program change driver repository that will be used as a support tool when applying the QUELCE method. The repository is envisioned as a source of program change drivers—what events occurred during the life of a program that directly or indirectly impacted its cost—along with their probability of occurrence.  The repository will also include information that will be used as part of the method for improving the reliability of expert judgment such as reference points based on the history of Mandatory Procedures for Major Defense Acquisition Programs.  Developing the repository is a major task planned for FY13.  We also plan to conduct additional pilots of the method including use of the repository and support tools. From those pilots, we will develop guidance for the use of the repository and make it available on a trial basis within the DoD.  After the repository is adequately populated and developed, we intend it to become an operational resource for DoD cost estimating. Transitioning to the Public During the coming year, our SEMA team will work to create guidance and procedures on how to mine program change relationships and related cost information from DoD acquisition artifacts for growth of the program change driver repository collaborate with Air Force Cost Analysis Agency to include results from analyzing Software Resources Data Report data in the program change driver repository assemble a catalog of calibrated mapping of BBN outputs to cost estimation models and make it available to the DoD cost community continue discussions with Defense Acquisition University (DAU), Service Cost Centers, and the DoD cost community about research and collaboration opportunities (for example, discussions at the DoD Cost Analysis symposium) Additional Resources To read the SEI technical report Quantifying Uncertainty in Early Lifecycle Cost Estimation (QUELCE), please visit www.sei.cmu.edu/library/abstracts/reports/11tr026.cfm. For more information about Milestone A, please see the Integrated Defense Life Cycle Chart for a picture and references in the "Article Library."  

David Svoboda Software Security Engineer CERT Secure Coding Initiative As security specialists, we are often asked to audit software and provide expertise on secure coding practices. Our research and efforts have produced several coding standards specifically dealing with security in popular programming languages, such as C, Java, and C++. This posting describes our work on the CERT Perl Secure Coding Standard, which provides a core of well-documented and enforceable coding rules and recommendations for Perl, which is a popular scripting language. Perl is a relatively young language, only slightly older than Java. Perl became popular early in its lifetime because it was the first general-purpose scripting language on many Unix platforms. Perl enjoyed a second burst of popularity as the web became prominent because it was especially well-suited to writing Common Gateway Interface (CGI) scripts. In recent years Perl's popularity has been cemented by CPAN, a public repository of free software libraries written in Perl. Any computer with Perl installed provides straightforward mechanisms to install and use any software library from CPAN. This feature enables programmers to use libraries provided by the community easily and quickly. Several new features in Perl began life as CPAN modules before being integrated into the language. As a result of its popularity, many important software systems are written in Perl, such as the Request  Tracker database (RT), an open-source project for managing tickets or bugs for a help desk, which is maintained by Best Practical Solutions. Many websites, such as amazon.com, also rely on Perl code on their servers. The CERT Perl Secure Coding standard is still young and growing. The C and Java standards have more than 200 rules in about 20 sections each. The Perl standard currently has slightly more than 30 rules in the following eight sections: Input Validation and Data Sanitization  - issues dealing with data provided by an attacker, such as XML injection and cross-site scripting (XSS). Declarations and Initialization - issues dealing with securely declaring variables and functions including package versus lexical variables, name clashes, and dangers of uninitialized data. Expressions  - issues dealing with Perl’s expressions syntax including list versus scalar contexts, when to use the $_ variable, and when to use the various types of comparison operators. Integers  - issues dealing with numbers, such as how to specify octal numbers. Strings  - issues dealing with strings and regular expressions (regexes) including the danger of providing a string literal to a subroutine that expects a regex. Object-Oriented Programming (OOP) - issues dealing with OOP are covered in this section, such as recognizing the convention of private variables. File Input and Output - issues dealing with how to safely work with files, including safely working with Perl’s filehandles. Miscellaneous - issues that don’t fall into other sections, such as handling dead code and unused variables. Addressing Security Vulnerabilities in Perl The Perl community has always prioritized practicality over theoretical elegance, and so it has always been considered an easy language to write code in—although Perl code is often considered ugly due to the tendency of some Perl developers to create "write only" programs. Perl was not designed as a secure programming language. However, problems relating to security in Perl programs have been discussed in security circles, and appear in databases such as the CERT vulnerability database. Moreover, companies that request software audits are just as likely to want Perl software audited as they are to request audits for C, C++, or Java. While the Perl community is interested in improving the language, the focus on security has historically tended to take a back seat to other priorities, such as new features and improved performance. Our work on the CERT Perl Secure Coding Standard therefore centers on addressing issues in the Perl language and libraries that deal specifically with security. The standard covers issues, such as XML injection, integer security, and proper input and output, as outlined above. By making the standard publicly accessible, we invite the Perl community to help us improve the standard. The standard leverages several sources to provide relevant material on security. For example, it takes advantage of the US-CERT vulnerability database, which contains entries on several vulnerabilities that address the Perl language or applications written in Perl. It also leverages experience gained from the Source Code Analysis Lab (SCALe), which has been used to perform security audits on several pieces of Perl code, including the previously-mentioned RequestTracker (RT) tool created by Best Practical Solutions. Other analysis tools, such as Perl::Critic, provide an automated audit of a Perl program by examining a codebase and producing a list of diagnostics. These diagnostics can range from insecure coding practices and bugs to stylistic issues. The SCALe project uses these tools to harvest the diagnostics that address security issues, while discarding diagnostics not relevant to security. The CERT Perl standard can leverage the other CERT standards for security issues that are not bound to any particular language. For instance, many issues about securely opening files on a Unix machine are language-independent.  As a result, these portions of CERT standards for security issues can affect any software that runs on Unix systems regardless of the language in which it is written. While Perl has many of the same security issues that plague C and Java, several issues are unique to Perl. For example, Perl's open() function can take two arguments, with the latter argument being either a file name or a shell command. The open() function either opens the file or executes the command. If the argument begins or ends with a | (pipe) character, it is interpreted as a command to execute. Consequently, if an attacker can specify a filename to Perl's open() function—and that filename begins or ends with |—the attacker can cause Perl to execute the command for which the file is named. This issue is discussed further in rule IDS31-PL in the CERT Perl Secure Coding Standard. Perl has some technology that appears similar to other languages but presents unique problems when examined more closely. For example, C, Java, and Perl all share the concept of an array, which is a continuous vector of items that can be accessed via an index. In C and Java, arrays are fixed-size, which means they are created to hold a specific number of elements and their size remains fixed until they are destroyed. Trying to refer to an element greater than the size of the array is illegal. For example, asking for the 11th element in a 10-element array in Java will cause an exception to be thrown, which usually causes the program to crash. In contrast, Perl's arrays can grow over their lifetime. Assigning a value to the 11th element of a 10-element Perl array causes the array to grow in memory such that the array contains 11 elements, so the request becomes valid. This quality makes Perl an especially agreeable language to work with Ibecause it never reports that an array is too small. If you were to assign a value to the 1,000,000,000th element of an array,however, Perl would attempt to grow the array enough to accommodate the request and might exhaust memory. Exhausting system memory, whether deliberate or unintentional, can lead to security vulnerabilities because a system with limited memory will refuse to provide memory to any program that requests more. At the same time, many programs fail to check whether their memory requests succeeded. A machine with no free memory, therefore, is likely to have running programs that crash, either unintentionally or by design, using some sort of "out of memory" error. Consequently, the CERT Perl Secure Coding standard contains rule IDS32-PL, which forbids allowing untrusted users from providing an array index, lest they cause Perl to exhaust memory with an excessively large number. What’s Ahead for the CERT Perl Secure Coding Standard We are adding several rules each week, and presumably the Perl secure coding standard can grow to about the same size as the C or Java standards since it’s comparable in scope. We welcome your assistance in helping us complete the standard. Editor's Note: In response to feedback from our readers, this post has been edited. The post originally stated "Asking for the 11th element of a 10-element Perl array causes the array to grow in memory such that the array contains 11 elements, so the request becomes valid." As our readers pointed out in the comments section, "Simple asking is not enough." The post now states that "Assigning a value to the 11th element of a 10-element Perl array causes the array to grow in memory such that the array contains 11 elements, so the request becomes valid." Additional Resources The CERT Perl Secure Coding Standard may be viewed at https://www.securecoding.cert.org/confluence/display/perl/CERT+Perl+Secure+Coding+Standard

By Douglas C. Schmidt Principal Researcher While agile methods have become popular in commercial software development organizations, the engineering disciplines needed to apply agility to mission-critical software-reliant systems are not as well defined or practiced. To help bridge this gap, the SEI recently hosted the Agile Research Forum, which brought together researchers and practitioners from around the world to discuss when and how to best apply agile methods in the mission-critical environments found in government and many industries. This blog posting, the first in a multi-part series, highlights key ideas and issues associated with applying agile methods to address the challenges of complexity, exacting regulations, and schedule pressures that were presented during the forum. Carleton’s Forum Introduction When introducing the forum, Anita Carleton, director of the SEI’s Software Engineering Process Management Program, summarized how agile methods can provide customer value sooner and enable organizations to respond to change more quickly. Carleton started by highlighting the four key tenets presented in the Agile Manifesto, which forms the foundation for many agile methods, such as Scrum and Kanban: people over processes and tools working software over comprehensive documentation customer collaboration over contract negotiations responding to change over following a plan Carleton explained that agility means the ability to move quickly and easily, the ability to think and reason quickly, or to possess intellectual acuity. "In a business context this is the definition of agile that matters," Carleton said.  Agility in business means having an organization that moves, thinks, and responds quickly to change, not only in the short term, but over the lifetime of the system product or relationship. Agility is the ability to provide customer value sooner, to align development tempos with operational tempos, and to turn fast response into the business competitive advantage. While agile methods have become popular with many information technology (IT) professionals as a means to replace perceived top-down bureaucracy with greater self-discipline and team-discipline, Carleton noted in her presentation that the SEI emphasizes measurable performance value for DoD programs and contractors who apply agile practices. Agility is not a capability you achieve by accident. Like agility in sports, which requires teamwork, strategy, training, management, and discipline, achieving agility in a software development organization is no different. Carleton explained that the SEI’s research and transition efforts have concentrated in recent years on enhancing lightweight approaches and processes to address what’s needed and required by our sponsors, partners, customers, and other stakeholders. Meeting these needs involves scaling up agile methods to the mission-critical software-reliant systems common in the Department of Defense (DoD), as well as mission-critical programs in other domains, such as finance, energy, telecommunications, space exploration, and aviation. Carleton outlined the following areas where SEI work is focusing on applying agile methods at-scale: defining and evaluating practical guidance for DoD project managers, systems engineers, and contracting officers who are considering the adoption of agile methods defining metrics that can be used to measure and appraise the performance and success of programs that apply agile methods working with DoD acquisition programs to pilot and roll-out agile methods in combat system development environments developing an architecture-focused measurement framework for managing technical debt in agile projects formulating a decision making framework for reducing integration risk with agile methods applying agile principles in strategic planning processes   Takai’s Keynote Presentation In her keynote presentation during the forum, Teresa M. Takai, chief information officer (CIO) for the DoD, discussed how agile methods have been introduced into the DoD software acquisition and development environment.  As the DoD CIO, Takai provides IT support for 2 million individuals across the globe: 1 million on the civilian side and 1 million on military side. Providing IT support for the DoD means delivering reliable computing and communications capability for warfighters, regardless of their location. IT is an essential part of what the DoD does to enable service men and women to perform their duties. Challenges that Motivate DoD Agility The DoD faces many challenges—including budget pressures and rapid technology insertion—that motivate the need for agile methods, Takai said during her keynote. These challenges have resulted in two realities for IT development: The DoD must be a good custodian of technology dollars, Takai commented during the forum, pointing out that the DoD spends at least $38 billion a year on IT. The DoD also needs to speed up IT delivery. Men and women hired by the DoD now expect to do their jobs using their smart phones, the same way that they do in their private life, Takai explained. Long development cycles don’t fit with user expectations. In the DoD, it can take up to 81 months to acquire or develop a new technology. "Typically, slow acquisition time has been blamed on the acquisition process, but that’s not always fair," Takai said. The challenge for the DoD is that the acquisition process encompasses front-end requirements gathering, recruiting industry partners, and, testing and implementation. The mandate for the DoD to change is enormous. Agile practices are not just a methodology, they involve a cultural change in the way business is conducted. Takai suggested to the audience that cultural change is the hardest part of adopting agile methods in the DoD. Nearly 33 percent of DoD IT programs are canceled during development because, as programs move through a process taking 81 months, they realize they can’t deliver the capability they had intended to deliver. Over 60 percent of DoD IT programs are late and/or over budget. Larger IT projects have a much larger risk of over budget and under delivery. Cultural and Process Changes Need to Support DoD Agility As part of the DoD IT acquisition reform effort, Takai explained that DoD leaders are examining how to take agile best practices, continue to educate the IT technology workforce on the meaning of these best practices, and ensure that all involved understand the overall DoD culture to ensure IT developers can apply agile methods effectively. This cultural change involves enhancements to established DoD policies and practices. The DoD is part of a larger government-wide effort that, under the US CIO, published a 25-point plan, a portion of which prescribed transitioning to agile methods, as elaborated in Takai’s 10-point plan for IT modernization in the DoD. Some segments within the DoD have already begun transitioning to agile methods, Takai said.  Based on these experiences, the DoD has been establishing the framework and a handbook that program managers can to guide their implementations of agile methods within its organizations. One of the DoD’s strategies has been to share best practices. An important best practice has involved establishing a governance process that promotes the successful adoption of agile IT methods. DoD organizations are accustomed to developing DoD specific solutions, such as radio or weapons systems, that involve rigorous processes. Unfortunately, with this approach it’s hard for program managers to decompose software systems into smaller, more deliverable chunks necessary for agile development. "The DoD can no longer simply write and sign off on requirements and then turn it over to acquisition to manage delivery," Takai said. The DoD must find a better way to involve the user through the development process. For the DoD that’s not always been the norm, so making that move involves cultural changes. Takai said that as the DoD considers large-scale IT development projects (some in the billion-dollar range), leaders need to learn how divide them into chunks effectively. This decomposition process should start by specifically examining the ongoing requirements process, involving the user, and ensuring a much stronger governance process. The DoD must also examine how to manage agile development from a risk-mitigation standpoint. One challenge of traditional waterfall methods is that the focus is often on avoiding risk. Risk avoidance is not what IT is about anymore, Takai said. Implementing agile methods will enable the DoD to mitigate risk and make the changes needed after a small increment of delivery, and then build on that concept to reach next stages of delivery. That approach is a tough concept for the DoD, which has historically focused on ensuring that requirements and processes remain iron clad, with minimal risk involved. Such an approach eliminates the ability to bring in innovative technologies, as well the ability to implement industry best practices. That’s not the way to move forward, she said, especially in an era of austerity in the DoD budget. What's Next Our next posts in this series will summarize discussions of four SEI researchers, including myself, at the Agile Research Forum who examined aspects of applying agile methods at-scale in mission-critical development environments: Mary Ann Lapham highlighted the importance of collaboration with end users, as well as among cross-functional teams, to apply agile approaches to DoD acquisition programs successfully at-scale. She noted that effective agile DoD teams are flexible, experienced, and able to work fluidly between disciplines. Ipek Ozkaya discussed the use of strategic, intentional decisions to incur architectural technical debt. The technical debt metaphor describes the tradeoff between taking shortcuts in software development to speed up product delivery and slower—but less risky—software development. James Over noted that lack of teamwork can critically impede agility. He advocated, among other principles, the building of self-managed teams, planning and measuring project process, designing before building, and making quality the top priority for achieving agility at-scale. Finally, I wrapped up the forum with a discussion on the importance of applying agile methods to crucial common operating platform environments (COPEs) at the DoD. I explained how agile methods can encourage more effective collaboration between users, developers, testers, and certifiers to help the DoD successfully build integrated, interoperable software systems. In addition to providing you weekly updates of the latest research from our technologists, the SEI blog has also become a catalyst for sparking thoughtful discussions on the latest challenges facing commercial and DoD organizations.  We therefore look forward to hearing your thoughts on applying agile at-scale in the comments section below. Additional Resources The slides and recordings from the SEI Agile Research Forum can be accessed at www.sei.cmu.edu/go/agile-research-forum/.  

By Douglas C. Schmidt, Principal Researcher While agile methods have become popular in commercial software development organizations, the engineering disciplines needed to apply agility to mission-critical, software-reliant systems are not as well defined or practiced. To help bridge this gap, the SEI recently hosted the Agile Research Forum, which brought together researchers and practitioners from around the world to discuss when and how to best apply agile methods in mission-critical environments found in government and many industries. This blog posting, the second installment in a multi-part series, summarizes a presentation made during the forum by Mary Ann Lapham, a senior researcher in the SEI’s Acquisition Support Program, who highlighted the importance of collaboration with end users, as well as among cross-functional teams, to facilitate the adoption of agile approaches into DoD acquisition programs. Lapham’s Talk on Agile Methods: Tools, Techniques, and Practices for the DoD Community The broad—and rapidly expanding—threats the DoD must address necessitates an ability to develop software faster, Lapham told the audience. "In today’s environment people want results faster. They want to use information technology (IT) applications and infrastructure sooner; there is a real need," Lapham said. In the commercial and DoD domains, the prevailing question is, "How can IT be delivered faster?" The answer, Lapham told the audience, is an iterative approach that lends itself to agile methods. Lapham noted that the SEI is focused on reducing the DoD information technology development cycle—which can currently take as long as 81 months—to short, increemental approaches that yield results more quickly. One complicating factor is that DoD acquisition programs (like other highly-regulated commercial environments) have a prescribed vision of how IT systems are developed, Lapham explained. She referenced the DoD 5000 Series Acquisition Lifecycle, which has traditionally employed a waterfall approach that focuses largely a sequential process of requirement analysis followed by design, implementation, and testing. Lapham said that she and other SEI researchers are working on developing an approach that will allow the DoD to develop applications in a shorter period, ideally 18 to 24 months. One aspect of Lapham’s research focuses on helping the DoD transition from a traditional method to more iterative and incremental development methods, while still operating within the regulatory boundaries of the overarching DoD 5000 Series Acquisition Lifecycle model. Implementing Agile Effectively for in DoD Environments Lapham said the SEI’s research in this field began in 2009 when a DoD client asked about using agile methods.  "We reviewed the 5000 series to see if something in it would preclude us from using agile, and there wasn’t. From there, we’ve gone on to investigate different parts of the acquisition process so we can help program offices and contractors understand how to implement agile effectively in DoD environments." Lapham said her team applied agile methods to study agile methods. Researchers started by interviewing practitioners who were experts with traditional waterfall methods about how those methods fit into the DoD acquisition lifecycle. Next, Lapham and her team identified the gaps that would occur if agile principles were applied by the DoD in the traditional acquisition lifecycle. After identifying the gaps, Lapham’s team consulted with DoD stakeholders to ensure they had identified the appropriate gaps. The researchers then characterized those gaps and built a model to form a complete overview of the lifecycle with agile principles. The research conducted by Lapham’s team yielded a compendium of topics that addressed the barriers of adopting agile methods in the DoD. "We have a list of 30 topics, including such questions as: How do I do agile contracting? How do I do agile requirements management? How do I do agile cost estimation, testing, and system engineering?" Lapham explained. Next, the researchers consulted with DoD acquisition stakeholders to ensure that the topics they are addressing are relevant. The team is in the midst of piloting their agile approach with practitioners. "We’re applying a lot of the agile methods that have been used successfully in the commercial arena," Lapham said, adding that their research accounted for the fact that certain agile terms in the commercial world differ from those in a DoD environment. The published results—which will be released starting later this year—will be a set of validated tools, techniques, and practices. Comparing and contrasting traditional and agile approaches to software development Lapham noted the research results thus far have yielded the following findings about traditional versus agile development:     Characteristics of traditional approaches an arms-length relationship between developers and acquirers hierarchical, command-and-control-based teams leader as keeper of the vision and primary source of authority to act conventional, representational documents used by the program management office to oversee the progress of developers in a software development lifecycle model with separate teams, particularly for development and  testing; some independent program teams involve multiple functions Characteristics of agile approaches collaborative relationships between developers, acquirers, and end users strong team relationships, with collocated teams, or effective communication mechanisms with distributed teams facilitated leadership, with the leader as champion and team advocate "just enough" documentation to maintain a product and continue to use it and evolve it (documentation is highly dependent on product context) cross-functional team relationships that includes all roles throughout the lifecycle where every member of the team performs their function, but they perform it together and reinforce it Lapham also described how SEI researchers are compiling a compendium of cultural issues that organizations need to consider when implementing agile, as described below Organizational Structure. Many DoD practitioners are content with traditional hierarchical structures where one person is in charge. Traditional DoD organizational structures are hard to change due to their command-and-control-based integrated-product teams that have formal responsibilities and roles.  They meet on a prescribed schedule, usually once a month. Often, those teams work through certain issues as part of their charter. Agile organizations, in contrast, are characterized by flexible and adaptive structures. Teams are cross-functional and small. An agile organization might have multiple teams working together in different locations, but still maintain constant communication. Teams will be self-organized, but that doesn’t mean they lack discipline, Lapham said. Instead, agile projects require developers with rigor across a core set of processes. Leadership. In traditional DoD software development approaches, the leader is the keeper of the vision and the primary source of authority to act. In an agile DoD organization, in contrast, the goal is facilitative leadership, the leader is an advocate and champion for the team. This approach is a different style of leadership that requires a paradigm shift in management styles in DoD organizations. Reward systems. A traditional DoD organization focuses on the individual and rewarding individuals for high performance. In an agile DoD environment, the team is the focus of the rewards system. Lapham commented that team members typically behave based on the activities for which they are incentivized. If developers are rewarded for being the hero, therefore, that may not create an environment conducive to team building. Staffing model. A traditional DoD organization uses a lifecycle model with separate teams, particularly for development and testing. Different roles are active at defined points in the lifecycle and are not substantively involved, except at those defined times. An agile DoD environment, in contrast, employs cross-functional teams, including all roles across the lifecycle of the project. The teams contain an agile mentor or coach who explicitly attends to the team’s process and ensures that they work together cooperatively. Communication and decision making. In organizations that employ a traditional approach to software development, top-down communication structures dominate. Likewise, external regulations drive the focus of the work while indirect communications, such as documented activities and processes, dominate over face-to-face dialogue. Program management office oversight tools focus on demonstrating compliance.  In an Agile DoD environment, in contrast, teams usually hold 15-minute daily standup meetings in which three main topics are discussed: What am I going to do today? What did I do yesterday? What problems did I have? (the goal is not to solve problems in this short meeting, but the agile coach determines whose responsibility it is to solve those problems at the end of the meeting) In an agile environment, teams hold frequent retrospectives to improve practices while information radiators are used to communicate critical project information to avoid surprises. Information radiators are entities (sometimes automated tools, sometimes just stickies on a board) that provide status and ensrue an open and transparent flow of information.  Documents serve to feed conversation among team members. Agile organizations produce enough documentation required to meet DoD acquisition regulations, which is highly dependent on product context. What's Ahead The first posting in this series summarized discussions by Anita Carleton, director of the SEI’s Software Engineering Process Management program, and Teri Takai, chief information officer for the DoD. Carleton provided an overview of the forum and discussed areas where SEI work is focused on applying agile methods at-scale. Takai then discussed how agile methods have been introduced into the DoD software acquisition and development environment. Our next posts in this series will summarize discussions of three SEI researchers, including myself, at the Agile Research Forum who examined aspects of applying agile methods at-scale in mission-critical development environments: Ipek Ozkaya discussed the use of strategic, intentional decisions to incur architectural technical debt. The technical debt metaphor describes the tradeoff between taking shortcuts in software development to speed up product delivery and slower—but less risky—software development. James Over noted that lack of teamwork can critically impede agility. He advocated, among other principles, the building of self-managed teams, planning and measuring project process, designing before building, and making quality the top priority for achieving agility at-scale. Finally, I wrapped up the forum with a discussion on the importance of applying agile methods to crucial common operating platform environments (COPEs) at the DoD. I explained how agile methods can encourage more effective collaboration between users, developers, testers, and certifiers to help the DoD successfully build integrated, interoperable software systems. We look forward to hearing your thoughts on applying agile at-scale in the comments section below. Additional Resources The slides and recordings from the SEI Agile Research Forum can be accessed at www.sei.cmu.edu/go/agile-research-forum/. To read Lapham’s SEI blog posting on Using Agile Effectively in DoD Environments, please visit http://blog.sei.cmu.edu/post.cfm/using-agile-effectively-in-dod-environments.  To read the SEI technical note, Considerations for Using Agile in DoD Acquisition, please visitwww.sei.cmu.edu/library/abstracts/reports/10tn002.cfm. To read the SEI technical report, Agile Methods: Selected DoD Management and Acquisition Concerns, please visit www.sei.cmu.edu/library/abstracts/reports/11tn002.cfm. To read the SEI technical report, A Closer Look at 804: A Summary of Considerations for DoD Program Managers, please visit www.sei.cmu.edu/library/abstracts/reports/11sr015.cfm. To read an article in Crosstalk by Lapham, DoD Agile Adoption: Necessary Considerations, Concerns, and Changes, please visit www.crosstalkonline.org/storage/issue-archives/2012/201201/201201-Lapham.pdf.

By Douglas C. SchmidtPrincipal Researcher While agile methods have become popular in commercial software development organizations, the engineering disciplines needed to apply agility to mission-critical, software-reliant systems are not as well defined or practiced. To help bridge this gap, the SEI recently hosted the Agile Research Forum. The event brought together researchers and practitioners from around the world to discuss when and how to best apply agile methods in mission-critical environments found in government and many industries. This blog posting, the third installment in a multi-part series highlighting research presented during the forum, summarizes a presentation made during the forum by Ipek Ozkaya, a senior researcher in the SEI’s Research, Technology & System Solutions program, who discussed the use of agile architecture practices to manage strategic, intentional technical debt. Ipek’s Talk on Strategic Management of Technical Debt In her opening comments to the audience, Ozkaya established that two decades ago Ward Cunningham coined the "technical debt" metaphor, which refers to the degraded quality resulting from overly hasty delivery of software capabilities to users. Cunningham stated that shipping code quickly is like going into debt. A little debt speeds up development, and can be beneficial as long as the debt is paid back promptly with a rewrite that reduces complexity and streamlines future enhancements. A delicate balance is needed between the desire to release new software capabilities rapidly to satisfy users and the desire to practice sound software engineering that reduces subsequent rework. Increasingly, the software engineering community and those adopting agile techniques are interested in understanding how to quantify technical debt and manage debt pay-back strategies. Ozkaya observed that organizations are often driven to agile techniques after observing increasing technical debt in their software-reliant systems. Ironically, adopting agile practices at scale without considering their long-term implications can also easily lead to technical debt. Ozkaya’s talk emphasized the need to explicitly acknowledge the tradeoffs between taking shortcuts in software development to accelerate product delivery versus applying slower—but less risky—software development methods. Ozkaya questioned whether it is possible to avoid technical debt altogether, especially given the increasing scale and complexity of software-reliant systems, coupled with trends in the DoD and other government agencies to sustain systems that are expected to operate for decades. Another factor impacting technical debt is workforce diversity and turnover, which often yields distributed teams that must be managed remotely. Given these factors, Ozkaya said, it is inevitable that technical debt will accumulate since environments, systems, and technologies will change, so technical debt has become an ongoing software engineering practice that must be understood and managed effectively. Recognizing the Financial Implications of Technical Debt Technical debt has financial implications, just like monetary debt payments. Developers can choose to pay interest on their technical debt in the form of additional time and effort required to understand and modify poor structured code.  Conversely, developers can pay down the debt by refactoring poorly designed code to reduce future effort. Ozkaya suggested that understanding the financial model implied by the "debt" metaphor can help establish the structural aspect of debt. These financial implications suggest the following questions that agile development teams must consider: What is the "interest rate" that organization signs up for when incurring technical debt? Can this interest rate be controlled? What is the period of the loan? What is it we’re borrowing? Time? Or, other opportunities we need to bring to bear when managing timeline of loan? How do we create a realistic repayment strategy? Identifying What Constitutes Technical Debt Much of the existing literature on technical debt focuses on code-level issues, such as reducing the time needed to modify software functions, add new features, or fix bugs. Ozkaya said it’s also important for organizations to consider how to best describe technical debt from architecture- and system-level perspectives. The SEI focuses on managing debt as an agile software architecture strategy. Specifically, SEI researchers are focusing on identifying any implications to the cost of architectural changes. Often when a particular symptom in a system is described as technical debt, it’s not just the code that is bad, but it’s also accumulating problems that happen in terms of architectural changes that have occurred throughout the system’s development. To establish a common understanding of the term "technical debt," Ozkaya referenced the taxonomy of technical debt created by Steve McConnell. To date, much work has focused on McConnell’s "Type 1" debt, which is unintentional and non-strategic.  This type of technical debt often results from poor design decisions and poor coding. Ozkaya specifically drew attention to the second type of debt described by McConnell: intentional and strategic, optimized for the present and the future. This "Type 2" debt can occur in an agile software development lifecycle when trying to accelerate development from a perspective that requires optimizing for short-term goals, such as shipping a product with known shortcomings to gain or protect market share. What is crucial when incurring Type 2 debt is to have a process for revisiting and reworking these short-term shortcuts to ensure system longevity. Ozkaya also highlighted Jim Highsmith’s prescription for managing technical debt.  Highsmith focuses on understanding and monitoring the accumulating cost of change as a result of technical debt. As years or iterations go by and new functions are added or new technologies upgraded, the cost of change can start to increase dramatically. Lastly, Ozkaya referenced Philippe Kruchten’s perspective on technical debt, which focuses on emphasizing a value perspective on system development. Value could include features that have immediate benefit to stakeholders. Value can also be negative, such as defects that must be resolved. Most of the time, however, value that goes unrecognized are invisible aspects of the software, which are often architectural features that enhance the system when done well, but incur technical debt when done poorly. Tracking and Analyzing Debt Ozkaya presented three strategies for managing technical debt: Do nothing. When using this approach, it’s important to understand the implications (both technical and economic) of  "doing nothing." Replace the whole system. In some cases, this approach might have high cost and risk associated with it; in others, it might be precisely what is needed. Incremental refactoring (commitment to invest). An explicit focus on architectural agility becomes an instrument in this approach. In large-scale software-reliant systems, the number of years spent before a system is launched is often detrimental to success since gaps in requirements and performance are not detected until very late in the lifecycle, when they are expensive to remedy. In such instances, using technical debt as a strategy and dividing the system delivery into chunks might be advantageous. Ozkaya told the audience that eliciting and quantifying the impact of technical debt with pay-back strategies is not yet a repeatable engineering practice. Factors to consider in quantifying debt include tracking defects, changing velocity (what actually got done during an agile iteration versus what was planned), and the cost of rework. These indicators could be mapped into the cost of development, which yields a greater understanding of the value of paying back technical debt versus not paying it back. Ozkaya’s work focuses on quantifying technical debt, which can include code, though Ozkaya is most interested in quantifying debt early in the lifecyle, when code analysis alone may not provide enough direction. Specifically, Ozkaya’s research focuses on understanding the right set of architectural models that can be used seamlessly within agile software development methods to provide feedback to development teams and help them understand the impact of rework. Ozkaya stressed that this rework might not be planned for, but could resurface as a change of requirements or technology. Technical Debt Tools and Analysis Among those organizations interested in managing technical debt, there is an increasing focus on tools for conducting structural analysis. Trends show increasing sophistication, support for structural analysis in addition to code analysis, and the first steps toward analyzing the financial impact of technical debt by relating structural analysis to cost and effort for rework. Several architecture-related capabilities also exist, including architecture visualization techniques, such as dependency structure matrix, conceptual architecture, architectural layers, and dependence growth architecture quality analysis metrics, such as component dependencies, cyclicity, architectural rules compliance, and architectural debt architecture compliance checking, such as defining design rules and ensuring that they are not broken, for example disallowing communication between certain components architecture sandboxing, such as providing features to enable easier discovery of the current architecture of the software, which may include navigating the file structure and moving components around easily Deciding to Pay Down Debt Ozkaya said the main motivation of structured technical debt analysis—and the emerging analysis tools—is to help organizations develop strategies for systematically paying down their debt. These strategies involve eliciting business indicators in a system that could be defined for a particular application domain and determining how those indicators are managed. Example indicators include increasing amount of defects. While this indicator may seem obvious, in some systems with many stakeholders, it involves an inability to deliver the system. slowing rate of velocity. At the first sign of slowing rate of velocity (the rate at which planned requirements for the iteration are being fulfilled) it could be easy to analyze implications (for example during an a sprint retrospect) and create an alternative strategy changing business and technology context. Often, organizations don’t realize that a change in business and technology could result in technical debt in a system that has been perfectly fine heretofore a future business opportunity. The need to embrace new business opportunities could motivate the need to rework the system time to market. Time to market is another key indicator to consider when deciding to pay down technical debt or not For large-scale software-reliant systems, adding technical debt to the backlog and continuously monitoring that debt should be common practice. Development teams can determine strategies for addressing and monitoring technical debt appropriate for the organization. For example, strategies could involve amortizing by 10 percent or conducting a dedicated iteration that focuses on paying back the debt. Ozkaya also pointed to a recent Crosstalk article she co-authored that highlights architectural tactics involved in strategically managing technical debt. The principles of both agile software development and software architecture improve the visibility of project status and offer better tactics for risk management.  These principles help software teams develop higher-quality features on time frames and on budget. The article described three tactics: aligning feature-based development and system decomposition, creating an architectural runway, and using matrix teams and architecture. Harmonious use of these tactics is critical, especially in large-scale DoD systems that must be in service for several decades, are created by multiple contractor teams, and have changing scope due to evolving technology and emerging needs. Future Areas of Research In describing future areas of SEI research on the strategic management of architectural technical debt, Ozkaya pointed out that this topic succinctly communicates key issues observed in large-scale, long-term projects, including solving optimization problems.  In some cases focusing on optimizing for the short-term puts the long-term into economic and technical jeopardy when debt is unmanaged creating appropriate design shortcuts. Design shortcuts can give the perception of success, but cannot focus on the present alone, but need to consider future iterations and plan accordingly modeling software development decisions. Decisions concerning architecture should be continuously analyzed and actively managed since they incur cost, value, and debt. SEI research is focusing on opportunities for developing and quantifying effective payback strategies In conclusion, Ozkaya recommended several immediate steps that organizations should take to managing technical debt: Make technical debt visible, even if it’s just acknowledging that there is a problem Differentiate strategic, structural technical debt from unintended debt incurred as a result of factors like low code quality, bad engineering, or practices that have not been followed Bridge the gap between the business and technical sides Associate technical debt with risk and track it explicitly What's Ahead The first posting in this series on the SEI Agile Research Forum summarized discussions by Anita Carleton, director of the SEI’s Software Engineering Process Management program, and Teri Takai, chief information officer for the DoD. Carleton provided an overview of the forum and discussed areas where SEI work is focused on applying agile methods at-scale. Takai then discussed how agile methods have been introduced into the DoD software acquisition and development environment. The second posting summarized discussions by Mary Ann Lapham, a senior researcher in the SEI’s Acquisition Support Program, who highlighted the importance of collaboration with end users, as well as among cross-functional teams, to facilitate the adoption of agile approaches into DoD acquisition programs. Our next posts in this series will summarize discussions of two SEI researchers, including myself, at the Agile Research Forum who examined the following aspects of applying agile methods at-scale in mission-critical development environments: James Over noted that lack of teamwork can critically impede agility. He advocated, among other principles, the building of self-managed teams, planning and measuring project process, designing before building, and making quality the top priority for achieving agility at-scale. Finally, I wrapped up the forum with a discussion on the importance of applying agile methods to crucial common operating platform environments (COPEs) at the DoD. I explained how agile methods can encourage more effective collaboration between users, developers, testers, and certifiers to help the DoD successfully build integrated, interoperable software systems. We look forward to hearing your thoughts on applying agile at-scale in the comments section below. Additional Resources To view the Crosstalk article, Architectural Tactics to Support Rapid and Agile Stability, please visitwww.crosstalkonline.org/storage/issue-archives/2012/201205/201205-Bachmann.pdf.To visit the International Workshop on Managing Technical Debt workshops website, please visitwww.sei.cmu.edu/community/td2012/.To view the Hard Choices Board Games website, please visitwww.sei.cmu.edu/architecture/tools/hardchoices/.To view blog posts about technical debt by Ozkaya and other SEI researchers, please visithttp://blog.sei.cmu.edu/archives.cfm/category/technical-debt.