By Nanette Brown, Senior Member of the Technical StaffResearch, Technology, and System Solutions program Occasionally this blog will highlight different posts from the SEI blogosphere. Today’s post is from the SATURN Network blog by Nanette Brown, a visiting scientist in the SEI’s Research, Technology, and System Solutions program. This post explores Categories of Waste in Lean Principles and Architecture, and takes an in-depth look at three of the eight categories of waste (defects, overproduction, and extra complexity) from the perspective of software development in general and software architecture in particular. Read more…

By Bill Novak, Senior Member of the Technical Staff, SEI Acquisition Support Program, Air Force Team Background: Over the past decade, the U.S. Air Force has asked the SEI’s Acquisition Support Program (ASP)  to conduct a number of Independent Technical Assessments (ITAs) on acquisition programs related to the development of IT systems; communications, command and control; avionics; and electronic warfare systems. This blog post is the first in a series that identifies common themes across acquisition programs that we identified as a result of our ITA work. This post explores the first theme: misaligned incentives, which occur when different individuals, groups, or divisions are rewarded for behaviors that conflict with a common organizational goal.   When performing the ITAs, the Air Force asked us to: Analyze the most frequently recurring acquisition issues and their possible root causes across multiple ITAs Examine the events, trends/patterns, and underlying structures present in the conducted ITAs Identify attributes of projects that are most likely to result in specific issues Provide initial recommendations, where possible, that could address and mitigate the root causes As part of our analysis, we interviewed people that had been on those teams and looked at documents and other information that we had gathered. We then collated all the findings that had come out of those programs to identify common themes and related issues that we observed. Out of that process came many different findings that we categorized into several overarching themes. We also found a series of emerging trends across this sample. While the sample space is specific to the U.S. Air Force, the trends are representative of what we’ve seen across all types of acquisition programs in the Department of Defense (DoD). The First Theme: Misaligned Incentives There is a saying: Individually optimal decisions often lead to collectively inferior solutions. Or, to put it more simply: If everyone acts in what they believe to be their own best interests, as opposed to the group’s best interests, the overall result for the group may be suboptimal—and in some cases, catastrophic. It is often the case in acquisition that individuals face situations where their incentives may not align with broader team objectives.  Likewise, team objectives may not align with broader organizational objectives. One example of this type of problem is the situation where a lengthy program duration further extends the schedule (also known as "Longer Begets Bigger"). While lengthy program duration enables the creation of greater capability, it also incentivizes the use of less-mature technology to avoid obsolescence at deployment. Likewise, it incentivizes requirements scope "creep" due to changing threats and new technologies while the program is in development. Although minimizing the growth of program schedule and cost should ideally be in everyone’s interests, there are conflicting incentives for stakeholders to do just the opposite; ostensibly to deliver a better, more capable system to the warfighter. Another example is the "Bow Wave Effect." In spiral development, there can be an incentive to postpone riskier tasks that were planned for an early spiral (originally intended to reduce risk) to a later spiral, in favor of doing simpler tasks up-front. The easier tasks will show good progress, making the program’s cost and schedule performance look better in the near-term. This deferral strategy increases risk in later spirals, however, by delaying complex development to a future point when there is less flexibility for change and less room in the schedule to complete the work successfully.  The short-term interests of good cost/schedule performance thus often take precedence over the longer-term interests of successful deployment. Misaligned incentives commonly occur in the absence of proper rules that control the rewards or penalties for participants. The underlying principle is that unless the rules incentivize them to do otherwise, people tend to act in their own self-interest. Two common types of misaligned incentives are those in which (1) an individual's interests are traded off against the group's interests and (2) long-term interests are traded off against short-term interests. If some stakeholder goals conflict with program goals, then either contractor self-interest (such as making more money) or Program Management Office (PMO) self-interest (such as making the program last longer) may drive decision-making. Neither situation is in the best overall interest of the program or the DoD. So the question is how can misaligned incentives in acquisition be addressed? The fact is that problems relating to misaligned incentives have been the subject of intensive study in many different fields ranging from social psychology, game theory, and behavioral economics. Many approaches to resolving specific types of incentive problems have been developed, both by using new theory and by identifying past strategies that have been used in different domains to successfully deal with these issues. When, as participants in an acquisition program, we find ourselves facing instances of misaligned incentives—and there are many—the goal is to try to align them. Not all incentives, however, are within our "sphere of influence" as engineers and managers. Some are inherent in the governance (i.e., the policies and regulations) that we operate within and can't be changed easily. When this is the case, one of the best ways to mitigate the consequences is simply to recognize their existence. Knowing what lies ahead allows managers to make a compelling case for considering workarounds and other alternative options. Ultimately, however, if misaligned incentives are not addressed by either the PMO, its parent organization the Program Executive Office (PEO), or the DoD, it can lead to such situations as PMOs that may support  the continuation of high-risk, poorly progressing programs due to possible impacts on incomes and careers "Cost-Plus" contracts that encourage longer programs because it means more revenue to the contractor Users demanding non-essential requirements and capabilities because they bear little cost for doing so Reviewing the alignment of the incentives acting on an acquisition program can be revealing. Work done by the SEI in Acquisition Archetypes illustrates some of these issues, and recommends some approaches for addressing them. Understanding the incentives will expose opportunities for improving governance by changing the rewards to bring the goals of the various parties into better alignment, and reduce conflict among stakeholder groups.   We don’t have enough space to discuss specific solutions to misaligned incentives in this posting, but will delve into this topic in future postings.  Future postings will also report on research we are conducting that combines characterizations of types of misaligned incentives, such as "social dilemmas," with complex system modeling. This combination will align the incentives so acquisition staff can make decisions that produce better program outcomes. This is the first post in an ongoing series examining themes in acquisition. Other themes that will be explored in the series include the need to sell the program, the evolution of science projects, and the move towards common infrastructure and joint programs New installments in the series will be published over the next several months on the SEI blog. Additional Resources: Check back for a forthcoming SEI Special Report An Analysis of Recurring Issues Found Across 12 U.S. Air Force Software-Reliant Acquisition Programs.

By Robert Ferguson Senior Member of the Technical Staff Software Engineering Process Management Program The Government Accountability Office (GAO) has frequently cited poor cost estimation as one of the reasons for cost overrun problems in acquisition programs. Software is often a major culprit. One study on cost estimation by the Naval Postgraduate School found a 34 percent median value increase of software size over the estimate.  Cost overruns lead to painful Congressional scrutiny, and an overrun in one program often leads to the depletion of funds from another.  This post, the first in a series on improving the accuracy of early cost estimates, describes challenges we have observed trying to accurately estimate software effort and cost in Department of Defense (DOD) acquisition programs, as well as other product development organizations. Periodically, the SEI is called to review a program’s software estimate usually because two independently generated estimates are far apart - sometimes by a factor of 10 or more.  Such disparate results will not pass any of the official milestone reviews and can delay program startup by several months. The frequency of this problem increased with 2008 changes in acquisition regulations by the DOD that require a full life-cycle cost estimate for review at Milestone A, which occurs at the end of the Material Solution Analysis Phase (For more information about Milestone A, see the Integrated Defense Life Cycle Chart for a picture and references in the "Article Library"). Formal acceptance of the Milestone A review is signified by the Acquisition Decision Memorandum (ADM). The ADM is required by law in order to issue a Request for Proposal (RFP) to contract for the Technology Development Phase(TDP). Before describing our approach, which we will do in the second post in this series, it’s important to understand and evaluate the traditional methods of preparing estimates for acquisition programs. Typically, estimators review the available program information, which includes the following documents: An Analysis of Alternatives (AOA) describing the proposed solution. An Initial Capabilities Document (ICD) and various strategy documents supporting a RFP. Preliminary plans for systems engineering, test, and evaluation and similar early planning documents. Estimators then seek out available cost estimation relationships (CERs) and data from past programs.  The estimators must determine which analogies make the most sense.  They then apply their expert judgment to prepare a single "most likely" value for size, cost and schedule for each major subsystem, which is often called a "point estimate." After computing a point estimate, the estimators then add a range, say +/- 25 percent, to the point estimate to account for future changes.  These estimates and the background information are delivered to the service (e.g., Army, Navy, Air Force) cost estimation center and the Cost Assessment & Performance Evaluation (CAPE) office for independent review.  The various estimates and plans become the content for review by the Milestone Decision Authority (MDA), which determines readiness for the TDP and issues the ADM. Reviewers of the estimate will make a careful examination of the assumptions made by the estimators. Consequently, the program estimate must reflect possibilities for future program change and must provide ranges for possible costs and schedule duration.  Potential changes in technology, program structure, mission, and contract must be considered simultaneously.  If the estimates are not reasonably close, the MDA is not likely to approve. In our experience, estimators and reviewers of estimates who use traditional methods of cost estimation are presented with the following challenges because the nature of the information available prior to Milestone A does not correspond well to the input required for these methods: The program will not have a detailed requirements document, making it hard to estimate scale or size.  Staff may not know the specific technologies that will be needed, meaning they may not know the tasks required and the number of trade studies needed. The program manager does not know who will perform the development work, so productivity cannot be estimated accurately. Estimators may not know what skills will be required, which makes it hard to determine staff training requirements. Our goal is to develop a method that overcomes the challenges with traditional estimation methods by identifying and expressing the potential changes and ranges of estimating inputs in terms of a probability model. Automation is a key factor in the model so that re-estimating can be quickly performed as the program discovers which changes become realities and which can be ignored. My next posting will describe research that the SEI’s Software Engineering Measurement & Analysis initiative is conducting to improve the accuracy of early estimates (whether it’s a DOD acquisition program or commercial product development work) and ease the burden of additional re-estimations during the program lifecycle. Additional Resources: For more information about the Software Engineering Measurement & Analysis Initiative, please visitwww.sei.cmu.edu/measurement/index.cfm

By Bill Novak, Senior Member of the Technical Staff, SEI Acquisition Support Program, Air Force Team Background: The U.S. Air Force has sponsored a number of SEI Independent Technical Assessments (ITAs) on acquisition programs that operated between 2006 and 2009. The programs focused on the development of IT systems, communications, command and control, avionics, and electronic warfare systems. This blog post is the second in a series that identifies four themes across acquisition programs that the SEI identified as a result of our ITA work. Other themes explored in the series include misaligned incentives, the evolution of science projects, and common infrastructure and joint programs. This post explores a related second theme, the need to sell the program, which describes a situation in which people involved with acquisition programs have strong incentives to "sell" those programs to their management, sponsors, and other stakeholders so that they can obtain funding, get them off the ground, and keep them sold. The Second Theme: The Need to Sell the Program Many studies have noted that defense suppliers are consolidating (becoming larger and fewer), and that future DoD budgets will likely shrink. To use scarce resources more effectively, DoD acquisition programs are increasingly combining multiple capabilities into single systems, meaning that there are fewer programs for which the smaller number of defense suppliers must compete. The consequence is that acquisition program awards become "must-win" competitions for those suppliers. In such an environment, it becomes critically important for acquisition program participants to "sell the program" to their management, sponsors, and other stakeholders so that they can obtain and keep their funding. A recent study on reforming defense acquisition found that in many situations "the acquisition culture has become an environment that promotes ‘selling’ programs and includes behavior fraught with unfounded optimism and parochialism." Such an environment creates incentives for acquisition program management and staff to sell the program and keep it sold by exaggerating the value of the system to the user or warfighter to raise its perceived importance underestimating the system’s cost to make the price tag more palatable defining ambitious requirements that promise substantial jumps in capability, to increase the system’s attractiveness to stakeholders impact its viability downplaying adverse information about the program or system delaying risky or complex tasks that could cause poor results or failure deferring longer-term investments (such as sustainment planning) that are critical, but provide no visible near-term "selling point" for the system minimizing real-world test and demonstration activities (such as comprehensive operational tests) that might reveal issues and using more advanced, and (unfortunately) sometimes less mature, technology to promise superior system capability. So why is the SEI looking at these issues? Because the role of software in defense programs has increased dramatically and continues to rise, and thus complicates these issues. For example, the complexity of software estimation actually promotes underestimation, since it increases the inherent uncertainty of the cost.  Also, the ease of deploying upgraded software to already fielded systems to improve capability is very attractive—but requires even more up-front sustainment planning, rather than less. Contractors also often have a vested interest in "selling the program" through underbidding and other activities. If the program is either not awarded, or cancelled, there is no income. As the GAO noted last year, there are "…prevailing pressures to force programs to compete for funds by exaggerating achievable capabilities, underestimating costs, and assuming optimistic delivery dates." Our concern is that in situations when all these incentives come together, the annual funding process creates a competition in which "success" is measured more in terms of the ability to obtain the full amount of the next year’s funding, rather than delivering promised capabilities on time to the warfighter. In short, the goal becomes maintaining the perception of high value and good progress for as long as possible. The resulting push to "sell" programs is a root cause for many of the recurring acquisition problems that we see in our work, including acquisition programs that run over schedule and budget. We can see how certain actions, such as underestimating costs and overpromising results, directly lead to schedule pressure due to the cost of development staff.  Likewise, the use of less mature technology raises risk, which frequently leads to schedule and cost overruns. When taken together, these incentives all push programs in the same direction: running over cost and schedule, underperforming, reducing functionality, and diminishing quality. The existence of an incentive to follow a particular course of action does not necessarily mean that the incentive will be successful in producing that behavior. Our goal at the SEI is to gain a better understanding of this problem so that incentives in acquisition can be aligned in such a way that they benefit the individual and the group, as well as the government and country. Acquisition program management and staff should not be forced to choose between what’s best for them and their program versus what may be best for their service or country. When faced with these kinds of misaligned incentives, most people believe they have the integrity to "do the right thing." When implicit incentives are built into the acquisition system, however, they encourage the acquisition community to overstate value, understate costs, use immature technology, and minimize risks and problems. When these incentives exist, there is often pressure to act accordingly, despite the best of intentions by participants in the process. As we look toward addressing this particular aspect of misaligned incentives, we face the same challenges that we laid out in the blog entry on the first overarching theme in acquisition. We need to start addressing each of these implicit incentives in the acquisition system. We begin by shining a light on them and recognizing them as counter-productive influences that often weaken—not strengthen—DoD readiness and effectiveness. The next step is to help inform the acquisition community about these issues, show them how they occur, and equip them with a toolkit of methods to counteract them to better serve warfighters and taxpayers. New research at the SEI is applying analysis and modeling tools to characterize acquisition behaviors and assess the effectiveness of different techniques for aligning incentives. We are developing interactive exercises that can be used in both classroom and eLearning contexts to help the acquisition community find ways of better managing the development of our essential defense systems. This is the second in an ongoing series examining themes in software-reliant acquisition. New installments in the series will be published over the next several months on the SEI blog, where a new post is published every Monday morning. Additional Resources:Check back for a forthcoming SEI Special Report An Analysis of Recurring Issues Found Across 12 U.S. Air Force Software-Reliant Acquisition Programs.

By Nanette Brown, Senior Member of the Technical StaffResearch, Technology, and System Solutions program Occasionally this blog will highlight different posts from the SEI blogosphere. Today’s post is from the SATURN Network blog by Nanette Brown, a visiting scientist in the SEI’s Research, Technology, and System Solutions program. This post, the second in a series on lean principles and architecture, takes an in-depth look at the waste of waiting and how it is an important aspect of the economics of architecture decision making. To read more...

By Robert Ferguson Senior Member of the Technical Staff Software Engineering Process Management Program The Government Accountability Office (GAO) has frequently cited poor cost estimation as one of the reasons for cost overrun problems in acquisition programs. Software is often a major culprit. One study on cost estimation by the Naval Postgraduate School found a 34 percent median value increase of software size over the estimate.  Cost overruns lead to painful Congressional scrutiny, and an overrun in one program often cascades and leads to the depletion of funds from others. The challenges encountered in estimating software cost were described in the first post of this two-part series on improving the accuracy of early cost estimates.  This post describes new tools and methods we are developing at the SEI to help cost estimation experts get the right information they need into a familiar and usable form for producing high quality cost estimates early in the life cycle. To help overcome the fact that the data available early in a program lifecycle does not correspond to the input data required for most cost estimation models, our method performs the following steps Identify program execution change drivers (referred to simply as "drivers") that are specific to the program Identify an efficient set of scenarios representing combinations of the driver states Develop a probability model (e.g., a Bayesian Belief Network (BBN)) depicting the cascading nature of the drivers Supplement traditional use of analogy with the BBN to predict the uncertainty of inputs to traditional cost models for each scenario Use Monte Carlo simulation to compute a scenario cost estimate based on the uncertain inputs from the previous step Use Monte Carlo simulation to consolidate the set of scenario cost estimates into a single, final cost estimate for the program The remainder of this post describes each step in more detail. In Step 1 we facilitate a short workshop with various program domain experts to identify how cost is affected by possible drivers, such as changes in program sponsorship or changes in supplier relationships.  Workshop participants first select a "nominal" state of each driver, along with one or more possible alternate states.  Notionally, the alternate states represent future conditions of each driver that will likely impact the program cost.  We selected the Navy-AF Probability-Of-Program-Success (POPS) criteria as a straw-man to kick-start this workshop and expedite the discussion of possible drivers for their program.  POPS contains seventeen categories, each with multiple decision criteria, which are mostly related to program management. We extended the straw-man with several other technical drivers, such as capability based analysis, capability definition, and systems design.  After the drivers and driver states are fully identified, workshop participants subjectively evaluate the probability that each driver state will occur in the future.  To avoid the common pitfalls of eliciting expert judgment of probabilities, we leverage recent published work on the calibration of expert judgment based on the book "How to Measure Anything" by Douglas Hubbard.  Step 2 is based upon techniques for scenario planning.  As Lindgren and Bandhold describe in their book, Scenario Planning - Revised and Updated Edition: The Link Between Future and Strategy, scenario planning has been extensively studied as a means to analyze and manage uncertainty in product development and support strategic planning.  In our use of scenario planning, a scenario consists of the combination of one or more drivers, each in a specific state.   A nominal scenario may therefore be cast as all of the drivers set to their nominal states.   A separate scenario may be cast as a small subset of the drivers, each set to one of their alternate states.   The combinations of drivers and their states can clearly explode as a combinatorial problem.   We employ a driver relationship matrix to subjectively ascertain the most likely cascading situations among the drivers, along with optional use of an orthogonal array (which is a statistical technique that allows a specific sample of scenarios to be evaluated thereby producing a model to explain all remaining scenarios) to produce a representative and efficient set of scenarios to guide cost estimation. In Step 3 we construct a Bayesian Belief Network (BBN), which is a probabilistic model that dynamically represents the drivers and their relationships as envisioned by the program domain experts.   Although initially populated with quantified relationships from the previous driver relationship matrix, this model may be further refined through analysis of quantitative data of the drivers from historical program completions.  Such refinement produces BBN-modeled drivers that go beyond simple binary states of nominal versus non-nominal, to drivers modeled with all of their detailed alternate states and the quantified relationships between the alternate states of different drivers.  This latter approach provides much greater modeled information and more accurate cost estimates compared with traditional statistical regression approaches that only cover a small fraction of the driver state combinations arbitrarily decided in advance.   With BBN’s, the driver states may be flexibly modeled to produce cost estimates, irrespective of which driver state combinations and data are available. In Step 4 the experts examine each scenario and apply their knowledge of past programs to select relevant program and/or component analogies and associated Cost Estimation Relationships (CERs), which are empirical formulas predicting cost based on domain-specific attributes researched over decades of DoD program data.  An example of a CER would be an Air Force CER which predicts cost and labor hours of aircraft development using factors such as aircraft quantity, maximum speed of the aircraft and aircraft weight.  Estimators may identify as many as two dozen CERs for use in the different components and subsystems for a given scenario.  After identifying the analogies and associated CERs, the workshop participants then use the BBN to compute uncertainty distributions for the input factors to the cost estimation model CER’s for each scenario.  The benefit of this approach is that the explicit knowledge of uncertainty of the CER input factors is overtly documented, rather than guessing a single value and trying to discern the final cost estimate answer. Step 5 uses traditional cost models in a Monte Carlo simulation, in which hundreds of thousands of "what-if" hypotheses are calculated using the various uncertain input factor values to estimate cost for a given scenario.   As a result, each scenario will then have a computed cost estimate in the form of a distribution.   This approach provides a confident range of behavior expected for the cost estimate, rather than a single "guestimate" point value.   The immediate benefit is a cost estimate with defined upside and downside uncertainty. Lastly, in Step 6, the set of scenarios and their corresponding cost estimate distributions are consolidated into a single cost estimate distribution using another Monte Carlo simulation.   From this distribution, statements of estimated cost at different confidence levels may then be derived. Our research thus far has focused on evaluating various steps of the method for practicality and effectiveness.  Through an industry pilot of Steps 1 and 2, we examined the typical rich set of possible drivers and their states, along with the combinatorial explosion of the possible scenarios. This pilot enabled us to refine our use of the driver relationship matrix and statistical orthogonal arrays to control the explosion. A subsequent workshop with representatives of the SEI Acquisition Support Program (ASP) enabled us to explore a possible common set of program execution drivers applicable to DoD programs, in addition to evaluating Steps 1-3.   This workshop also enabled us to build an initial BBN model that will be applied in future DoD pilots and become an impetus for more detailed DoD characterization and data collection of program execution drivers. We’ve made consistent progress at each stage of the research thus far and reactions from participants have been positive. Our goal of cost estimate "transparency" mentioned in the first post is certainly being realized.  Recent comments from service cost center staff confirm that the detailed discussion of program execution change drivers and scenarios provides far greater insight to the cost estimate and would significantly assist those who need to review the cost estimates.  Further work must be done to evaluate the accuracy of the resulting estimates and impact of the estimating process.  We’ll keep you posted. Additional Resources:For more information about the Software Engineering Measurement & Analysis Initiative, please visitwww.sei.cmu.edu/measurement/index.cfm

By Grace Lewis, Senior Member of the Technical StaffResearch, Technology, and System Solutions Program The Department of Defense (DoD) is increasingly interested in having soldiers carry handheld mobile computing devices to support their mission needs. Soldiers can use handheld devices to help with various tasks, such as speech and image recognition, natural language processing, decision-making and mission planning. Three challenges, however, present obstacles to achieving these capabilities. The first challenge is that mobile devices offer less computational power than a conventional desktop or server computer. A second challenge is that computation-intensive tasks, such as image recognition or even global positioning system (GPS), take a heavy toll on battery power. The third challenge is dealing with unreliable networks and bandwidth. This post explores our research to overcome these challenges by using cloudlets, which are localized, lightweight servers running one or more virtual machines (VMs) on which soldiers can offload expensive computations from their handheld mobile devices, thereby providing greater processing capacity and helping conserve battery power. This leverage of external resources to augment the capabilities of resource-limited mobile devices is a technique commonly known as cyber-foraging.  The use of VM technology provides greater flexibility in the type and platform of applications and also reduces setup and administration time, which is critical for systems at the tactical edge.  The term tactical edge refers to systems used by soldiers or first responders that are close to a mission or emergency executing in environments characterized by limited resources in terms of computation, power and network bandwidth, as well as changes in the status of the mission or emergency. Cloudlets are located within proximity of handheld devices that use them, thereby decreasing latency by using a single-hop network and potentially lowering battery consumption by using WiFi instead of broadband wireless which consumes more energy. For example, a cloudlet might run in a Tactical Operations Center (TOC) or a Humvee. From a security perspective, cloudlets can use WiFi networks to take advantage of existing security policies, including access from only specific handheld devices and encryption techniques. Related work on offloading computation to conserve battery power in mobile devices relies on the conventional Internet or environments that tightly couple applications running on handheld devices and servers on which computations are offloaded.  In contrast, cloudlets decouple mobile applications from the servers. Each mobile app has a client portion and an application overlay corresponding to the computation-intensive code invoked by the client. On execution, the overlay is sent to the cloudlet and applied to one of the virtual machines running in the cloudlet, which is called dynamic VM synthesis. The application overlay is pre-generated by calculating the difference between a base VM and the base VM with the computation-intensive code installed. The only coupling that exists between the mobile app and the cloudlet is that the same version of the VM software on which the overlay was created must be used. Since no application-specific software is installed on the server, there is no need to synchronize release cycles between the client and server portions of apps, which simplifies the deployment and configuration management of apps in the field. Dynamic VM synthesis is particularly useful in tactical environments characterized by unreliable networks and bandwidth, unplanned loss of cyber foraging platforms, and a need for rapid deployment. For example, imagine a scenario where a soldier needs to execute a computation-intensive app configured to work with cloudlets. At runtime, the app discovers a nearby cloudlet located on a Humvee and offloads the computation-intensive portion of code to it. Due to enemy attacks, network connectivity, or exhaustion of energy sources on the cloudlet, however, the mobile app is disconnected from the cloudlet. The mobile app can then locate a different cloudlet (e.g., in a TOC) and—due to dynamic VM synthesis—can have the app running in a short amount of time, with no need for any configuration on the app or the cloudlet. This flexibility enables the use of whatever resources become opportunistically available, as well as replacement of lost cyber-foraging resources and dynamic customization of newly-acquired cyber foraging resources. As part of our research, we are focusing on face recognition applications. Thus far we have created an Android-based facial recognition app that performs the following actions: It locates a cloudlet via a discovery protocol It sends the application overlay to the cloudlet where dynamic VM synthesis is performed. It captures images and sends them to the facial recognition server code that now resides in the cloudlet. The application overlay is a facial recognition server written in C++ that processes images from a client for training or recognition purposes. When in recognition mode, it returns coordinates for the faces it recognizes, as well as a measure of confidence. The first version of the cloudlet is a simple HTTP server that receives the application overlay from the client, decrypts the overlay, decompresses the overlay, and performs VM synthesis to dynamically set up the cloudlet. The first phase of our work has focused on creating the cloudlet prototype described above. In the second phase we will conduct measurements to see if computations in a cloudlet provide significant reductions in device battery power. In addition, we will gather measurements related to bandwidth consumption of overlay transfer and VM synthesis to focus on optimization of cloudlet setup time. Assuming we are successful, our third phase will create a cloudlet in the RTSS Concept Lab to explore other ways to take computation to the tactical edge. As part of our research, we are collaborating with Mahadev Satyanarayanan, the creator of the cloudlet concept and a faculty member at Carnegie Mellon University’s School of Computer Science. We will be blogging about the progress of our research in future posts. Additional Resources: To read more about the cloud computing research conducted by the SEI’s System of Systems team, please visit www.sei.cmu.edu/sos/research/cloudcomputing/ To view an SEI Webinar on cloud computing, please visit www.sei.cmu.edu/library/abstracts/webinars/Cloud-Computing.cfm

By Donald Firesmith,Senior Member of the Technical Staff Acquisition Support Program In our research and acquisition work on commercial and Department of Defense (DoD) programs ranging from relatively simple two-tier data-processing applications to large-scale multi-tier weapons systems , one of the primary problems that we see repeatedly is that requirements engineers tend to focus almost exclusively on functional requirements and largely ignore the so-called nonfunctional requirements, such as data, interface, and quality requirements, as well as technical constraints. Unfortunately, this myopia means that requirements engineers overlook critically important, architecturally-significant, quality requirements that specify minimum acceptable amounts of qualities, such as availability, interoperability, performance, portability, reliability, safety, security, and usability. This blog post is the first in a series that explores the engineering of safety- and security-related requirements. Quality requirements are essential to a system’s architecture and its acceptability by stakeholders. There are several reasons, however, why quality requirements are rarely well specified. Functional requirements are central to how stakeholders tend to think about the system (i.e., what functions the systems performs for its users). Popular requirements engineering techniques, such as use case modeling, are effective for identifying and analyzing functional requirements. Unfortunately, these techniques are inadequate and inappropriate for non-functional requirements, which include quality requirements, as well as interface requirements, data requirements, and architecture/design constraints. By specifying how well the system performs its functions, quality requirements logically follow functional requirements. Most acquisition programs do not explicitly use quality models, which are used to define the different types of quality, their units of measure, and associated metrics (e.g., as defined in ISO/IEC 9126-1 Software Engineering - Product Quality - Quality Model). Without relatively complete quality models, stakeholders and developers are often unaware of—and tend to overlook—the many types of quality. It is also hard for many stakeholders to specify the required level of these qualities. Requirements engineers rarely receive any training in identifying and specifying quality requirements and thus have far less experience engineering them because they are often considered the responsibility of specialty engineering groups, such as reliability, safety, security, and usability (human factors). While many types of quality requirements are important, safety and security requirements are two of the most vital; almost all major commercial and DoD systems have significant safety and security ramifications, and many are safety- and security-critical. It is far better to build safety and security into a system than to add them once a system’s architecture has been completed, much less after the system exists and has been fielded. Yet system requirements rarely specify how safe and secure a system must be to adequately defend itself and its associated assets (people, property, the environment, and services) from harm. Far too often, requirements do not specify what accidents and attacks must be prevented, what types of vulnerabilities the system must not incorporate, what hazards and threats it must defend against, and what the maximum acceptable safety and security risks are. How big is this problem? On production projects, poor requirements lead to budget and schedule overruns, missed or incorrectly implemented functionality, and systems that are delivered but never used.  According to Nancy Leveson, a respected expert in software safety, up to 90 percent of all accidents are caused, at least in part, by poor requirements. For example, conventional requirement documents typically do not specify what systems should do in unlikely situations when valuable assets are harmed accidents and attacks occur system internal vulnerabilities exist system external abusers exploit these vulnerabilities safety hazards and security threats exist and safety and security risks are high Requirements also rarely specify that the system must detect when these safety- and security-related events occur or conditions exist. Similarly, the requirements often don’t specify what the system must do when it detects them. To summarize, requirements typically do not adequately specify the safety and security-related problems the system must prevent, the system’s detection of these problems, and how the system must react to their detection. The next post in this series will discuss the obstacles that acquisition and development organizations face when engineering safety- and security-related requirements. Our final post in the series will present a collaborative method for engineering these requirements based on a common ontology of the concepts underlying safety and security a clear model (including proper definitions) of the different types of safety- and security-related requirements and a shared set of safety and security analysis techniques that is useful to engineers from both disciplines.   Additional Resources: To view a tutorial on engineering safety-and security-related requirements for software-intensive systems, please visit www.sei.cmu.edu/library/abstracts/presentations/icse-2010-tutorial-firesmith.cfm To read an SEI technical note on the common concepts underlying safety, security, and survivability engineering, please visitwww.sei.cmu.edu/library/abstracts/reports/03tn033.cfm To read an SEI technical report on the Security Quality Requirements Engineering (SQUARE) method for engineering security requirements, please visit www.sei.cmu.edu/library/abstracts/reports/05tr009.cfm

By Bill Novak, Senior Member of the Technical Staff, SEI Acquisition Support Program, Air Force Team Background: Over the past decade, the U.S. Air Force has asked the SEI’s Acquisition Support Program (ASP)  to conduct a number of Independent Technical Assessments (ITAs) on acquisition programs related to the development of IT systems, communications, command and control, avionics, and electronic warfare systems. This blog post is the third in a series that enumerates common themes across acquisition programs that we identified as a result of our ITA work. Other themes explored in this series include misaligned incentives, the need to sell the program, and common infrastructure and joint programs. This post explores the third theme in this series, the evolution of "science projects," which describes how prototype projects that unexpectedly grow in size and scope during development often have difficulty transitioning into a formal acquisition program. The Third Theme: The Evolution of "Science Projects" A growing theme in acquisition is the increasing prevalence of programs that are often initially called "science projects," and the difficulties associated with evolving such efforts into larger systems and more formal acquisition efforts. The name "science project" refers to a program that starts out small—often as an experimental development or prototype system—so that it can clarify user requirements better understand the problem produce a compelling "proof of concept" to help solicit funding  and ‘sell’ the program The recurring dynamic of science projects has received little discussion in acquisition literature, even though the pattern is increasingly common. Due in part to urgent demands for new technologies in theaters such as Iraq and Afghanistan, a quarter of the programs that we assessed for the Air Force study mentioned above could be characterized as having begun as science projects. Many defense programs that started out as science projects ultimately produced important advances in technology that leapfrogged our adversaries and gave American warfighters a critical edge in conflicts around the world. In many cases the ability to quickly develop and deploy a new technology has been key to adapting effectively to rapidly changing conditions and threats. The question is not whether we need science projects—but how we can most effectively—and sustainably—move the technology from a prototype into the hands of the user or warfighter. Science projects are often initiated—and even frequently managed—by their user community, who are often experts in the relevant domain area but lack expertise in software engineering and management. Some projects may be organized as less formal "in-house" developments, while others are begun using contractors who specialize in building advanced prototype systems. Still others are the product of laboratories or Advanced Concept Technology Demonstration (ACTD) efforts. In any case, science projects often evolve organically without the structure or investment in design that should be put into building a large and mission-critical software-reliant system. These short-cuts sometimes occur because there isn’t yet adequate demand for the capability to justify a formal program. Other times there has been a conscious decision to try to develop it as quickly as possible by "flying under the radar" and avoiding the bureaucratic overhead and cost of the formal acquisition process. A potential downside with science projects, however, is that these systems can become the victims of their own success. The natural and laudable instinct of commanders to provide valuable new capabilities to their warfighters in the field as quickly as possible can become a threat to the project.  The initial prototypes of science projects often grow incrementally without a guiding vision or architecture and are tested in the field, often with very positive initial reactions to the system’s new capabilities. Once the value of the system’s capability is recognized by warfighters and other users, demand increases quickly, and the warfighters may soon become unwilling to part with these new capabilities. The systems have already incorporated many new features in response to warfighter needs, with the code base becoming increasingly hard to maintain, and bugs being injected as each change is made or new feature is added. Documentation—rarely a strong point of most prototype development efforts—is no longer an option. In such an environment, science projects are forced to evolve rapidly into full-scale acquisition programs that can reliably deliver a robust production system growth spurt that was neither anticipated nor properly planned for in many cases.  It is at this point that many science projects "hit the wall" with a large, mostly undocumented, convoluted, defect-laden, and unmaintainable code base, unable to make progress at their early development rates. Almost any change made to such a system now will have multiple adverse unintended consequences on performance and robustness. The software—like the plumbing in an old house—can only keep working for so long before the entire system must be scrapped and re-implemented. Sound software engineering practice dictates that when a prototype’s objectives have been met, it shouldn’t be fielded as an operational system directly. Although many aspects of the prototype can and should be reused, systematic development and quality assurance methods are still needed. This evolution can take many forms, including agile approaches, but all such methods still require software expertise and rigor. Instead, what can happen is that a science project tries to transform its prototype into a full-scale system built on top of an unsound foundation.  The system will then often experience problems with robustness, capability, performance, and usability, among other issues. As major new capabilities must be added to the prototype to satisfy the still-growing user demand, the project’s managers see the shortcomings of its evolved design and would like to pause development to re-architect it. Insistent user demand, however, won’t allow the project to "go dark" for months—much less years—to do work that will produce little in the way of visible new features for the end users.  Even if the government were still inclined to discard the prototype and start new development from scratch, the contractor has an incentive to persuade the government to reuse the prototype software as the platform for the new work, thereby making the contractor more indispensable to the future development effort. When the technical aspects encounter difficulties, the management aspects do as well. The project infrastructure (the project team size, its processes, the level of software development and management experience, etc.) that may have been adequate for building a prototype can be inadequate for developing a formal production system intended for wide deployment. These mismatches can mean that most aspects of the project are now inappropriate for their new purposes and must either be discarded or substantially changed and expanded. We’re seeing science projects morphing awkwardly and ineffectively into convoluted production systems more frequently due to the desire to deploy new capabilities to the field that demand new technologies at an ever faster pace. Science projects don’t have to "hit the wall," as long as the needs to scale up both the system design and project organization are recognized and acted upon. Unfortunately, in an environment of scarce funding and chronic schedule pressure, there is a strong temptation to continue development on the same code foundation, with the original project infrastructure, in the often mistaken hope that it will suffice. We collectively need to do a better job of working with science projects to help them be more successful in "crossing the chasm" from prototype development efforts to formal acquisition programs. The challenge is to help acquisition staff identify their situation early on, understand the issues involved, and recognize and act upon the need to scale up both the organization and system architecture as needed to be able to complete a successful acquisition effort. In the blog post, Enabling Agility by Strategically Managing Architectural Technical Debt, my colleague, Ipek Okzaya, describes a promising new methodological advance that could help to mitigate some of these problems.  Likewise, my colleague Rick Kazman describes strategies for architectural documentation that help make the fundamental design rules of science project software more explicit,  thereby helping developers, testers, and maintainers understand the software and work together more effectively. By better understanding how concepts like "technical debt" and "architectural documentation" can help us to more efficiently manage the evolution of software, and apply methods such as refactoring to restructure our code, we are taking important steps toward mastering "science projects." This is the third post in an ongoing series examining themes in acquisition. The first post explored misaligned incentives. The second post explored the need to sell the program. New installments in the series will be published over the next several months on the SEI blog, where a new post is published every Monday morning.

By Donald Firesmith, Senior Member of the Technical Staff Acquisition Support Program Background: In our research and acquisition work on commercial and Department of Defense (DoD) programs, ranging from relatively simple two-tier data-processing applications to large-scale multi-tier weapons systems, one of the primary problems that we see repeatedly is that acquisition and development organizations encounter the following three obstacles concerning safety- and security-related requirements: Safety, security, and requirements engineers typically know little about each others’ disciplines. Safety, security and requirements engineers often reside in separate teams that rarely collaborate when engineering safety- and security-related requirements. Safety and security are viewed as specialty engineering disciplines that are not well integrated into the overall systems engineering process until after the architecture is largely finalized. This is the second in a series exploring the engineering of safety- and security-related requirements. The first post in the series explored problems with quality requirements.  This post takes a deeper dive into key obstacles that acquisition and development organizations encounter concerning safety- and security-related requirements. In the third part of this series, we will introduce a collaborative method for engineering these requirements that overcomes the obstacles identified in this blog. The first obstacle is a lack of understanding of each other’s disciplines. The safety, security, and requirements communities each have their own terminology, methods, techniques, models, and documents. They read their own journals and books, and they attend their own conferences. In short, they form separate stovepipes that rarely interact. Safety engineers know how to perform safety (hazard) analysis, security engineers know how to perform security (threat) analysis, and requirements engineers know how to perform requirements analysis. Unfortunately, they are rarely trained in each other’s disciplines. These three communities have independently developed effective techniques and methods for performing their own analyses, but they are largely unaware of each others’ disciplines. In practice, however, safety techniques and methods are often quite appropriate—with little or no modifications—for performing security analyses and vice versa. This lack of awareness limits the options available to members of these communities and frequently leads to duplication (often inconsistent or incomplete duplication) of each others’ work. Requirements engineers have an additional problem. Although they know how to engineer functional requirements, we have seen many projects where they utilize functional decomposition or use-case modeling to the exclusion of all other requirements analysis techniques. While these approaches may work well for functional requirements, they are not effective for engineering quality requirements, such as safety and security, as discussed in the first blog posting. The second obstacle is a lack of close collaboration among these three types of engineers, which is especially detrimental to safety and security; two sides of the same coin. Safety and security engineering are concerned with preventing negative events and conditions. The primary difference between safety and security is that safety deals with unintentional (accidental) negatives, whereas security deals with intentional (malicious) ones. Although safety and security engineers have different, albeit complementary responsibilities, they are not completely independent. Accidents can cause vulnerabilities that can be exploited during an attack. Likewise, attacks can cause vulnerabilities that lead to accidents. For example, an electrical power outage (accident) could cause the failure of a physical access control system (vulnerability), leaving security doors unlocked so that unauthorized people can enter into secured areas (attack). Similarly, safety critical software could be infected by malware (attack), causing it to fail (vulnerability), which is a hazard that can lead to associated accidents. As mentioned above, however, safety and security engineers rarely interact, so each tends not to appreciate what the other does. As safety and security engineers have recognized the need to prevent accidents and attacks, security engineers are starting to claim parts of safety, while safety engineers are beginning to claim parts of security. The lack of collaboration between the two teams can lead to a duplication of tasks. Stepping on the other’s turf often leads to resentment, rather than recognition of the need for, and value of, collaboration. This brings us to our third and final obstacle: the view that safety and security are specialty engineering areas that need not be incorporated into systems engineering until after the system requirements that drive the architecture are defined and the primary architecture decisions have been made.  This omission means that safety and security engineers are rarely empowered to make architectural decisions that ensure that the actual safety or security requirements are met. It is therefore more than just a problem of not working closely together; it’s that safety and security engineers rarely become involved in the system engineering until after key requirements and architecture decisions have been completed, which is too late. This viewpoint causes safety and security engineers to concentrate on fixing architectural weaknesses, rather than specifying the requirements that would prevent these weaknesses from being incorporated into the architecture. Safety and security engineers also tend to apply industry standard controls, such as shielding and interlocks (safety) and encryption and passwords (security) rather than asking themselves how safe and secure the system needs to be. In other words, what are the safety and security-related requirements that should be driving the architecture? It is critical to build safety and security into the system from the beginning because it is very difficult and expensive to add it to an existing architecture afterwards. Given these obstacles, how can we overcome them to properly engineer safety and security requirements?  The answer is by providing safety, security, and requirements engineers with an effective method for collaborating closely when engineering these requirements.  In the third part of this series, we will introduce a collaborative method for engineering these requirements that overcomes the obstacles identified in this blog. Additional Resources: For more information, please visitwww.sei.cmu.edu/library/abstracts/presentations/icse-2010-tutorial-firesmith.cfm