By Julien DelangeMember of the Technical StaffSoftware Solutions Division Introducing new software languages, tools, and methods in industrial and production environments incurs a number of challenges. Among other necessary changes, practices must be updated, and engineers must learn new methods and tools. These updates incur additional costs, so transitioning to a new technology must be carefully evaluated and discussed. Also, the impact and associated costs for introducing a new technology vary significantly by type of project, team size, engineers’ backgrounds, and other factors, so that it is hard to estimate the real acquisition costs. A previous post in our ongoing series on the Architecture Analysis and Design Language (AADL) described the use of AADL in research projects (such as System Architectural Virtual Integration (SAVI)) in which experienced researchers explored the language capabilities to capture and analyze safety-critical systems from different perspectives. These successful projects have demonstrated the accuracy of AADL as a modeling notation. This blog post presents research conducted independently of the SEI that aims to evaluate the safety concerns of several unmanned aerial vehicle (UAV) systems using AADL and the SEI safety analysis tools implemented in OSATE. At the April 2014 AADL standardization committee meeting in Santa Barbara, CA, Jerome Hugues, a professor at the Institue Supérieur de l’Aéronautique et de l’Espace (ISAE) in Toulouse, France, presented a project that aims to evaluate the safety concerns of several UAV systems using AADL and the SEI safety analysis tools implemented in OSATE. This study’s results were quite a surprise for us: the study was done independently, without any support from the SEI research team, even when safety analysis tools were under development. More impressive was that the team acquired the basics of AADL and its associated error notation very quickly, created the models for different systems, and generated safety validation materials automatically from the models within a month. This experiment shows how quick and easy it is to learn AADL and use its associated tools to analyze a system. I wanted to learn more about this project, and Hugues gave me the opportunity to ask him a few questions: Can you introduce yourself? Please tell us a little bit about where you work and how your organization uses AADL. When and how were you introduced to AADL? I am an associate professor at ISAE, the reference engineering school in aeronautics and space engineering in France. Since 2010, I have served as co-chair of the Advanced Master on Embedded Systems (SM EMS) program at École Nationale Supérieure d'Électronique, d'Électrotechnique, d'Informatique, d'Hydraulique, et de Télécommunications (ENSEEIHT) and ISAE. I specialize in the teaching of software architecture. My research focuses on embedded systems architecture, with a strong focus on the AADL. I have been a member of the AADL standardization committee since 2006, and a member of its steering committee since 2011. I was first introduced to AADL as part of the IST ASSERT project. I remember quite well my first encounter with Bruce Lewis, our AADL "Sherpa" during the Ada-Europe Conference on Reliable Software Technologies in Palma de Mallorca: it was 10 years ago, in June 2004. At that time, we were looking for an architecture design language with clear semantics and a textual representation, all of which were part of the initial requirements of the Society of Automotive Engineers (SAE) AADL standard. Can you introduce/explain your project, its context, and objectives? What challenges did you want to address, and why did you select AADL as the modeling language? As part of my research, I evolved from modeling patterns to modeling distributed, embedded systems and code generation to system engineering concerns. I am interested in a pragmatic solution to define stringent, tool-supported methodologies for the engineering of critical systems. Achieving this solution implies code generation that meets space industry standards as part of the TASTE project, scheduling analysis, and model checking. More recently, I became interested in defining a better coupling between analysis tools and models. Using AADL, as well as being part of the group defining it, provides a perfect context in which to discuss all those concerns and define practical solutions. Figure 1. One of the UAV Systems Used by the ISAE Team Please explain for us how you used AADL in this project. If possible, please give us the nuts-and-bolts details. Recently, the AADL committee started an effort to support reliability analysis as part of modeling and verification activities supported by the language. In the meantime, the SEI developed various tools to support those analyses in the scope of the ARP4761 standard. I became curious about how far we could go with these tools. As head of a master’s degree program at ISAE, I have the chance to teach students with strong backgrounds not only in safety but also in aeronautical engineering. I thus proposed a challenge to my students: model one family of four UAVs that we develop at ISAE using AADL, and perform safety analysis to evaluate the reliability metrics of our UAVs. It started as an exploratory project, yet the results went beyond our expectations. Not only could we abstract the key elements of the UAVs; we could also perform a wide range of analyses using a single model, as opposed to using separate tools. Being an aeronautics school, ISAE is developing prototype UAVs for both research and teaching purposes. These UAVs range from quadrotors (two variants) to fixed-wing planes.  They share a common embedded part, on top of which various command laws, sensors, and actuators are deployed. Our modeling objective was to capture product-line concerns in the form of a library of components, and then to platform specific configurations as specific instances of these building blocks. This first step demonstrated that AADL could capture many of the elements considered when engineering UAVs. The second step was to model faults and their propagation in the architecture. To do so, we defined problematic situations in our systems. Examples of these problematic situations include bad errors, loss of a component, transient errors in communication, etc. The final step was to perform safety analysis. Our team of researchers could directly apply analysis plug-ins bundled with OSATE2, thanks to the SEI,. We performed all kinds of analyses required by the aeronautic industry, including functional hazard assessments, failure mode and effects analysis, fault tree analysis, and more. Figure 2. Overview of the Architecture Model Designed by Dr. Hugues and his Team In addition to yourself, who else is involved in the project, and what are their backgrounds? How do they learn AADL? Have they faced any specific issues or problems? The team——in addition to myself, the team included Nicolas Chatonnay, Julien Bosseboeuf, Jérôme Pierra (MS EMS students), Jacques Lamaison and Alain Hostalier (ISAE/DMIA engineering team)—was made up of three students with backgrounds in telecommunications, two lab engineers, and me. I was the only person with a solid background in AADL and related model-based technologies. My students were exposed to a 20-hour AADL class that I teach. This teaching was combined with the availability of the book Model-Based Engineering with AADL by Peter Feiler and David Gluch. The modeling activity was straightforward thanks to the definition provided in the Error Model Annex defined by the AADL committee. This straightforwardness is the big strength of AADL: the language and its annexes are defined using the engineer’s words, not through specific modeling-language verbiage.  Project meetings were more focused on "what do we want to model?" rather than "how do we model such a pattern?" This focus is definitely a good point when considering a specific modeling notation. What tools did you use to model and analyze the system? We relied on OSATE2.0.5 and the OpenFTA toolset. OSATE maps error models to OpenFTA for further analysis. Figure 3. Generating the Fault Tree from the AADL Model How much time was required to learn AADL basics and error model notation and tools? How long did it take to resolve the complete problem from start to finish? The full project—including learning EMV2, mastering tools, modeling, and finally, analyzing—was done in four weeks. This time span included writing reports detailing all elements. The models represent 2,000 source lines of code of AADL, covering all components, interconnections, and associated error models. What feedback have you received from this project? What is your takeaway?  Students were quite impressed to achieve all the objectives in such a short time frame. Collaborative modeling, analysis plug-ins, and power of expression of the language are satisfactory to conduct safety analysis for UAVs. The ability to achieve these objectives constitutes a good case study for those interested in AADL. We plan to publish this case study along with accompanying notes in the near future Do you have plans for future work on this project? If yes, do those future plans involve AADL?  Now that we have demonstrated safety figures for the UAV, the next big step at ISAE is to start from these models and move toward code generation. ISAE specializes in control systems, and we have design tools for code generation from AADL models and could also integrate Simulink models. It would be great to first generate back code that was previously handwritten, and then ascertain that the quality of the code does not hinder safety. We have an ongoing project to apply theorem provers to generated code. Wrapping Up and Looking Ahead  This experience report shows the ease with which AADL can be used to model a family of systems and apply safety-analysis techniques. ISAE’s experience applying UAVs also demonstrates that AADL analysis tools are relatively straightforward and do not require experienced skills in model-based engineering. As Hugues reports, learning the language does not seem to be an issue and the team learned the language basics and received significant results within a few weeks. The ease with which Hugues’ team learned the language demonstrates that engineers with appropriate backgrounds can quickly acquire and apply the technology. And because "the language and its annexes are defined using the engineer’s words," as Hugues reports, it is easy to apply and transition engineering concepts in AADL. For that reason, users were able to capture systems concepts with the language and focus on their engineering domain (i.e., safety). This study focuses mostly on safety aspects, but it could also apply to other engineering domains, such as security and performance. AADL is an extensible language, so users can tailor it to capture the quality attributes that they would like to analyze.  We welcome your feedback below. Additional Resources  For more information about AADL, please visit http://www.aadl.info/aadl/currentsite/and our wiki at https://wiki.sei.cmu.edu/aadl/.

By Ian GortonSenior Member of the Technical Staff Software Solutions Division In earlier posts on big data, I have written about how long-held design approaches for software systems simply don’t work as we build larger, scalable big data systems. Examples of design factors that must be addressed for success at scale include the need to handle the ever-present failures that occur at scale, assure the necessary levels of availability and responsiveness, and devise optimizations that drive down costs. Of course, the required application functionality and engineering constraints, such as schedule and budgets, directly impact the manner in which these factors manifest themselves in any specific big data system. In this post, the latest in my ongoing series on big data, I step back from specifics and describe four general principles that hold for any scalable, big data system. These principles can help architects continually validate major design decisions across development iterations, and hence provide a guide through the complex collection of design trade-offs all big data systems require. First Principle: System Costs Must Grow More Slowly Than System Capacity Imagine you’re asked to build a system that will initially manage and analyze 1 petabyte (PB) of data, and the budget to build and operate this system in the first year is $2 million. In addition, data and processing are expected to double in size every year, and performance and availability requirements are expected to remain stable. This growth pattern means that in four years, your system will be managing 16PB of data, and in six years, 64PB. For many development teams, this growth rate would be a daunting requirement. Now imagine that your design can sustain this growth, but the associated costs will also double each year, which means that in four years the project will require a budget for development and operations of $32 million. That’s probably not an estimate many clients will like. What is reasonable will of course depend on how the system functionality will evolve, but linear cost growth estimates are more likely to be acceptable. This principle that costs must grow much slower than capacity is depicted in Figure 1 below. Figure 1. Costs grow slowly as capacity grows quickly Adhering to this principle requires the system architecture to directly address and minimize the costs associated with rapidly growing capacity. For example, choosing a database that can be expanded with minimal manual intervention (for example, one that distributes data using consistent hashing) will make increasing data capacity a fixed, low cost activity. If a request load is cyclic, experiencing peaks and troughs, you can create an elastic solution that handles peaks by provisioning new cloud-based resources as required, and tearing these down to save costs once the peaks are over. If historical data will be accessed infrequently, it may be possible to store it on slower, low-cost storage media and build summarized views of the data for rapid access. Or, if the access patterns are amenable, the most frequently accessed historical data could be cached in online data stores. Scalable software architectures therefore need to constantly seek out and implement efficiencies to ensure costs grow as slowly as feasible, while data and processing capacity grow as rapidly as requirements demand. These efficiencies pervade all layers of big data architectures, and by testing design options against this principle, scalable solutions are much more likely to emerge. Second Principle: The More Complex a Solution, the Less Likely it Will Scale Most of us are taught at an early age that if a deal sounds too good to be true, it probably is. Common sense tells us investments that are guaranteed to grow at 100 percent a year are almost certainly bogus or illegal, so we ignore them. Unfortunately, in building scalable software systems, we commonly see common sense put on the back burner when competing design alternatives and products are evaluated as candidates for major components of big data systems. Let’s take a simple example: Strong consistency in databases is the bedrock of transactional systems and relational databases. Implementing strong consistency, however, is expensive, especially in distributed databases. To build highly scalable and available systems, the NOSQL database movement has consequently weakened the consistency models we can expect from databases. This trend has occurred for a good reason: weak consistency models are inherently more efficient to implement because the underlying mechanisms required are simpler. In response, relational databases and the NewSQL technologies are now turning to new implementation models that provide strong consistency. NewSQL solutions aim to achieve the same scalable performance of NoSQL systems for online transaction processing workloads, while simultaneously supporting the atomicity, consistency, isolation, and durability (ACID) properties found in traditional SQL database systems. This approach sounds attractive, and some of the new open source technologies that exploit main memory and single-threading show immense promise. But fundamentally, achieving strong consistency requires more complexity, and as the scale of the problem grows, it is almost certainly not going to scale as well as weak consistency models. Of course, weak consistency models will give your application greater scalability, but there are trade-offs. You probably have to de-normalize your data model and, hence, manage any duplication this introduces. Application code has to handle the inevitable conflicts that arise with weak consistency models. As always, there’s no free lunch. But, if your data and workload are amenable to a weak consistency model (and many are, even ones we think of as needing strong consistency), it will be your path to scalability. Another example of the second principle is the scalability of many simple concurrent read requests versus concurrent queries that invoke complex, statistical machine learning and data mining approaches to analyze and process tens of MBs of data. We often see requirements that demand the latter, along with 2-second response times. Common sense tells us that is unlikely to happen and certainly won’t scale (this is when you consider materialized views and caches), but often in these circumstances, common sense simply doesn’t seem to prevail. This principle is depicted below in Figure 2. There is one more key point to remember. Even though one design mechanism may be fundamentally more scalable than another, the implementation of the mechanism and how you use it in applications, determines the precise scalability you can expect in your system. Poorly implemented scalable mechanisms will not scale well, and from our experience these are not uncommon. The same applies to inappropriate usage of a scalable mechanism in an application design, such as trying to use a batch solution like Hadoop for real-time querying. Adhering to the second principle requires thinking about the fundamental distributed systems and database approaches that underpin design decisions. Even simple rules of thumb can be enormously beneficial when considering how a design may scale. Ignore these details at your peril, as many have recently found out. Third Principle: Avoid Managing Conversational State Outside the Data Tier State management is a much debated and oft misunderstood issue. Many frameworks, for example the Java Enterprise Edition (JEE), support managing state in the application logic tier by providing explicit abstractions and simple application programming interfaces (APIs) that load the required state from the database into service instance variables, typically for user session state management. Once in memory, all subsequent requests for that session can hit the same service instance, and efficiently access and manipulate the data that’s needed. From a programming perspective, stateful services are convenient and easy. Unfortunately, from a scalability perspective, stateful solutions are a bad idea for many reasons. First, they consume server resources for the duration of a session, which may span many minutes. Session lengths are often unpredictable, so having many (long-lived) instances on some servers and few on others may create a load imbalance that the system must somehow manage. When sessions do not terminate cleanly (e.g., a user does not log out), an instance remains in memory and consumes resources unnecessarily before some inactive timeout occurs and the resources are reclaimed. Finally, if a server becomes inaccessible, you need some logic, somewhere, to handle the exception and recreate the state on another server. As we build systems that must manage many millions of concurrent sessions, stateful services simply do not scale. Stateless services, where any service instance can serve any request in a timely fashion, are the scalable solution. In a stateless architecture, the session state is securely passed as parameters from the client with each request, and the data is accessed from the persistent data tier and/or from caches, where it is ephemeral and only causes a performance hit if not present. In server failure cases, requests can be simply routed to another service instance, and new service nodes can be started at any time to add capacity. Passing  state with each request does consume more network resources, but the amount of resources is typically small as  the state that must be communicated is conversational.  A conversational state is only needed while a session (conversation) is in progress to control a sequence of interactions from a single client, and, hence, is limited in scope and size. Stateless solutions scale trivially by adding resources, simplifying application logic and system operations. The inevitable design trade-off is that stateless solutions place more load on your data tier, making the scalability of an application’s databases a crucial design factor. As Martin Kleppmann writes in his blog post, Six Things I Wish We Had Known About Scaling, "the hard part is scaling the stateful parts of your system: your databases."  For this reason, I am working with other SEI researchers to develop LEAP(4BD), to help organizations fully understand the implications of their data tier choices for big data systems. Fourth Principle: You Can’t Manage What You Don’t Monitor Big data systems rapidly attain deployment scales that change many accepted wisdoms of software engineering. Put simply, two of these challenges to accepted wisdom are The more software and hardware components big data systems have, the higher the likelihood that failures occur. It is impractical to fully test new code because tests will become obsolete as soon as the size of the data on which the code operates grows. As Kleppmann explains, "realistic load testing is hard." The only feasible solution to these two problems is to weave powerful monitoring and analysis capabilities into your applications and deployment infrastructure. By carefully monitoring how systems behave as code and databases and deployments scale, it becomes possible to more easily respond to failures and even proactively take actions as pressure points build up. This last topic is complex, and worthy of treatment by itself, which I’ll do in my next installment in this blog series. In summary though, deep and flexible monitoring and analysis capabilities are fundamental to the success of big data systems, and your architectures must be designed to take into account the costs of these capabilities, which can be considerable. As we’ll see, this requirement for production-time data capture and analysis has many implications for successful big data system deployments. Final Thoughts and Looking Ahead The four principles described above hold for any big data system, so adhering to them will always be a good thing. In contrast, unconsciously violating these principles is likely to lead a system into downstream distress, slowing capability delivery, massively inflating costs and potentially leading to project failure. Of course, simple expediency may mean you have to violate a principle to meet a near-term deliverable. Violations are not a bad thing, as long you recognize the technical debt that has been incurred and plan accordingly to pay this debt back before the interest incurred becomes a major problem. The next post in this ongoing series on big data will examine the challenges of scale and how observability is fundamental to the success of big data systems. Additional Resources To listen to the podcast, An Approach to Managing the Software Engineering Challenges of Big Data, please visithttp://url.sei.cmu.edu/iq. For more information about the  the Lightweight Evaluation and Architecture Prototyping (for Big Data), known as LEAP(4BD), please visit http://blog.sei.cmu.edu/post.cfm/challenges-big-data-294.   

By Grace Lewis Principal Investigator, Edge-Enabled Tactical SystemsSoftware Solutions Division Many warfighters and first responders operate at what we call "the tactical edge,"  where users are constrained by limited communication connectivity, storage availability, processing power, and battery life.  In these environments, onboard sensors are used to capture data on behalf of mobile applications to perform tasks such as face recognition, speech recognition, natural language translation, and situational awareness. These applications then rely on network interfaces to send the data to nearby servers or the cloud if local processing resources are inadequate. While software developers have traditionally used native mobile technologies to develop these applications, the approach has some drawbacks, such as limited portability. In contrast, HTML5 has been touted for its portability across mobile device platforms, as well an ability to access functionality without having to download and install applications. This blog post describes research aimed at evaluating the feasibility of using HTML5 to develop applications that can meet tactical edge requirements.  Foundations of Our Work  HTML5, developed by the World Wide Web Consortium (W3C), is the fifth release of the core technology markup language used for structuring and presenting content for the World Wide Web. HTML5 mobile applications are similar to web applications in that they use the same technologies for software application development: HTML, JavaScript, and CSS3. In addition to portability, HTML5 allows users to access functionality without having to download and install applications. In fact, only an HTML5-enabled web browser is needed to run HTML5. This flexibility also makes it easier to distribute applications to edge users, especially if they do not have frequent access to their base of operations or headquarters to get apps and updates. Our Approach  To help military personnel in edge environments, mobile applications must  exploit available sensors so that contextual information can be captured with easy and minimal user interaction process, store, and forward sensed and captured information  be resilient to intermittent communications connectivity and opportunistic in using communication capabilities as they become available  manage resources on the mobile devices so that they are used as efficiently as possible to maximize the availability of the system by reducing power consumption   At the Carnegie Mellon University Software Engineering Institute, the Edge-Enabled Tactical Systems Research Team focuses on helping warfighters and field personnel who use mobile devices in these environments. I conducted the research reported in this blog posting together with Bryan Yan, a recent graduate of CMU’s Master of Information Technology - Embedded Software Engineering (MSIT-ESE) program.  As outlined in our technical note on this research, Evaluation of the Applicability of HTML5 for Mobile Applications in Resource-Constrained Edge Environments,after defining the requirements of edge applications, we created a set of development features that mapped to these requirements.  We then wrote simple HTML applications that implemented these features.  Finally, we tested the code on the Google Chrome, Firefox for Android, and Dolphin web browsers on two different mobile devices with different versions of Android.  As an example, the code snippet shown below is an HTML5 application that supports geolocation. The code snippet registers the location tracking feature of the mobile device and reports the longitude, altitude, heading, speed and accuracy of the location in plain text:   Our results evaluating other HTML5 features as of December 2013 are as follows:    The Use of Bridging Frameworks  Given that not all features developed using HTML5 are supported, we next explored the use of bridging frameworks, also called cross-platform mobile development frameworks,  to see if they could be used to implement the edge features not supported by HTML5. Bridging frameworks are software packages created by developers that create hybrid software applications merging native mobile application developments with the advantages of HTML5.  As described in our report, we investigated the use of PhoneGap to create a notification application and a battery status application for the Android platform, which are features that are not supported by the HTML5 standard alone. The hybrid mobile applications generated by PhoneGap appear as native applications to the mobile platforms because PhoneGap creates a single Android Activity application (i.e., a wrapper application) that uses the Activity’s onCreate () method to load an HTML5 web page containing all the application logic and presentation. The code snippet below shows the wrapper application source code for the notification demo application for the Android platform:  While hybrid applications allow developers to access the mobile platform’s native application programming interfaces (APIs) via the PhoneGap framework, these applications appear native to the mobile platforms and thus must be installed onto the device. This compromises one of the main benefits of software applications developed using only HTML5, which is the ability to run an application with only a browser.  Software Architecture Implications  We also looked at HTML5 with respect to maintainability, performance, and portability. For maintainability we specifically looked at the effect of evolution in HTML5, browser support for HTML5, and bridging frameworks on existing HTML5 applications. We concluded that based on the information we found in change logs related to HTML5 there would be no effect on existing applications. To migrate hybrid mobile applications to HTML5 mobile applications as new features become supported, however, application logic must be modified, though the user interface (presentation layer) of the mobile application can be preserved. Having a coding standard that localizes JavaScript code in specific areas of the HTML5 file or places it in external files can facilitate the process. For performance we looked at specifically at execution time and memory usage and discovered that HTML5 mobile applications are over 3 times slower and use 3 times more memory than native Java Android applications, which is expected because the applications run inside a browser. The larger memory consumption and slower performance is a potential drawback of using HTML5 for edge applications, given limitations in computing and battery power.  Finally, for portability we looked at portability of HTML5 applications across mobile web browsers, in particular Firefox, Chrome and Dolphin, and found that portability of HTML5 mobile applications across mobile web browsers varies. We expect this to change as browsers start supporting more parts of the HTML5 specification, but this has been the major drawback of HTML5 and one of the reasons why developers turn to bridging frameworks.  Wrapping Up and Looking Ahead Warfighters and first responders at the tactical edge face limited computing resources, intermittent or no connectivity to the enterprise, and the potential of large amounts of field-collected data. These environments are highly fluid and unpredictable due to factors like mission changes, threats, or changing weather conditions. Our analysis found that as of June 2014, the HTML5 standard still lacks support for many critical features needed to support development and deployment of applications for these types of edge environments.  Even though we expect that HTML5 will evolve rapidly to support the development of edge applications given the tremendous support it receives from the developer community (e.g., the Firefox operating system is built entirely using HTML5), the use of HTML5 for edge applications for its portability and maintainability advantages will have to be evaluated against constraints in battery, computing power and screen size of mobile devices Researchers on the Edge-Enabled Tactical Research Team at the SEI will continue to examine HTML5 for use in mobile applications at the tactical edge.  We welcome your feedback on our research. Please leave feedback in the comments section below.  Additional Resources  To read the SEI technical note, Evaluation of the Applicability of HTML5 for Mobile Applications in Resource-Constrained Edge Environments, please visithttp://resources.sei.cmu.edu/library/asset-view.cfm?assetid=295973. 

On Wednesday (02/04/14) Adrian and I started the next part of our journey and joined the project at Carers Allowance in Preston. We arrived in time to join in the daily morning stand up. This is a short meeting for each person to let the team know where they are up to on their work and any issues they’ve got. We were impressed with how quickly the Scrum Master got through all 15 people (11 minutes!). We introduced ourselves and were given a very warm welcome. We were invited to attend a User Feedback presentation by Ben Holliday. This was a very interesting session to see how the work the project is viewed by the end users (customers). All their feedback is considered and any amendments built in to future Sprints. Following this we were invited to join in the planning session for the next Sprint. This gave us a really good insight into how Sprints are built up and why releases are done in small parts. After lunch we set about making a plan for the objectives we have been set Our objectives to learn and use during our "settling in" Check back for updates on how we are getting on……………..  

In the job market, we’ve all come to accept a little bit of exaggeration. We don’t like to think of ourselves as liars, but whether it’s to make ourselves or someone else feel better, save someone from trouble, or to make a story all the richer, human beings come with the baggage of having to lie for a number of reasons. Lying is bad, lying can be good; lying can lead people to disastrous situations, lying can save a life. It’s the definition of a necessary evil. Fibbing on your resume or interview feedback...

By Sarah SheardMember of the Technical Staff Software Solutions Division The role of software within systems has fundamentally changed over the past 50 years. Software’s role has changed both on mission-critical DoD systems, such as fighter aircraft and surveillance equipment, and on commercial products, such as telephones and cars. Software has become not only the brain of most systems, but the backbone of their functionality. Acquisition processes must acknowledge this new reality and adapt. This blog posting, the second in a series about the relationship of software engineering (SwE) and systems engineering (SysE), shows how software technologies have come to dominate what formerly were hardware-based systems. This posting describes a case study: the story of software on satellites, whose lessons can be applied to many other kinds of software-reliant systems.   The Evolution of Satellites to Today and Beyond My first professional job started in 1980 when I was hired as a satellite systems engineer at Hughes Aircraft Company in the space and communications division. At this time, there were a few career software developers, but individual engineers created most technical software for their own one-time use. The software that engineers developed ran on large and heavy computers that stayed earthbound, so it was not considered part of the satellite system.  In contrast, software today provides a large fraction of the operational capability of satellite systems. Satellite software initially appeared inside some of the hardware boxes, after the needed system functionality was allocated as requirements to the hardware elements. In contrast, in today’s software-reliant systems, the logic implemented by the software is the single dominant aspect of capabilities provided by the systems and must be considered early, perhaps before any other capabilities.  This posting presents six figures that demonstrate the evolution from satellites that had little to no software in the 1970s, to satellites of today and beyond, which use software-driven services to link existing networks.    In all figures, the blue boxes that represent the satellite system are the same size for simplicity, even though cost, size, or weight have all grown significantly during this time.  Red represents the software within the satellite.  The amount of red is meant to suggest only roughly whether there is a little or a lot of software: the growth in actual lines of code has been exponential, far greater than the growth shown in the area of red. Background The satellites discussed in this post are geostationary communication satellites. These satellites are situated above the equator at the altitude whose orbit period is 24 hours, so the satellite appears always at the same point in the sky with respect to the rotating earth. Geostationary communication satellites have been known as "bent pipes" in the industry because the satellite receives communication signals from the ground and bends the same signals (amplified) back toward the ground.   Since satellites are space systems, they have different constraints from business systems or ground-based weapon systems. Launch weight is a huge determinant of cost (the rule of thumb in the 1980s was $20,000/pound). Radiation hardness is required to allow the satellite to pass through the Van Allen radiation belts unscathed and to survive solar winds and other orbital conditions. Temperatures are extreme outside the atmosphere, so satellite components must be qualified to large temperature ranges and/or protected with thermal control equipment, such as Kevlar thermal blankets.  Rocket technology made vast leaps in the 1960s race to the moon, allowing for more economical satellite launches that became commercially affordable. Early satellite technology challenges were hardware-focused, such as how to handle increasing numbers of channels with more output power and with better sensitivity to signals received from the ground, and how to maintain operations for more years (such as by increasing the fuel for orbit maneuvers). The weight constraint meant that technology improvements that shrank hardware, made it from lighter materials, or allowed more functions per box were highly desirable. It is clear now that software was the obvious answer, but it was not clear at the time, mostly because the computers on which the software ran were so heavy. Software on satellites through the decades: 1960s and 1970s In the 1960s satellites had little to no software since computers were too big and heavy to launch into orbit. As computer hardware shrank, it could be included on satellites, and satellite computer software developed. The first satellite computer software was embedded into the computer hardware. My colleague Patrick Place worked on the first reloadable satellite software, which was necessary because the launch window was prior to finalization of the software. In the 1970s, computers were shrinking in size, and became feasible for use on satellites. The challenge was to obtain space-qualified chips of high capacity and low weight. I had a t-shirt from 1980 that celebrated the space-qualification of a RAM chip with ¼ Kbyte capacity (256 bytes)!  Imagine the software constraints posed by that limited capacity, which was state-of-the-art at the time!  By the end of the 1970s, satellites included some processors that performed small amounts of computation. For example, software was part of the subsystem that controlled satellite orientation (attitude control subsystem).  The 1970s figure shows one box on a satellite included a little software.   1980s During the 1980s computers were getting smaller, lighter, and more powerful. Engineers of several subsystems found weight savings by using custom, space-qualified computer chips loaded with software. The 1980s figure shows there were a few boxes on the satellite that contain some software, including the attitude control processor, the telemetry and command boxes, and perhaps some payload boxes.  In the mid-1980s, I was involved in studying the tradeoff criteria when power subsystem engineers suggested they could save a great deal of weight by having the attitude control processor take over the logic for switching of solar panels between recharging batteries and supporting the power bus.  Software would eliminate 10 power switches and their wire harnesses. The attitude control subsystem engineers were reluctant to take on logic that had nothing whatsoever to do with their subsystem, didn’t save them any weight, and complicated the software development and test process. (The outcome was that the switching was done in software). As a systems engineer in the 1980s, I was only indoctrinated into the discipline by learning on-the-job. At the time, few other resources existed. Few textbooks existed; the Defense Systems Management College (DSMC) guidebook, project management books (also fairly new) and Hatley-Pirbhai structured requirements analysis were some of the few subjects with available instructional material.    1990s By the 1990s, many of the subsystems and units on the satellite included computer hardware and software. More important, the software in many hardware units was beginning to interact with software in other units on the spacecraft, and point-to-point communications was established among them.   The 1990s figure shows the boxes starting to communicate with each other. 2000s By 2000, the satellite industry had started to view software as an important aspect of the entire spacecraft. Software within the various subsystems and boxes was being coordinated, protocols were established, and data buses were set up that ensured compatible messaging and rapid communication. In addition, the interfaces to the outside world were being shown on software architecture diagrams. The 2000s figure shows a typical architecture of the software modules in the satellite system. 2010s and Interoperability Between 2000 and the present, the concept of interoperability of systems has matured. Initially interoperability meant being able to connect to other technological devices. Today, it means the ability to communicate and exchange data, using a common semantic information exchange model so that the numbers do not change meaning at different points in the transmission. Some consider use of open standards to be critical for true interoperability (unencumbered by patents, proprietary claims, or dominance by one market competitor). The need for this interoperability has led to common architectural practices that are known well by software engineers, but not necessarily by systems engineers.  By the 2010s, diagrams of satellites commonly looked like software architecture diagrams, with layers, protocols, and strong interaction with software outside the satellite, both to ground stations and to users beyond the satellite terminals.  The 2010s figure shows the strong layering and the relationship of the layers to external software modules. The modules interact to perform much more interesting functions than individual modules would; this means their architecture is more important and depends greatly on their implemented quality attribute requirements. At this point the satellite system box just looks like a boundary drawn around software.  Of course, other views of the satellite exist that depict the electrical power distribution, physical box placement, gain and link budgets, deployment sweeps, propulsion routing, and the like. But importantly, it’s becoming more widely accepted that the software or logical view of the satellite should be well-engineered and developed as early as the first hardware specification. Near Future The last diagram shows my projection into the future. Satellite hardware is now seen more as a commodity, selected so that the satellite can provide services and interconnectivity of the right kind among existing and new networks. The environment contains large, mature, evolving networks of many types that may be considered as data sources or users of new, possibly fused or mined data. The United States Armed Forces have a number of different wartime and business networks, some built on old technology that should be modernized, and which have various levels of classification. Commercial industry has its own networks. For example, health care is connecting medical devices in doctors’ offices, hospitals, pharmacies, and even individuals without necessarily having worked out security implications.  Satellites can be seen as deliverers of information among such networks. DARPA’s "System F6" or "fractionated spacecraft" program investigates effectively this: whether the tradeoffs to achieve the functionality can be best served if the software is split among multiple hardware nodes (small satellites), or in what cases a single hardware spacecraft is best. I expect that systems of the 2020s will start with a recognition of what networks exist in the environment relevant to a new system and to its needed capabilities. The systems that will be built will have capabilities that essentially "hang" on the relevant networks like modular kitchen accessories hang on racks at the hardware store. System capabilities will be determined by what functions and responses can be created first by connecting data and processes available on one network with data and processes on others; then the system’s capabilities and hardware will be specified and designed. In the software engineering field, it is time to recognize this evolution in the relationships of systems to software. Satellites are a microcosm of broader trends in both military and commercial system development, whether weapons systems, software embedded in hardware, or business systems. In the past, capability needs led to a plan to acquire systems, and the systems required pieces of software within them. In the future, capabilities will lead to a plan to acquire needed software, to connect existing networks with new kinds of data and processing, and this will drive the need for specific systems.  Satellites, of course, are only an example. Wrapping Up and Looking Ahead  The capabilities of the future demand attention to software, and software architecture, first. Systems engineers who participate in early acquisition must bring software architecture knowledge into early specification of system activities and move toward the day when it is the software architecture of the real world, and secondarily that to be specified within a system, that determines what systems are brought into being. In addition we need to involve systems engineers who bring particular systems into being to help them understand the repercussions of various systems engineering and acquisition decisions as far as software capability is concerned. Naturally, not only software should be involved early. Systems engineers, with their broad views and customer-needs focus, must drive the early specification of capabilities, but they will not be able to do that without bringing the software engineers in as partners from day one. In the next post in this series, I will discuss complexity as it relates to systems and software engineering. Additional Resources To read the blog post, The Value of Systems Engineering, by Joe Elm, please visit http://blog.sei.cmu.edu/post.cfm/the-value-of-systems-engineering.    A seminal systems engineering textbook was published in 1962 by Arthur Hall, A Methodology for Systems Engineering. The methods described in the book seem remarkably current, including eliciting requirements and looking for the business need behind the customer’s requests for a new system.    By 2000, many systems engineering texts had become available including those by Howard Eisner of George Washington University (Essentials of Project and Systems Engineering Management), Eberhardt Rechtin of the University of Southern California (Systems Architecting), and Richard Stevens et al. in the United Kingdom (Systems Engineering: Coping with Complexity). INCOSE had also released the first two versions of its systems engineering handbook by July 2000. None of these documents included much understanding of the important role software had come to play by that point, however.   In 1994 NCOSE participated in two efforts to define the body of practice of systems engineering: the Systems Engineering Capability Maturity Model (SE-CMM) and the Systems Engineering Capability Assessment Model (SECAM), NCOSE’s counterpart. By the late 1990s, NCOSE had become INCOSE, the International Council on Systems Engineering. The organization’s membership was growing and had played a role in merging the two systems engineering models and in writing five systems engineering standards.  

As part of our continued development, we have been discussing the idea of holding a "hack day" This is hacking in a good way, nothing to do with hacking into the Pentagon! We will have a group of a approximately 10, working in pairs on a set problem. Through pair programming, a solution will be written in code and tested in a "show and tell" to the rest of the group. We have now arranged to hold the first DWP Hack Day in Preston on 16th April. Check back for updates on how we get on!

By James Cebula Senior Member of the Technical Staff SEI CERT Division This blog post was also co-authored by Lisa Young. Organizations are continually fending off cyberattacks in one form or another. The  2014 Verizon Data Breach Investigations Report, which included contributions from SEI researchers, tagged 2013 as "the year of the retailer breach." According to the report, 2013 also witnessed "a transition from geopolitical attacks to large-scale attacks on payment card systems." To illustrate the trend, the report outlines a 12-month chronology of attacks, including a January "watering hole" attack on the Council on Foreign Relations website followed in February by targeted cyber-espionage attacks against The New York Times and The Wall Street Journal. The well-documented Target breach brought 2013 to a close with the theft of more than 40 million debit and credit card numbers. This blog post highlights a recent research effort to create a taxonomy that provides organizations a common language and set of terminology they can use to discuss, document, and mitigate operational cybersecurity risks. Foundations of Our Work Organizations of all sizes in the public and private sectors increasingly rely on information and technology assets that are supported by people and facilities. An attack that disrupts these assets can be devastating. In March, The Economist reported that the Target breach "cost the company US $61m in response costs in the fourth quarter alone and helped fuel a 5.5 percent drop in transactions during the crucial holiday shopping season." For the purpose of drafting our taxonomy, we defined operational risks as  those arising due to the actions of people, systems and technology failures, failed internal processes, and external events We defined operational cybersecurity risk as follows:  operational risks to information and technology assets that have consequences affecting the confidentiality, availability, or integrity of information or information systems While there are a tremendous number of risks that organizations have no control over, organizations can control their understanding of risk in the context of business objectives. Specifically, organizations need to ask the following questions:  What are our most important mission aspects?  What threats and risks are we most concerned about?  It is not enough for organizations to understand what controls they are mandated to implement. For instance, federal agencies need to conduct a thoughtful consideration of risk specific to their business so that they can make financially responsible decisions when allocating resources. Our initial taxonomy, along with this latest effort, attempts to help organizations and federal agencies secure information systems and manage risk effectively.  As we outlined in the recently published SEI technical note, A Taxonomy of Operational Cybersecurity Risks, the taxonomy can be used as a tool to help identify all applicable cybersecurity risks within an organization. The impetus for revisiting and updating this taxonomy is the release in April 2013 of the National Institute of Standards (NIST) Special Publication 800-53 rev. 4, Security and Privacy Controls for Federal Information Systems and Organizations, which offers updated security controls for agencies. These controls are intended to protect federal agencies and should be applied tactically at the information-system level. To address expanding cyber threats, the NIST report addresses mobile and cloud computing; applications security; trustworthiness, assurance, and resiliency of information systems; insider threat; supply chain security; and the advanced persistent threat.  Our taxonomy is structured around a hierarchy that comprises the four main classes from the definition of operational risk actions of people (or inaction) taken either deliberately or accidentally that impact cybersecurity  systems and technology failures including failure of hardware, software, and information systems failed internal processes including problems in the internal business processes, that adversely affect the ability to implement, manage, and sustain cybersecurity, such as process design, execution, and control external events including issues often outside the control of the organization, such as disasters, legal issues, business issues, and service provider dependencies The taxonomy further decomposes each of the above listed classes into subclasses, and each subclass is described by its elements.  In attempting to address risk, organizations need to understand that risks can cascade: risks in one class can trigger risks in another. For example, a software failure due to improper security setting could be the result of any of the elements of inadvertent or deliberate actions of people. Organizations therefore need to analyze a particular risk by involving several elements from different classes.  Consider the following example regarding external events pulled from our technical note:  Class 4 External Events  External events describes a class of operational risk associated with events generally outside the organization’s control. Often the timing or occurrence of such events cannot be planned or predicted. The supporting subclasses of this class include disasters, legal issues, business issues, and service dependencies.  Subclass 4.1 Hazards  The hazards subclass deals with risks owing to events, both natural and of human origin, over which the organization has no control and that can occur without notice. The elements supporting this subclass include weather event, fire, flood, earthquake, unrest, and pandemic.  4.1.1 weather event—adverse weather situations such as rain, snow, tornado, or hurricane  4.1.2 fire—fire within a facility or disruption caused by a fire external to a facility  4.1.3 flood—flooding within a facility or disruption caused by a flood external to a facility  4.1.4 earthquake—disruption of organizational operations due to an earthquake  4.1.5 unrest—disruption of operations due to civil disorder, riot, or terrorist acts  4.1.6 pandemic—widespread medical conditions that disrupt organizational operations  Subclass 4.2 Legal Issues  The legal issues subclass deals with risks potentially impacting the organization due to the elements regulatory compliance, legislation, and litigation.  4.2.1 regulatory compliance—new governmental regulation or failure to comply with existing regulation  4.2.2 legislation—new legislation that impacts the organization  4.2.3 litigation—legal action taken against the organization by any stakeholder, including employees and customers  Subclass 4.3 Business Issues  The business issues subclass, described by the elements of supplier failure, market conditions, and economic conditions, deals with operational risks arising from changes in the business environment of the organization.  4.3.1 supplier failure—the temporary or permanent inability of a supplier to deliver needed products or services to the organization  4.3.2 market conditions—the diminished ability of the organization to sell its products and services in the market  4.3.3 economic conditions—the inability of the organization to obtain needed funding for its operations  Subclass 4.4 Service Dependencies  The service dependencies subclass deals with risks arising from the organization’s dependence on external parties to continue operations. The subclass is associated with the elements of utilities, emergency services, fuel, and transportation.  4.4.1 utilities—failure of the organization’s electric power supply, water supply, or telecommunications services  4.4.2 emergency services—dependencies on public response services such as fire, police, and emergency medical services  4.4.3 fuel—failure of external fuel supplies, for example to power a backup generator  Mapping to Existing Federal Initiatives  As with our initial taxonomy, which we also detailed in a technical note, this latest version also spans a broad swath of industries and explores how to reconcile risk mitigation efforts with recent federal government initiatives:  the Federal Information Security Management Act of 2002 (FISMA 2002), which applies to U.S. federal agencies and provides a standardized definition of information security that links identified operational cybersecurity risks to specific examples of consequences that impact confidentiality, integrity, and availability security guidance provided in NIST special publications, which provide a control catalog that can be applied to federal information systems based on an analysis of the system’s relative importance and consequence of loss the threat profile contained within the CERT Operationally Critical Threat, Asset, and Vulnerability Evaluation (OCTAVE) method. The OCTAVE method uses the concept of asset-based threat profiles. Octave uses four standard threat categories:  -human actors using network access -human actors using physical access -system problems  -other problems including failed internal processes and external events For example, NIST SP 800-53 rev 4 includes, for the first time, a number of controls associated with mitigating insider threats. These and other new controls are now mapped into the taxonomy. In Conclusion and Looking Ahead  Although estimates vary, a recent survey by Ponemon Institute estimated the cost of cybercrime in 2012 to be $8.9 million per company. As shown by the Verizon DBIR report that was mentioned in the introduction, the nature of cybersecurity will continue to evolve on new fronts, prompting new revisions to our taxonomy. In the meantime, we are field testing our taxonomy with various organizations subject to regulatory compliance and risk tolerance. The results of these field tests will also inform future revisions to this taxonomy.  We welcome your feedback on our research in the comments section below.  Additional Resources To read the SEI technical report, A Taxonomy of Operational Cyber Security Risks, Version 2, please visit http://resources.sei.cmu.edu/library/asset-view.cfm?assetid=91013.   

On 16th April we held the first DWP Hack Day in Preston. The day was attended by 10 enthusiastic people within DWP IT, including the DWP Developer team. Attendee’s were; Paul Anderson Paul Bowman Wayne Hopkins Nathalie Harrold Donna Kirk Mike Morris Suzanne Russell Daniel Sherman Adrian Stone Ian Tidmarsh The day was delivered by a Raspberry Pi trainer, Les Pounder, who specialises in delivering Hack Days to inexperienced people. The day started in a unusual style. We had to programme a robot, without using a computer. Or a robot…….. We split in to 3 teams, each nominating a team member to adopt the role of "robot" and set about inventing our own language to direct our human around a maze. This was a very useful exercise in teaching the structure of programming and also understanding that you can achieve a lot by using clear and accurate commands. Following a short break, we fired up the Raspberry Pi’s and began our computer learning of the day. We were using a programme called "Scratch". Scratch in action This is an excellent tool to teach the structure and concepts of code without the horror of seeing pages of symbols and letters. This allows you to place blocks and snap together groups of commands. We worked in pairs, using a technique called "Pair Programming" where one person "Navigates" and the other person "Drives". This means, one person types the code, drives, and the other person watches and directs where needed, navigates. "Hackers" pair programming We broke for lunch and returned for the main event. Building and coding a real robot. It was more Wall-E than Robocop, but exciting nonetheless! After working out which pins power which motor (and getting them to spin the wheels the same way!) we set about building a series of blocks in Scratch to make the robot move around a maze with keyboard input. This worked very well and allowed us to see how our code was working with Left, Right and Forward keyboard inputs. Following this, we then transferred our blocks of Scratch into real blocks of Python code. This allowed us to automate the robot to move around the maze with no user input. We achieved this by fine tuning the length of time each motor spins. Building a robot Build complete We also decided to blu-tac a whiteboard marker to the robot and make it into an elaborate Spiro-graph! This was an excellent way to visually demonstrate how loops work. Raspberry Pi Drawing As the day drawn to a close we learnt by splitting code up into pieces, it is not as daunting as it first appears and by applying logic, you will produce very impressive results.

By Ian Gorton Senior Member of the Technical Staff Software Solutions Division The term big data is a subject of much hype in both government and business today. Big data is variously the cause of all existing system problems and, simultaneously, the savior that will lead us to the innovative solutions and business insights of tomorrow. All this hype fuels predictions such as the one from IDC that the market for big data will reach $16.1 billion in 2014, growing six times faster than the overall information technology  market, despite the fact that the "benefits of big data are not always clear today," according to IDC. From a software-engineering perspective, however, the challenges of big data are very clear, since they are driven by ever-increasing system scale and complexity. This blog post, a continuation of my last post on the four principles of building big data systems, describes how we must address one of these challenges, namely, you can’t manage what you don’t monitor.  The Challenges of Scale While debate rages over the precise characteristics of a big data system, from a software-engineering perspective, the major characteristics are well understood. These characteristics revolve around the scale of applications and the inherent complexity that emerges as applications grow to manage and analyze ever-growing amounts of data. As I’ve written in earlier blog posts, the requirements of scalability mandate that we adopt new design and engineering approaches because many existing tenets of software engineering simply don’t hold at scale. A fundamental characteristic of big data systems is that there will be an increasing number of software and hardware failures as the scale of an application increases. Component failure, in fact, must be seen as the norm, and applications must be designed to be resilient to failures and diligently handle them to ensure continued operations. Note that software failures induced by scale may occur in application components or in third-party components integrated into the system, both open-source and commercial.  A second characteristic is that, as scale increases, so does complexity. There are more component interactions, increasingly unpredictable request loads on data collections, and increased competition for shared resources, including central processing units (CPUs) on multicore nodes, memory, cluster interconnects, and disks. This inherent complexity and non-deterministic behavior makes diagnosing aberrant behavior an immense challenge. If performance suddenly becomes a problem for particular transactions, it can be immensely time-consuming and challenging to diagnose, whether the cause lies in the transaction implementation itself or is a result of unexpected interactions with other components.  A third characteristic of big data systems is that scale makes thorough testing of big data applications before deployment both impractical and infeasible. For example, if your application will store and analyze one petabyte of data, it is probably impractical to create a test environment and test data at this scale. Moreover, even if you could test at deployment scale, as soon as your data grows, your code is operating beyond its tested tolerances. In reality, the only way to discover if your new components operate correctly is to deploy them on the production system and use techniques such as canary testing to validate their behavior.  Finally, it’s important to see these challenges of failure handling, complexity, and testing in the context of contemporary big data system deployments. The scale of companies like Netflix, which uses Cassandra to manage data on 750 nodes as part of its cloud-based software infrastructure, are well documented examples that herald the future for many large government and business organizations. Even today, there are many conventional government and business systems that require massive resources to satisfy their requirements. Here are some examples from the healthcare domain. The United Kingdom’s National Health Service (NHS) Spine2 system is being initially deployed on hundreds of processors running on dozens of Linux-based commodity servers located across multiple data centers. Spine2’s database contains clinical records for each of the 70 million NHS patients, and this data is requested and amended more than 2.6 billion times a year. Spine2 also connects and services more than 21,000 organizations and links over 27,000 software systems within these organizations. Ease of scalability to support information growth and high availability to support clinical care are key drivers behind the design of Spine2. Similar systems have been built to support the Danish medical records system, and, in the United States, the Military Health System stores several petabytes of patient and clinical data and integrates with a large number of external health providers. Observability: Fundamental to Successful Big Data Systems The term observability defines a collection of capabilities that make it possible to monitor, analyze, and proactively and reactively respond to events that occur at runtime in a big data system. As big data systems scale, it must be possible to observe and reason about changes in behavior so the system can be operated and evolved reliably. The adage "you can’t manage what you don’t monitor" is especially true for complex, distributed systems that have an overwhelming number of moving parts, both hardware and software, that interact with each other in many subtle and unanticipated ways. Here’s a simplified example of the problems that can arise in big data systems. Two separately developed, completely independent business transactions were providing the expected response times in operations across a horizontally partitioned database. Suddenly, one transaction slowed down, intermittently, making it occasionally non-responsive to user needs. Extensive investigations over several days, including detailed logging in production and attempts to recreate the situation in test, eventually led to identifying the root cause. Essentially, periodic and brief request spikes for one of the transactions were overloading a small number of database nodes. During these overload conditions, when the other transaction was invoked, it attempted to read data using a secondary index that is distributed across all nodes. These secondary index reads from the overloaded nodes were taking tens of seconds to respond, leading to unacceptable latencies for those transactions.  How could an observability framework have helped discover the root cause of this problem more quickly? If the developers could have analyzed performance data to visualize transaction volumes for the first transaction against latencies for the second, it would have been immediately obvious that there was a correlation. This visualization would have highlighted the areas of the code that should be investigated as the root cause of the high transaction latencies stemmed from this subtle interaction . Of course, this type of performance data does not come for free from databases or web and applications servers. Capturing the necessary performance data to perform this type of analysis requires applications to be instrumented with application-relevant measures for observability  a data collection capability to capture and store observability data analysis capabilities for developers and operators to gain rapid insights from observability data  By necessity, internet companies operating at immense scales have built their own observability solutions. These solutions are extensive and powerful and have been built at considerable cost specifically for each operational environment. In recent discussions, we discovered teams building and evolving observability solutions comprising 10 to 20 people in large internet organizations. The resources used by these solutions, both at runtime and in terms of storage, ranged from 3 percent to 18 percent of the total system resources and in one case monitored approximately 500 million separate metrics per minute. When systems execute on thousands of nodes with petabytes of data, a lot of resources must be dedicated to observability, which is a testament to the necessity of this capability. An example description of Twitter’s solution in this area can be found here. Likewise a description from Netflix on its efforts to design and build its next generation of innovative insight tools for operational visibility is available here.  Observability Stack The figure below represents an abstraction of the common components of an observability solution, organized as a layered architecture. We describe key capabilities required and inherent design trade-offs for each layer in the following:  Data collection: There are two elements to data collection: specifying what to collect and periodically gathering raw data from every node in the system. A component should be dynamically configurable  to specify the application-relevant data to capture—this can be done using a framework such as Archaius. Measured data should then be transported to a distributed data collection capability that either periodically pulls data from every node or accepts data that is pushed from multiple nodes. Key design decisions in this layer revolve around the number of measures collected, at what frequency, and whether nodes themselves perform some local pre-preprocessing of data, with a potential loss of fidelity, or whether raw data is transported for further processing. These alternatives represent design trade-offs that can greatly affect the amount of resources used by the observability framework and its ability to scale. Data aggregation: Aggregation is essential to constructing application-relevant metrics from raw data points. As a simple example, in a horizontally distributed database, the number of reads and writes served per time interval is the sum of the requests served by every database partition. Another example is for a business transaction level; the overall latency will be determined by the response times of all the lower-level services called. Should the business transaction itself measure these? Or, should the services themselves report their latencies so that they can be aggregated across all requests from different services? In terms of design trade-offs, the amount of aggregation performed will determine the type of queries that can be efficiently asked of the data and the amount of storage it consumes.  Data storage: Persisting observability data is necessary for downstream analysis purposes. Major design decisions revolve around whether to store raw as well as aggregated data, and how long the data should be kept. Observability data can quickly become a big data problem in itself, and hence the resources it consumes can become costly. This realization leads to design approaches that only keep fully aggregated and/or raw data for a specific time window  (e.g., one month, six months), and dynamically create lower fidelity snapshots of the data for specific time periods (e.g., last day, last week) that are updated daily.  Data analysis: Both developers and operators need tools to query observability data so they can anticipate and diagnose problems. Querying observability data requires user interfaces for constructing queries and visualizing performance measures in a user-driven fashion. Users should also be able to define triggers that fire alarms when certain metrics stray out of expected ranges (e.g., 95th -percentile response times for a core service exceeds 500 milliseconds). Important design decisions revolve around the usability of the user interface in terms of the ease of constructing queries and visualizing results.  Existing Solutions Distributed system monitoring tools have existed for many years. Commercial tools, such as AppDynamics,  are comprehensive products. However, like their commercial counterparts in the database arena, license costs can become an issue as systems scale across hundreds and thousands of servers. Open source equivalents such as Nagios and Ganglia are also widely used, and a useful comparison of technologies in this space can be found here. Adopting these technologies and tailoring them to highly heterogeneous execution environments to observe application-relevant measures, as well as making them operate at the scale required by the next generation of big data applications, will represent a major challenge for any development organization. At the SEI, we are researching the creation of highly configurable, scalable, and extensible observability solutions. In this context, being configurable requires tools that would allow a development team to rapidly configure the data collection requirements and trade-off analytic fidelity against resource usage, both for data collection and storage. Being scalable requires a lightweight core framework that is engineered from the ground up to ensure extreme scalability. Finally, being extensible demands rapid modifications are possible to interact with new application endpoints (e.g., databases and middleware) that are introduced during system evolution. Our aim is an observability framework that can be adopted in a cost-effective manner and rapidly customized by a wide range of organizations to help attain high levels of assurance and reliability.  If you're interested in this work, we’d like to hear from you. Please leave a comment below. Additional Resources To view Ian Gorton’s webinar, Software Architecture for Big Data Systems, please visithttp://resources.sei.cmu.edu/library/asset-view.cfm?assetid=298346.  To listen to the podcast, An Approach to Managing the Software Engineering Challenges of Big Data with Ian Gorton and John Klein, please visithttp://www.sei.cmu.edu/podcasts/podcast_episode.cfm?episodeid=294249&wtPodcast=AnApproachtoManagingtheSoftwareEngineeringChallengesofBigData. 

There is no shortage of hype around digital transformation but never has the mantra: ‘innovate or die’ been more true. There are some who see technology and the internet as disruptive, forcing organisations to break with the past and re-engineer how they work. Others see it as a game-changing force for good - breaking down the barriers of entry for new businesses and creating a level playing field for both entrepreneurs and innovators. Both are true but it won’t surprise you that I’m definitely drinking from a glass that’s more than half-full rather than half-empty. That said, I do not underestimate the unique challenges and opportunities that digital transformation brings for Government. At DWP, it provides the opportunity to revolutionise how we interact with millions of people everyday and improve the services we provide. Already 80% of Jobseeker’s Allowance claims are made online, but we know there is more to do to bridge the digital gap. Anyone contacting our staff today will probably go to a Jobcentre or call us on the phone. In the coming years, our vision is that most of those interactions will be done via the web or a mobile phone interface. We will continue the move over to automated transactions, which will improve the claims process and free up resources so that our frontline staff can focus more on helping people into work. What DWP does matters for society and the millions of customers who depend on our services. We are reforming the welfare system to ensure it promotes work and helps people lift themselves out of poverty. We are also bringing in root-and-branch change to the pensions system, including a new state pension that increases the attractiveness of saving for retirement. The scale of the change and the challenge is huge. It is this unique intellectual, cultural and operational challenge that persuaded me to come to DWP, having spent most of my career at companies like Goldman Sachs and Vodafone. I have seen for myself during various visits, such as one to Hackney Jobcentre, the pride and passion of DWP staff and the excitement about the potential for change. I am not alone in embracing this challenge, it is a trend we are seeing across the Department with more and more best-in-class digital talent from outside Government seeing DWP as their Whitehall destination. At DWP, we offer the chance to get involved in something that is genuinely transformational and unparalleled in terms of complexity and scale by anything in the UK private sector. Today we take the next step in a journey of changing the DNA of the DWP. We are starting the recruitment for 25 posts to further enhance our already burgeoning digital talent and capability. It builds on the steps we have already taken including launching the DWP’s first-ever Digital Academy. The Academy will provide a range of intensive training courses in how to work in our digital programmes. It is supported by other departments, including Government Digital Service who are forerunners in blazing the trail for digital innovation in Government. We want hundreds of our key employees to go through it in coming months. The new recruitment focuses on the following roles: Web Ops, Developers and User Experience (UX) designers. They will make a huge difference in improving the customer and claimant experience. The projects the successful individuals will work on are across a whole range of major programmes including the Single Tier pension, Carers Allowance and Personal Independence Payment as well as Universal Credit. This most radical transformation of the welfare state in Britain has understandably attracted attention in the media. But, as I saw recently in Hammersmith Jobcentre, Universal Credit is profoundly better than the service it replaced and is changing lives for the better. I truly believe that we are making good progress in the development of an enhanced future service that we will start testing later this year. But we are not complacent about the digital challenges ahead and we need to embrace digital change in Government. We need to be even more confident and bold, embracing digital innovation and taking a radically new approach, testing new ideas like Buzzfeed. It might mean that instead of the DWP working with suppliers to develop a piece of software in six months, that we up the pace and try to do it every two weeks - allowing us to continually improve the service and test new ideas. Enhancing our digital capability will help us do this and deliver a better service for the public. Today is another important step in that journey. You can find out more detail about the jobs on offer here: Web Ops: http://ow.ly/wKhxI Java Developers: http://ow.ly/wKhiL User Experience designers: http://ow.ly/wKg74

By Joseph Elm Program Integration Manager Software Solutions Division In today’s systems it’s very hard to know where systems end and software begins. Software performs an integrating function in many systems, often serving as the glue interconnecting other system elements. We also find that many of the problems in software systems have their roots in systems engineering, which is an interdisciplinary field that focuses on how to design and manage complex systems over their life cycles. For that reason, staff at the Carnegie Mellon University Software Engineering Institute (SEI) often conduct research in the systems engineering realm. Process frameworks, architecture development and evaluation methods, and metrics developed for software are routinely adapted and applied to systems. Better systems engineering supports better software development, and both support better acquisition project performance. This blog post, the latest in a series on this research, analyzes project performance based on systems engineering activities in the defense and non-defense industries. The Case for Systems Engineering An understanding of the value of systems engineering is necessary to justify a project’s investment in systems engineering resources and activities. In 2010, the DoD encouraged the National Defense Industrial Association (NDIA) and the SEI to develop a stronger business case for systems engineering by expanding on a 2007 survey of systems developers. We responded by conducting a new survey in collaboration with not just the NDIA, but other professional organizations including the IEEE-Aerospace and Electronic Systems Society, and the International Council on Systems Engineering (INCOSE). For this latest study, we surveyed 148 individual projects at organizations developing systems to obtain answers to the following questions: What systems engineering activities do you perform on your project? How well does your project perform against cost, schedule, and technical objectives? How challenging is your project? Although most of the surveyed projects supply systems for the U.S. defense sector, we also received some responses from organizations serving other market sectors and operating in different countries. We distributed a questionnaire that collected information from participating projects along three dimensions: systems engineering (SE) capability. We assessed systems engineering capabilities deployed on the project by investigating both the presence and the quality of work products resulting from SE activities. These work products were selected from those listed in the CMMI framework by a panel of SE experts. Based on this assessment, SE deployment for each project was categorized as low, medium, or high. project performance. We assessed project performance as a combination of cost performance (satisfaction of budget), schedule performance, and technical performance (satisfaction of requirements). Again, based on this assessment, project performance for each project was categorized as low, medium, or high. project challenge (PC). Some projects are inherently more challenging than others due to factors such as size, duration, technology maturity, interoperability, requirements, etc. Based on the combination of these factors, project challenge was categorized as low, or high. In the 2012 report The Business Case for Systems Engineering Study: Results of the Systems Engineering Effectiveness Study, we reported our initial results, which found a strong relationship between SE deployment and project performance. In particular, as projects applied more SE capabilities, they delivered better performance. For example, among those projects deploying the least SE, only 15 percent delivered the highest level of project performance. Among those deploying the most SE, however, 56 percent delivered the highest level of project performance. As one would expect, our initial analysis also showed an inverse relationship between project challenge and project performance—more challenging projects do not perform as well as less challenging ones. But, we also learned that SE practices became even more valuable when used with these challenging projects. For the most challenging projects, the number of projects delivering high project performance increased from 8 percent to 62 percent with increased SE deployment. This result shows the increasing need for SE as projects become more challenging. Defense vs. Non-Defense Projects In our latest research, we revisited this data to study differences between projects in the defense and non-defense domains. We wanted to know if there were differences in the application of systems engineering, the performance of projects, and the effectiveness of the applied SE. Our aim with this most recent analysis was to identify best practices in one domain that could be transplanted to the benefit of the other domain. As detailed in our recently published report on this latest research, The Business Case for Systems Engineering: Comparison of Defense-Domain and Non-Defense Projects, our latest analysis compared 75 defense projects with 32 non-defense projects from the fields of electronic equipment, communications, and health care. The 75 defense projects were a subset of the 116 defense projects responding to the 2012 survey. This subset was chosen to produce a sample of projects representing the same degree of project challenge as the non-defense projects. We found that—when examining budget, schedule, and requirements satisfaction—performance against budget and technical requirements was the same in both defense and non-defense domains. With respect to schedule, however, non-defense projects performed somewhat better than did defense projects. We examined the degree to which systems engineering was deployed among defense and non-defense projects. We measured deployment of SE practices on a scale of 1 (little SE deployment) to 4 (significant SE deployment). We also examined both the total SE applied to the project, and the SE applied in specific areas such as product architecture development, requirements, and verification. As shown in the figure below, we found that, for almost all specific areas, non-defense projects performed less systems engineering than defense projects. When we analyzed the relationship between SE deployment and project performance, we found results similar to those of the initial study. In both the defense and non-defense domains, projects that deployed more SE delivered better performance. However, we also found that most of the relationships between SE deployment and project performance were stronger in the non-defense domain than they were in the defense domain. Thus, the non-defense projects seem to getting a larger return from the SE investments. This result is shown in the figure below. Wrapping Up and Looking Ahead The findings of our analysis are summarized as follows: For both defense domain projects and non-defense projects, projects with better SE deployment deliver, on average, better project performance. Non-defense projects deploy slightly less SE than defense projects Non-defense projects deliver slightly better project performance than defense projects, primarily due to better adherence to schedule. The strength of the relationships between SE deployment and project performance is stronger for non-defense projects than for defense projects. The data we have collected does not provide any insight into the root causes of these differences; it merely indicates a measurable difference. These differences may be the result of an evolutionary construct. Perhaps non-defense projects are adopting new development technologies more rapidly than defense projects. Or the difference could be attributed to the processes that defense projects follow. Further study of these questions may identify processes and practices that can be transplanted from the non-defense domain to the defense domain to produce improved performance and acquisition in the Department of Defense. Looking ahead, I am interested in conducting further research with projects in non-defense domains to gain insight into how they deploy their systems engineering and to develop a more concrete understanding of why non-defense projects are more effective in their systems engineering performance, with respect to schedule. We welcome your feedback on this research in the comments section below. Additional Resources The technical report The Business Case for Systems Engineering: Comparison of Defense-Domain and Non-Defense Projects, can be downloaded at http://resources.sei.cmu.edu/asset_files/SpecialReport/2014_003_001_91760.pdf. The technical report The Business Case for Systems Engineering may be downloaded at http://www.sei.cmu.edu/library/abstracts/reports/12sr009.cfm.

Think about your workplace and the meetings that you attend throughout the week. Which ones are well-run and which are a waste of time? What is it about the well-run meetings that make them so useful and effective? Do they start on time, end on time and stick to an agenda? Are there next steps and assignments? Are side conversations quickly averted? Time Wasted How much time do we spend in meetings?  According to the online article How Much Time Do We Spend in...

Its been a busy few weeks since our last update. Some of us have started on a new project, we’ve delivered a presentation to a conference of 400 people whilst still continuing on the journey of building our capability. Shortly after the hack day we were asked to present our team at the ITPD Conference. The theme was "Capability" so it was a natural for us to be asked to demonstrate our skills. We decide the use the Raspberry Pi robot we’d already built and talk about the journey we have been on, demonstrating the skills we’ve learnt so far. We were allocated 30 minutes for our session. By using Skype we were able to show live the robot moving around our purpose built maze. We demonstrated the robot moving, first by user input (which resulted in a lot of errors and crashing!) and automated script (which worked perfectly). We were very well received with many people commenting on how what we were doing was long over due and interested to follow what we were doing. We also received quite a lot of interest from people with coding experience on wanting to join the team! Also on the day we had a session on Agile Delivery, presented by Jim Downie. This involved nominating a project manager around our table, who were then given a list of requirements, weighted with differing scores depending on complexity. We were then tasked to build the requirement using coloured blocks. Queue lots of discussions and frantic searching for "one more green 2×2 block!". PM’s, BA’s and "Dev’s" working in harmony……(!)   Also in May, four of the team (Nathalie, Mike, Donna and Paul) started work with the PIP project, providing some assistance with writing a PIP Information Service. There is an aggressive deadline for delivery of PIPIS (anticipated go-live for PIPIS to coincide with phase 3 release for PIPCS in September ‘14). We have been using html and css and using the Razor syntax based on the C# programming language, the language used most often with ASP.NET MVC. There are also elements of JavaScript and JQuery which we have been getting to grips with, and we have been using the MVC framework [model view controller] which is a software architectural pattern for implementing user interfaces (for example, the PIPIS screens for Personal Details, Entitlement Details, Payments Details etc) This week, Capgemini have transferred the work we have done into their own dev environment and are guiding us through our first Agile project management environment, with co-location, pair programming, daily stand ups, 2 week time-boxed delivery Sprints and the like. Seeing first hand what a true Agile delivery should look like - and at pace - is going to be invaluable to us and we hope to bring what we have seen and learned back into DWP Finally, Adrian and I are still here in Carers. We are learning Java (from a real life, thick, heavy textbook!) continuing our learning of Agile in a live environment and gleaning as much knowledge as possible from the guys on the project. We have been busy compiling a list of software used by the project that will need procuring, following the processes involved for the fortnightly releases. In the coming weeks we have been asked to talk at Civil Service Live (North West) about building capability within DWP IT and continuing our own learning in our respective projects.

By David Keaton Senior Researcher CERT Secure Coding Initiative According to a 2013 report examining 25 years of vulnerabilities (from 1998 to 2012), buffer overflow causes 14 percent of software security vulnerabilities and 35 percent of critical vulnerabilities, making it the leading cause of software security vulnerabilities overall. As of July 2014, the TIOBE index indicates that the C programming language, which is the language most commonly associated with buffer overflows, is the most popular language with 17.1 percent of the market. Embedded systems, network stacks, networked applications, and high-performance computing rely heavily upon C. Embedded systems can be especially vulnerable to buffer overflows because many of them lack hardware memory management units. This blog post describes my research on the Secure Coding Initiative in the CERT Division of the Carnegie Mellon University Software Engineering Institute to create automated buffer overflow prevention. For greenfield software development projects, which are projects developed without constraints from any previous system, it may be practical to mandate a language subset or annotations. Most development efforts, however, are extensions or modifications of existing code. It is therefore necessary to find a solution to eliminating buffer overflows that can work with legacy code. The catalyst for this research occurred prior to my arrival at CERT, while I was consulting for a large company that deployed a 3-million-line code base. The same potential buffer overflow problem kept cropping up, so I launched a project at that company to remediate the problem. We picked several thousand instances of the potential buffer overflow bug. We then assembled 10 engineers and, after dividing the work equally, went in and fixed the bugs manually. What we found was that while we were successful in preventing a lot of bugs associated with buffer overflows, we introduced more bugs than we would have introduced had we been writing the code from scratch. We knew this because that company kept very good records about the quality of their code at each stage. Clearly, our approach was sound, but how we went about it was wrong. We needed an automated approach to buffer overflow protection to eliminate the potential for human error. Foundations of an Automated Approach Because C focuses on performance and being close to the hardware, bounds checks on array references and pointer dereferences are not part of the language. Sometimes a compiler can prove at compile time that an access is in bounds, but usually this is not the case. If the user desires bounds checking, then for the memory accesses that cannot be resolved at compile time, the compiler must insert additional instructions into the generated code to perform the checks. The time the processor takes to execute those extra instructions at run time is the performance overhead of a bounds checking mechanism. I have found that many researchers report excellent low overheads for automated buffer overflow elimination schemes while in practical use they are actually substantially higher. For example, one set of researchers may report very low overheads by taking advantage of a trap on misaligned memory accesses, but many processors do not trap when dereferencing misaligned pointers. As a result, the check for misalignment must be performed by software, which causes a much higher overhead. Another set of researchers may use a shadow memory by dividing up the address space for the program and for the overflow checking. Again, this may not work in the practical domain because it requires operating system modifications. I investigated what performance could be achieved in the practical realm and how that performance could be improved for deployment of an automated, compiler-based memory safety checking tool. Because the goal was to support legacy code, there were additional constraints: any approach used could not require changes to the source code application binary interface (ABI) compatibility should also be maintained because developers do not have control over the complete system on which the application will run  it should be possible to link checked code with unchecked binary libraries for which the source code might not be available I began with two memory safety checkers that meet these criteria, SAFECode and SoftBound.  Methodology I selected a small set of programs to use as a benchmark from the SPEC CPU 2006 suite because they are written in C and compile cleanly with Clang/LLVM 3.2, the latest version of the compiler for which both SAFECode and SoftBound were available. I measured the overhead with just the source code available on the Internet and then applied my own optimizations and measured it again. As described in an August 2014 technical note I wrote with Robert Seacord, Performance of Compiler-Assisted Memory Safety Checking, I made three performance enhancements to SoftBound to investigate their effects on performance. First, I hoisted spatial memory access checks out of loops when the loop bounds were known on entry. As an example, consider the following function: #include <stddef.h>void foo(int *a){  for (size_t i = 0; i < 100; ++i)    a[i] = i;} The figure below shows the generated LLVM code, including the effect of hoisting the check of the store to a[i] out of the loop. The optimization removes the struck-out text preceded by minus signs in the figure and inserts the bold text preceded by plus signs. The beginning of the check is adjusted to be the first element accessed (the beginning of array a), and the length of the check is adjusted to include all accesses that will be performed by the loop (400 bytes), rather than checking each iteration separately. define void @foo(i32* nocapture %a) nounwind uwtable ssp {entry:  %0 = tail call i8* @__softboundcets_load_base_shadow_stack(i32 1) nounwind  %1 = tail call i8* @__softboundcets_load_bound_shadow_stack(i32 1) nounwind+ %bitcast = bitcast i32* %a to i8*+ tail call void @__softboundcets_spatial_store_dereference_check(i8* %0, i8* %1, i8* %bitcast, i64 400) nounwind  br label %for.bodyfor.body:                                         ; preds = %for.body, %entry  %i.04 = phi i64 [ 0, %entry ], [ %inc, %for.body ]  %conv = trunc i64 %i.04 to i32  %arrayidx = getelementptr inbounds i32* %a, i64 %i.04- %bitcast = bitcast i32* %arrayidx to i8*- tail call void @__softboundcets_spatial_store_dereference_check(i8* %0, i8* %1, i8* %bitcast, i64 4) nounwind  store i32 %conv, i32* %arrayidx, align 4, !tbaa !0  %inc = add i64 %i.04, 1  %exitcond = icmp eq i64 %inc, 100  br i1 %exitcond, label %for.end, label %for.bodyfor.end:                                          ; preds = %for.body  ret void} Figure 1: LLVM Code Showing the Effect of Hoisting a Bounds Check Out of a Loop To prevent spurious error reports, with the check being executed and the offending access not executed, the check is hoisted only if it post-dominates the first basic block inside the loop. The check post-dominates the loop’s first basic block if all possible execution paths from that basic block to a loop exit pass through the check. I next hoisted bounds checks out of a function and into its callers when the compiler could see all calls to the function, so that a bounds check will be executed somewhere (if necessary) if it is deleted from its original function. To see how this might be beneficial, consider the following program: #include <stdio.h>#include <stdlib.h>static void foo(unsigned char *a){  for (size_t i = 0; i < 100; ++i)    a[i] = i;}int main(void){  unsigned char a[100];  foo(a);  for (size_t i = 0; i < 100; ++i)    printf(" %d", a[i]);  putchar('\n');  return EXIT_SUCCESS;} If the bounds check for the access to a[i] is first hoisted outside the loop in foo and then up to main, it is now in a position where the size of array a is known. Because the length of the bounds check is also known, it can be compared against the size of a to determine if the bounds check can be eliminated entirely. This elimination occurs in the case of the preceding program. If, after hoisting the bounds check into the caller, there still is not enough information to eliminate the bounds check, it is performed within the caller in hopes that it can still be eliminated along other call paths to the original function. The mechanism used to accomplish this optimization is to treat a bounds check in the called function as a precondition for that function (called a requirement by Plum and Keaton). In this case the precondition is that the memory space pointed to by a is at least 400 bytes long. Then all requirements are checked at their call sites to see whether the bounds checks can be eliminated for that call path or merely hoisted out of the called function. Inlining can accomplish the same thing by effectively hoisting a bounds check into the calling function, where there might be enough information to eliminate the check. Therefore, this mechanism provides a benefit in cases where inlining is not performed, such as when the called function is large. SoftBound is implemented so that it operates on optimized code. First the optimizations are run, then SoftBound is run, and then the optimizations are repeated in an attempt to improve any code added by SoftBound. I found that unrolling loops thwarted some attempts to hoist bounds checks. Fully unrolled loops contained a sequence of memory accesses in straight-line code in place of one access within a loop. I therefore disabled loop unrolling. Alternative approaches would have been to disable it only for the first pass of optimizations or to write an additional optimization to combine the adjacent bounds checks that result from unrolling loops. The third change was to test the performance of bounds checks on stores only (to prevent arbitrary code execution), or on strings only (because incorrect string management is a leading cause of vulnerabilities), or only on stores to strings. Limiting the bounds checks in this way can provide some insight into the tradeoff between security and performance. I also measured the performance of SAFECode and SoftBound with checking turned off, to discover the performance effect of the maintenance and propagation of metadata via the object table maintained by SAFECode or the pointer table maintained by SoftBound. To accomplish this, I disabled the portions of SAFECode that emit pointer arithmetic checks and bounds checks, and disabled the portions of SoftBound that emit bounds checks, leaving the bookkeeping code only and thereby establishing a ceiling for the performance benefit of bounds check optimizations. Results Overall, with the optimization of hoisting bounds checks out of loops, the average program ran 4.72 times slower with buffer overflow checking, a rate that was significantly slower than I had hoped. The programs experienced an average slowdown of 5.36 times without the optimization. Next, as is also detailed in the technical note, I charted the slowdown for performing bounds checks only on stores, only on strings, and only on stores to strings, in addition to hoisting bounds checks out of loops and functions. With the optimization plus checking only stores, the average program ran 2.58 times slower than it did without buffer overflow checking. All three cases provided a performance benefit. The three are similar in magnitude, indicting that a useful tradeoff between performance and security may be achieved by checking only stores. Next, I investigated the performance breakdown by examining the metadata propagation performed by SAFECode and SoftBound to gain more insight as to how the time is spent. There are two parts to run-time automatic buffer overflow elimination. One part involves tracking the bounds information that is needed to check for buffer overflows. This metadata must be available for the other part, the actual checking against the bounds to detect buffer overflows. In SAFECode, the actual checking occurs at pointer arithmetic and at loads and stores. In SoftBound, the actual checking occurs only at loads and stores. When I turned off all of the checking code except for what was needed to pass the bounds data around inside the program, I found that the overhead required by metadata propagation by itself yielded an average overhead of 2.07, which is almost the entirety of the overhead. For SoftBound, which is the main codebase that I was working with for buffer overflow elimination, there would be a vast advantage to working on metadata propagation as opposed to the checks that occur at each store. I compared this with SAFECode. SAFECode has much worse overhead overall. The average overhead, without any optimization, was 41.72 times slowdown with the buffer overflow checking, but with no optimization. When I checked the overhead of SAFECode’s metadata propagation only, the average slowdown was only 3.02, which is not much worse than SoftBounds’ metadata propagation overhead, which was 2.07. SAFECode’s slowdown was 41.72, and its metadata slowdown was only 3.02, which clearly shows that in SAFECode the most benefit can be gained by increasing the speed at which checks occur at pointer arithmetic and at loads and stores. The conclusion drawn from the results above is that each pointer overflow checking mechanism needs to be considered separately. Challenges and Adjustments In addition to hoisting bounds checks out of loops, I also tried to hoist them out of functions. I reasoned that if I hoisted a bounds check into the caller, there might be information in the calling function to reveal the size of the objects, eliminating the need for a bounds check. Unfortunately, hoisting bounds checks out of functions did not provide a significant benefit. One reason is that this issue is already addressed by inlining. So, if I have a small function working on an array, it will often get inlined into the larger function and then see the object in the calling function at compile time. Another challenge involved the physical structure of the LLVM compiler, which is divided into many different passes. LLVM will perform a pass through the code that it is compiling, and then it will perform another pass. There are function passes, which are performed once for every function in the program, and there are module passes, which are performed once for every module, which is a file (i.e., a collection of functions). SoftBound, it turns out, is a module pass so that it can operate on multiple functions concurrently. In the way LLVM is structured, however, a module pass cannot use information generated by a function pass, which uses information generated by another module pass. This design decision allows for the parallelization of different passes, but negatively impacts constant folding. As a result, constant folding does not occur as intended. If the code contains the computation, 2+2, it will remain as a 2+2 because constant folding is a function pass that makes use of information from another module pass. To mitigate the issue, I added the constant folding myself, manually, to make it work. Now, the 2+2 comes out as a 4, which makes optimization much easier. Looking Ahead There is hope that automated buffer overflow checking will one day perform fast enough to work in future performance-critical systems. After this study was finished, Intel announced a set of future hardware protections called "MPX" Memory Protection Extensions. Once Intel installs those hardware improvements, it is possible that we will be able to use compiler enforced buffer overflow elimination on performance-critical code as well. I welcome feedback on this research in the comments section below.  Additional Resources To read the technical note, Performance of Compiler-Assisted Memory Safety Checking, by David Keaton and Robert C. Seacord, please visit http://resources.sei.cmu.edu/library/asset-view.cfm?assetid=299175.

It’s a great time to be working at the Department for Work and Pensions. We are changing the way we work to make sure that we build digital services which have users at their heart and in a way that allows continuous improvement and iteration. We are using agile approaches and bringing those skills and that culture into DWP to make sure that we deliver the right services for our claimants and customers. For us Digital means One of the critical enablers for our business transformation is digital. For DWP digital means one of the ways that we interact with our claimants and customers. It’s also the automated and efficient way we want our business to operate, and it’s the working culture, performance measures and tools we use to develop our services. Our Digital Academy Leeds Digital Academy students Our Digital Academy is at the heart of our commitment to improve the way we deliver services. Mixing up theory and practice - with site visits and work placements thrown in too - DWP’s Digital Academy programme provides an intensive burst of digital training to specific colleagues across the Department. Based in either London or Leeds, the initial six-week, face-to-face training programme aims to develop the skills needed for us to create and deliver our own successful digital services. The course involves 4 days’ training each week and is led by a mix of DWP officials, members of the Government Digital Service, and external experts. Refresher sessions, follow-up gatherings and online communities keep learning moving forwards after the initial 6 week course. Civil Service Live It’s exciting to be part of something that’s truly transformational and we are proud to be able to demonstrate some of our progress at Civil Service Live 2014. At the Liverpool, Newcastle & London Civil Service Live 2014 events we will be showcasing the work of our Digital Transformation Group, the Digital Academy and 3 of our digital exemplar services. These exemplars services  are Single Tier Pension , Carers Allowance Online and Universal Credit. We’re keen to engage and share information about how each of our exemplars are developing over time and how we are building digital capability within DWP through the Digital Academy. If you are at any of the events I hope you’ll make time to find our stand and come along to say hello. You can ask about, or suggest, future content for this blog by adding comments below and subscribing to future updates. I look forward to hearing from you. You can follow us on Twitter @DigitalDWP.

  Complaint Procedure (With More "Fixes" to Come)  On June 29 - for the twenty-first straight year - - I will have the pleasure to share my expertise on handbooks at SHRM’s 2015 Annual Conference.  Over the years, approximately 10,000 SHRM members have walked away with a new way of looking at their handbooks: "every word counts." Looking back on my Annual Conference programs, my greatest regret has been echoed in my speaker evaluations:  too much material and too little time.  In seventy-five minutes, attendees were given many "take aways" - - but I barely scratched the surface. Until...

I’m sitting in a near pitch black room with only the glare of two monitors and a window into the adjacent room. In it sits a Government Digital Service (GDS) researcher and a member of the public. As I watch the monitors and occasionally look through the window, my immediate thoughts are "what we’ve delivered is good, but how can it be better?" It’s a user research day in Salford for the Carer’s Allowance Digital Service (CADS) with fellow members of the team that have helped deliver the current beta version of CADS. We’re here to observe users completing a prototype version of CADS that the GDS research team has put together to test potential improvements. The users know they are going to help complete a service but nothing more specific, they have all placed themselves low on the Digital Inclusion scale; a 1 to 9 rating of internet and computer skills. Each user is shown into a room with a computer and recording equipment to capture the onscreen clicks, reactions to prototype design choices and answers to the questions the researcher asks. The improvements that we are testing, observing and analysing today are as a result of previous research days that the team have run. In the room containing the team, each member is busily scribbling observations about what we are witnessing while the GDS researcher offers occasional help in completing CADS and asking pre agreed questions to help the team gain insight. Each observation is taken back to the Preston Office Centre where the CADS team is based. These observations are affinity mapped (Post It notes placed on a work board), these are grouped together into themes which will uncover insights into what we’ve seen on the day and what actions can be taken away to improve the service. Some observations confirmed an insight from a previous testing day; these are then turned into user stories to be delivered into CADS beta by the Digital Service Team and the development partner Valtech. So far roughly 60 hours of user testing has gone into the current CADS system, various prototypes tested with lots of users with different circumstances and I.T abilities to bring together a service that is accessible to all, yet tailored to the individual. The testing has helped reduce completion times for users from 45 minutes to 28 minutes and seen an uptake in digital claims from 25% to 50% in only a few months. The CADS team are currently working towards live accreditation which requires the service to meet the 26 GDS standards, the first of which is "Understand user need", something the CADS team are very passionate about. These days provide a fascinating insight into user needs, observing users struggle to complete certain sections due to their circumstances or lack of confidence in their I.T skills really drives home the message that "what we’ve delivered is good, but how can it be better?".  

By Neil Ernst Member of the Technical Staff Software Solutions Division This post is co-authored by Stephany Bellomo Continuous delivery practices, popularized in Jez Humble’s 2010 book Continuous Delivery, enable rapid and reliable software system deployment by emphasizing the need for automated testing and building, as well as closer cooperation between developers and delivery teams. As part of the Carnegie Mellon University Software Engineering Institute's (SEI) focus on Agile software development, we have been researching ways to incorporate quality attributes into the short iterations common to Agile development. We know from existing SEI work on Attribute-Driven Design, Quality Attribute Workshops, and the Architecture Tradeoff Analysis Method that a focus on quality attributes prevents costly rework. Such a long-term perspective, however, can be hard to maintain in a high-tempo, Agile delivery model, which is why the SEI continues to recommend an architecture-centric engineering approach, regardless of the software methodology chosen. As part of our work in value-driven incremental delivery, we conducted exploratory interviews with teams in these high-tempo environments to characterize how they managed architectural quality attribute requirements (QARs). These requirements—such as performance, security, and availability—have a profound impact on system architecture and design, yet are often hard to divide, or slice, into the iteration-sized user stories common to iterative and incremental development. This difficulty typically exists because some attributes, such as performance, touch multiple parts of the system. This blog post summarizes the results of our research on slicing (refining) performance in two production software systems. We also examined the ratcheting (periodic increase of a specific response measure) of scenario components to allocate QAR work. Slicing and Allocation in Agile Projects QARs are often provided in unstructured and unclear ways. QAR refinement is the process of elaborating and decomposing a QAR into a specifiable, unambiguous form achievable in one release and finding suitable design fragments. This process, also known as slicing or sizing, is typically applied to user stories, short, textual requirements popularized by agile methodologies. The requirements community calls such iterative refinement the Twin Peaks model because you iteratively cross between the ‘peaks’ of requirements (the problem domain) and architecture (the solution domain). QAR refinement should produce a unit of work small enough to test, small enough to fit in an iteration, and useful enough to produce value. It should separate abstract requirements into constituent parts so they and their interrelationship can be studied. "[QAR refinement] is a design activity," a software developer recently posted on Twitter. "It surfaces our ignorance of the problem domain."  There are a number of ways to size requirements. Some methods are based purely in the problem space, some are based on the work involved in satisfying the requirement, and some are a mixture of analyzing the problem and possible solutions. These approaches include And/Or decomposition: Split on conjunctions and and or in the requirement. For example, for the requirement "display user name and account balance," one piece would deliver the user name, the other piece the account balance. Acceptance or test criteria: Satisfy one criterion per slice. User stories (an Agile form of requirement with user-focused textual descriptions) often have a list of reasons  for accepting the requirement (story) as done. For instance, "login function works, password validated." Each criterion is a slice. Spike: Add exploratory spikes to backlog where steps are unknown—investigate versus implement. Spikes are specific work dedicated to learning, such as the impact of a new version of a key framework. Horizontal slices: Slice according to architectural layers (such as database, UI, business logic). Commonly seen as an anti-pattern since it creates false divisions of the work. Operation type: Slice according to database operations. One story for create operations, one for read, etc. Hamburger slicing: Create horizontal slices that map steps in a use case, then extend vertically according to improved quality criteria. For example, a user login use case has steps for validating passwords, displaying user history, etc. These are then expanded ‘vertically’ for different options, such as login options OAuth, social media account, or simple user:password. In the two examples described below, we found the most common slicing approach was ratcheting, popularized by Martin Fowler in his blog post "Threshold Testing." Our Approach  Our research question was How do projects slice quality attribute requirements into iteration-sized pieces, and how are these pieces allocated to a project plan or sprint backlog?  QARs, including performance, security, and modifiability, significantly influence architecture. We conducted case studies of production systems (we omit identifying details to protect confidentiality). For each case, we conducted interviews with team leads and architects to understand the teams’ deployability goals and architectural decisions they made to enable deployability. We then performed semi-structured coding to elicit common aspects across the cases.  Based on our analysis, we described the quality attribute work as a series of increments, representing the current state of the QAR, as the requirement is progressively refined and enhanced. Table 1 below shows our conceptual framework. We used a scenario to define each state, with each scenario consisting of a stimulus, some context or background, and a response to that stimulus. Our approach is analogous to an Agile, "Behavior-Driven Development" (BDD)-style "Given <>, When <>, Then <>" model. The value column reports the outcome of that increment. Based on our interviews, refinement happens not only in the stimulus response, but also the context and the stimulus itself. In other words, one can tighten the response (from 10 to 1 ms, for example); one can loosen the context (for all customers instead of just certain customers); or one can expand the range of stimuli (any order versus special order types). We believe refinement of QARs using scenarios creates a reasonably sized chunk of work for analysis, implementation, and possibly even testing during an iteration. Project A Project A developed financial support software for a mid-size firm. The software supported the buying and selling of financial securities. Performance was a key concern, particularly at the close of the financial day. Customers may have to pay interest if they have to borrow large sums of money to hold sell orders overnight if they were not processed by the close of trading. Development cycles were deliver-on-demand, rather than fixed iterations, while customers provided feedback to the developers that informed the work in the subsequent state. Table 1 shows a summary of the state transitions. Below, we describe states that we identified in the first project: These states were refined in three different dimensions (stimulus, context, response). Stimulus: A-S4 to A-S5 illustrates refinement in the stimulus dimension, moving from a single- to multi-user perspective. A second example is A-S2 to A-S3, which illustrates moving from batch processing to individual transaction processing. Context: The initial requirement focused development on the system behavior (simple baseline case) to test ideas before dealing with complexities and uncertainties of the environment. A-S1 to A-S2 illustrates refinement in the context dimension, moving from manual processing to the autopilot solution, by adjusting the boundary of the system with respect to its environment, including the user’s role. Moving from A-S4 to A-S5 accounts for the complexities of the rotary algorithm in the environment. Response: A-S3 to A-S4 illustrates refinement in the response dimension by ratcheting the response measure. The response measure value was refined to less than 1 second for processing individual transactions. This analysis demonstrates that each refinement we captured reflects a change in the evolving requirements context, such as increasing number of orders to be processed or reducing performance threshold for a single order. The team also must make tradeoffs in the measurement criteria, for example, first optimizing for throughput (A-S2) before dealing with latency (A-S3, A-S5). With three dimensions to adjust (stimulus, context, response), ratcheting in one dimension may require easing up on another to make progress. Project A explained that separating performance-related from feature-related requirements in the backlog or when planning and allocating work to iterations was not useful. Project A followed evolutionary incremental development by doing performance-related analysis work concurrently and loosely coupled from implementation sprint work. Separating feature work from performance work is a common pattern that we observe in industry. As analysis work was completed, the work was allocated to sprints. Well-understood changes refining existing features, such as A-S1 through A-S3, were allocated to implementation sprints with minimal analysis. However, in cases where significant analysis was needed (e.g., A-S4), the team created an exploratory prototype to learn more about the problem and investigate alternative solutions while continuing to mature the system and implement ongoing requirements. For A-S4 the changes were more substantial, so the work was allocated to multiple sprints. Project B Project B, described in Table II, was also a financial system with stringent performance requirements. The system functionality included high-speed, stock order processing. Like Project A, the performance iterations are described as state transitions. This project was in the pre-release phase, focusing on evaluating the design to ensure it could meet requirements. The team used a scenario-driven approach to investigate performance risks. Output of scenario analysis is input to the evolving system design and architecture of the next state, as described below. The analysis of this example also shows changes to the stimulus, context, and response, as summarized below. Stimulus: BS-3 shows variations in the stimulus with the artifact changing to focus on a specific part of the system, the queue. Context: B-S2 shows variations in the context by increasing the concurrently processed orders to 1,000 orders while maintaining response time at 0.1 ms. Response: The response goal remained consistent at 0.1 ms throughout all the state transitions. In interviews, however, we discovered that this target was reached over several rounds of tweaking the design, analyzing the response under increasingly stringent conditions, such as described in B-S2, and making incremental improvements. Project B did not separate feature-related and performance-related requirements in their software development lifecycle either. Like Project A, the team conducted exploratory performance analysis and design work concurrently with maturing other features and continuing implementation. Work was integrated into implementation sprints as it became better defined through analysis of design artifacts, prototyping, or both.  Conclusions Ultimately, the purpose of refining QARs is to decompose a stakeholder need or business goal into iteration-sized pieces. The process of allocation then takes those pieces and determines when to work on them. This activity involves both analysis and design: refinement and allocation are explorations of the problem and solution spaces, and evolutionary, iterative development allows for course changes when new information is acquired. Developers work toward satisfying cross-cutting concerns in the context of the effort and ultimate value. We observed that developers refined performance requirements using a feedback-driven approach, which allowed them to parse the evolving performance requirement to meet increasing user expectations over time (expressed as state transitions). Within each state transition, developers refined cross-cutting concerns into requirements by breaking them into their constituent parts in terms of the scope of the system and response to stimuli in a given context. The system and cross-cutting performance requirements evolve as stimuli, context, and response are ratcheted. We see evidence of projects that are better able to sustain their development cadence with a combination of refinement and allocation techniques guided by measures for requirement satisfaction, value, and development effort. As we retrospectively analyzed these examples, we found that these teams did not follow a formal technique; however, they did have common elements in how they refined the work into smaller chunks, enabling incremental requirements analysis and allocation of work into implementation increments. Martin Fowler describes ratcheting as a performance requirement in terms of increasing a specific response measure. Based on what we learned from our interviews, we suggest that this ratcheting concept can be broadened. For example, changes in the evolving context, such as increasing the number of orders to be processed or reducing the performance threshold for a single order, allow for breaking a cross-cutting concern to a reasonably sized chunk of work for analysis, allocation, or testing. We suggest that these examples, which demonstrate ratcheting in multiple dimensions, could be useful for teams struggling with how to break up and evolve cross-cutting concerns during iterative and incremental development. Additional Resources This blog post is a summarization of a paper that was presented at the International Conference on Software Maintenance and Evolution. 

When considering new job opportunities or career moves, there are lots of factors that come into play. Of course people think about salary, benefits, and potential for growth. Many employees also focus on their "fit" [1] with an organization’s culture. But where does that culture come from? The Attraction-Selection-Attrition (ASA) Model [2] was developed to explain how cultures form, how they are reinforced, and why they sometimes need to be forcibly changed. Attraction: Job seekers are attracted to certain companies more than others. Maybe this is because of the company’s brand recognition, its mission, or...

By Mike Konrad Principal ResearcherSEI Software Solutions Division As recent news attests, the rise of sociotechnical ecosystems (STE)—which, we define as a software system that engages a large and geographically-distributed community in a shared pursuit—allows us to work in a mind space and a data space that extends beyond anything that we could have imagined 20 or 30 years ago. STEs present opportunities for tackling problems that could not have even been approached previously because the needed experts and data are spread across multiple locations and distance. Since STEs can be complex and have many diverse stakeholders, a key challenge faced by those responsible for establishing and sustaining them is eliciting requirements to inform their development efforts. Yet stakeholders often have requirements that they are not aware of, so they do not specify them. Uncovering these unstated requirements can be hard and is not well-supported by traditional approaches to requirements elicitation. This blog post describes initial results of an effort by researchers at the Carnegie Mellon University Software Engineering Institute—in addition to myself, the team included Nancy Mead, Robert Stoddard, and Mary Beth Chrissis—aimed at developing an approach for determining the unstated needs of stakeholders typical of large, diverse programs and especially STEs. Foundations of Our Work Apple founder Steve Jobs famously said "But in the end, for something this complicated, it's really hard to design products by focus groups. A lot of times, people don't know what they want until you show it to them." What Steve was saying is that there are often critical product features that end users are not even aware they need, and that an insightful designer will figure out that need and offer it.  Therefore, "unstated requirements" can remain unstated not because users hold back or forget them, but rather because they are unaware of them. Users may have sensed something was wrong with the current software system in particular situations, but did not explicitly recognize it as a problem that could or should be addressed. Perhaps with additional information or after reflection, such users might come to realize and then express what it is that occurred that could have been prevented or mitigated. Unstated user and stakeholder needs can be a source for innovative system features and key architectural drivers and therefore can fundamentally impact the design, implementation, performance, and evolution of complex software systems. For example, the new iPhone 6 sports 64-bit computing on the new A8 chip (as did the previous generation iPhone 5S with its A7 chip) which might lead observers to ask, Why 64 bits? One reason might be: with the amount of random access memory (RAM) that exists in smartphones, we are approaching the maximum RAM storage possible for 32-bit computing, which is four gigabytes. By introducing 64-bit processors now, Apple may be positioning the broader iPhone ecosystem for 64-bit processing, setting the stage for an explosion of memory-intensive healthcare and home automation apps. Such forethought illustrates an appreciation for subtle longer-term needs that are best addressed early in the product (and in this case, product-line) lifecycle.  Prior research conducted by a team on which I participated foresaw an emerging need for requirements elicitation involving multiple and diverse parties proactively engaged in an early and ongoing conversations about the health of the ecosystem as a whole and for greater visibility into the goals, activities, and capabilities of various ecosystem participants. From such discussions, preventable problems can be identified and globally-sound and efficient solutions can be jointly-developed.  Our research team thus decided to initially focus on this research question:  Could existing requirements elicitation methods that address unstated needs be adapted to work across a geographically distributed community of users, stakeholders, and software developers? A Review of Existing Methods  Early in the project, then, our team conducted a literature review to determine which requirements elicitation methods had been investigated addressed unstated needs had potential for automation and analytic tool support  Most current requirements elicitation methods are specification-driven and do not address unstated needs. Those that do either do so directly by only targeting a small number of stakeholders and a small number of features (e.g., as in prototyping and simulation) or indirectly by analyzing user scenarios for a range of distinct user types (e.g., persona-based approaches). While other methods have some unique benefits, the team selected the KJ method, because of its systematic approach to eliciting and organizing user experience data with the goal of identifying potentially innovative new features and quality attributes. The KJ method is named after Jiro Kawakita, who originally conceived it, and consists of the following steps:  Step 1: Evaluate existing knowledge of requirements Step 2: Design open-ended probing questions Step 3: Conduct KJ interviews Step 4: Analyze output to form context need/activity statements Step 5: Conduct KJ affinitization Step 6: Identify unstated needs and candidate innovative solutions Step 7: Conduct Kano Analysis to determine must-be’s, satisfiers, delighters KJ is typically applied in co-located settings in a time-boxed fashion. Our research focused on enhancing KJ so that it works in a distributed environment—one in which not only users but requirements analysts and involved technologists are at different locations.  Challenges Several challenges are common to requirements elicitation methods and not necessarily unique to our work. During requirements elicitation, underrepresenting a key stakeholder does more than simply omit the immediate impact that the stakeholder could offer. That stakeholder may be focused on a cross-cutting concern (e.g., security, dependability, usability, etc.) that has important consequences for other stakeholders and end users. Stakeholders can include mid-stream suppliers to the product or system being developed and/or smaller niche solution providers—it is easy to overlook stakeholders that can be the source for innovative requirements. Another, more formidable, challenge that we encountered is that many requirements elicitation methods tend to be applied in co-located, time-boxed team settings, providing only one or two days to identify and analyze use cases or quality attribute scenarios. Such requirements elicitation methods tend to proceed in a lock-stepped, sequential fashion limited by the number of attendees that can be in one location, the range of perspectives that can be brought to bear and thus the comprehensiveness of the requirements analysis is limited. These sacrifices in scope and attention can result in an incomplete identification of key system requirements and priorities. For example, incompleteness can manifest itself as a lack of attention to critical security requirements because software developers easily overlook user motivations and limitations; trends in markets; or the emergence of new technologies that may negatively impact the system and its users and other stakeholders. There is also the concept of externalities, which occurs when stakeholders take a local action driven by a local perception of utility (value proposition) and fail to consider the extra expense that they cause other stakeholders who are not involved in the dialogue. It is important to step outside of that mind trap and consider the broader issues: Who is gaining? Who is losing? What is the real cost to society as a whole?  Failing to consider these broader issues can result in a compromised system where privacy can be lost and data breaches can occur, as seen from some of the deep dives undertaken as part of another recent research project. Finally, teams often suffer from group think, in which members fail to explore alternative or dissenting viewpoints to avoid creating conflict within the group. Without adopting a data-driven approach to requirements elicitation, requirements analysts can suffer from group think regarding what they perceive to be the primary needs of users and other stakeholders.  Current Status and Looking Ahead In recent months, we have completed the definition for our method, which we call "KJ+" and developed training and initial versions of the tools that support use of the method. KJ+ tackles each of the challenges noted: it can be applied in a distributed fashion and over a longer period of time, and thus is more inclusive than co-located, timeboxed approaches would be. Our method also proves to be more scalable than current approaches for the same reason. KJ+ inherits KJ’s focus on the unstated and unobvious, and because it is data-driven, it can mitigate the effects of group think.   In coming months, we plan to further pilot the method and compare its performance to results obtained using more traditional requirements elicitation approaches. Longer-term, we hope to have the opportunity to further research how we might augment or streamline the first few steps of KJ+ by automating analyses of user and developer discussion forums to identify more subtle and longer-term problems being encountered and thereby identify opportunities to improve the health of the STE. Such an approach might also support a more thorough discussion of externalities associated with particular solutions before they get broadly deployed. We welcome your feedback on our research. Please leave comments in the feedback section below.  Additional Resources To read the paper The KJ Method: A Technique for Analyzing Data Derived from Japanese Ethnology by Raymond Scupin, please visithttp://sfaa.metapress.com/content/x335923511444655/. To read the paper A Persona-Based Approach to Exploring Architecturally Significant Requirements by Jane Cleland-Huang, Adam Czauderna, and Ed Keenan please visit http://link.springer.com/chapter/10.1007%2F978-3-642-37422-7_2#page-1.  To learn more about KJ+, obtain a copy of the full-day tutorial presentation, Eliciting Unstated Requirements, which was presented in August 2014 at the IEEE International Requirements Engineering Conference, by visitinghttp://resources.sei.cmu.edu/library/asset-view.cfm?assetid=309174. 

Alison Price - Business Analyst During my time at DWP - a tad shy of three decades - I’ve seen a fair amount of change. In that time I’ve seen huge transformations in the way DWP customers apply for benefits and the impact of how their applications are processed. In the main though, applications have been clerical, on paper forms. When you think about it, that process is rather archaic, especially when we all do so many things online these days. So, with that in mind, on my last few projects, I’ve been learning how to deliver digital services. In my own DWP journey I’ve been involved in lots of projects, some better than others. So when I joined the Carer’s Allowance digital team in Preston earlier this year as a Business Analyst I welcomed a new challenge but; I wasn’t prepared for the way the work the team worked or how fast things moved. Learning, learning, learning The first couple of weeks were a bit like a doing a tricky jigsaw; putting the pieces together by shadowing the team to understand how they’d got to a Beta service, and how they managed and delivered the workload. It was during this time that I got a really good view of the work that was being done at Carer’s Allowance and the real difference that had been made in such a short time. I’ve had to learn very quickly. The digital service standard requires skilled multi-disciplinary team- so no pressure there then. I’m still learning and thoroughly enjoying it. The two-week sprint cycle is impressive and I work closely with the Valtech Business Analyst; we’re a proper Tweedledee and Tweedledum. Joking apart working with Steve Moody has helped me understand how my role helps every sprint happen. If I’m not writing user stories and acceptance criteria, I‘m maintaining the Kanban board, running workshops with the business, reviewing work or attending user-research days, preparing show and tell sessions… I could go on but you get the idea. My days are full on. Meeting user needs However, this isn’t about me. It’s about the service we give to our customers and how we improve that experience. Meeting user needs is not without its challenges. Take the subject of accessibility. It’s been a priority for the team and part of every sprint from the start. At first, this was all a bit perplexing. I had difficult conversations and lots of meetings to understand the Carer’s Allowance digital service could meet the DWP’s own accessibility standards. Initially it didn’t, so we compiled a detailed accessibility evaluation which we discussed with the Senior Responsible Officer who then approved the service. Not only does the way work we did ensure that we meet the digital service standard, it also helps us make a service that works on any device, any browser and with all the popular assistive technologies. We’re meeting user needs. It’s all about delivery All of these experiences have contributed to the sprints that followed. Right now, we’re replacing in-office printing with a service that allows more staff to access submitted claims. Because we’re delivering new releases at least every two weeks, we can quickly address any problems that come up. Needless to say the work the team are doing at Carer’s Allowance is, as I said at the start of my blog, amazing. I’ve been around for years and am used to change; but seeing it happen so rapidly and effectively is something else.

By Will Hayes  Senior Member of the Technical Staff Software Solutions Division  More and more, suppliers of software-reliant Department of Defense (DoD) systems are moving away from traditional waterfall development practices in favor of agile methods. As described in previous posts on this blog, agile methods are effective for shortening delivery cycles and managing costs. If the benefits of agile are to be realized effectively for the DoD, however, personnel responsible for overseeing software acquisitions must be fluent in metrics used to monitor these programs. This blog post highlights the results of an effort by researchers at the Carnegie Mellon University Software Engineering Institute to create a reference for personnel who oversee software development acquisition for major systems built by developers applying agile methods. This post also presents seven categories for tracking agile metrics.  An Empirical Approach to Software  Increasingly, the DoD and federal agencies procure software-intensive systems instead of building them with internal resources. However, acquisition programs frequently have difficulty meeting aggressive cost, schedule, and technical objectives. The research reported in this blog is part of our ongoing work to help acquisition professionals in the DoD adopt and use agile software development methods more effectively to overcome these challenges. Our latest research focuses on progress measurement and contractors, which extends our previous research examining selected DoD management and acquisition concerns, including information assurance.  Many program offices in government and military organizations that we support have found that their providers or contractors are adopting agile methods. These program offices have found new ways to deliver software products more rapidly and in smaller increments than is customary in these environments. Program offices struggle with these techniques, however, because they lack experience with the metrics required to gain insight on progress. There is an elaborate infrastructure and a fairly well understood set of definitions for measures traditionally used in that space.  When we examined what is written, trained, and discussed about the term agile, the focus tends to be on a team of seven plus or minus two individuals working together in a self-directed setting. These teams have a keen focus on value to the user and employ an empirical approach, but most of the discussion centers on the small team.  It is important to note that in Scrum, decisions are made based on observation and experimentation rather than on detailed upfront planning. Empirical process control relies on three main ideas: transparency, inspection, and adaptation.  Use of agile methods in the context of major DoD programs is not unprecedented. Until recently, however, publications and training courses have focused too narrowly on the development team. Our findings show that organizations that apply agile methods are doing a good job filling in those abstractions between the small team-focused measurement and what is needed at an enterprise or program level. So, one of the challenges to overcome then, is meeting the needs of large-scale-program-management without violating the environment necessary for a self-directed team to succeed using agile methods. In many medium-to-large sized organizations, measurement is often conducted at the request of another individual and typically an obligation imposed on one party to benefit another party. Far too often, people asked by team leaders and project managers to provide metrics get defensive. This dynamic may limit the value of agile methods, which are intended to serve as the basis for an empirical approach to software.  This empirical approach involves enacting Edward Deming’s plan-do-check-act-cycle, but at a much more immediate and individually focused level. This approach to software involves more frequent conversation among developers. Another hallmark of the approach is that those conversations are very focused on the product itself.  When viewed in the context of the Agile Manifesto and its 12 principles, the best way to demonstrate progress is to demonstrate capability: an actual working product in lieu of an abstraction on paper. Obviously, the product could not have been built without the abstractions, but in the context of agile methods, the focus is on demonstrable results and data collected by the team for its own use.  Agile Metrics For agile software development, one of the most important metrics is delivered business value. These progress measures, while observation-based, do not violate the team spirit. Our primary goal with this work was to help program managers measure progress more effectively. At the same time, we want teams to work in their own environment and use metrics specific to the team, while differentiating from metrics that are used at the program level. The technical report that we published on this topic—Agile Metrics: Progress Monitoring of Agile Contractors, which was co-authored by myself, Suzanne Miller, Mary Ann Lapham, Eileen Wrubel, and Timothy A. Chick—details three key views of agile team metrics that are typical of most implementations of agile methods:    Velocity. Simply stated, velocity is the volume of work accomplished in a specified period of time, by a given team. Typically, this is measured as story points accomplished per sprint. This measure is sometimes called "yesterday’s weather" by agile practitioners, as if to indicate its sensitivity to local conditions as well as seasonal trends. Indeed most experts explain that velocity is "team-unique" and thinking of this measure as a parameter in an estimating model is a mistake. The team must establish its own velocity for the work at hand.  Sprint Burn-Down Chart. As detailed in our technical note, this graphical technique provides a means for displaying progress for the development team during a sprint. As items in the backlog of work are completed, the chart displays the rate and amount of progress. This chart is typically provided for viewing on a team’s common wall, or electronic dashboard. Release Burn-Up Chart. A complementary graphical technique for the sprint burn-down, the release burn-up chart is also commonly used. Many cling to the convention that sprints burn down and releases burn up—though there is no mathematical principle that governs this choice. With each completed sprint, the delivered functionality grows, and the release burn-up chart depicts this progress in an intuitively logical fashion. This concept makes use of workflow management tools and other extensions of the concept. Our research involved interviewing professionals who manage agile contracts who gave us insight from professionals in the field who have successfully worked with agile suppliers in DoD acquisitions.  Based on our interviews with personnel who manage agile contracts, our technical note identified seven successful ways to monitor progress that help programs account for the regulatory requirements that are common in the DoD:  Software Size is typically represented in story points when agile methods are used. This approach is supported by the decomposition of functionality from a user’s perspective—into user stories. Tracing these user stories to system capabilities and functions, a hierarchy within the work can be meaningfully communicated and progress monitoring based on delivered functionality will focus on utility and function—rather than proxies like lines of code or function points. Effort and Staffing must be tracked because they tend to be the primary cost drivers in knowledge-intensive work. Use of agile methods will not change this fundamental fact, nor will it be necessary to make major changes to the mechanisms used to monitor progress. What does change, however, is the expected pattern of staff utilization. With the steady cadence of an integrated development team, the ebb and flow of labor in specialized staff categories is less prevalent when using agile methods. In general, agile teams are expected to have the full complement of needed skills within the development team—though some specialized skills may be included as part time members on the team. Rules of thumb applied in monitoring this element of performance on a contract must be revised. The expectation of a slow ramp-up in staffing during the early phases of a development effort may be problematic, and plans for declining use of development staff during the last half of the program (when testing activities traditionally take over) must be recalibrated. Organizations may establish test teams to perform system testing or regression testing outside the context of the development team. Schedule is traditionally viewed as a consequence of the pace of work performed. In agile development, the intent is to fix this variable, and work to maximize performance of the development team within well-defined time boxes. This places important requirements on stakeholders who must communicate the requirements and participate in prioritization of the work to be performed. Quality and Customer Satisfaction is an area where agile methods provide greater opportunity for insight than traditional development approaches tend to allow. The focus on frequent delivery of working software engages the customer in looking at the product itself, rather than the intermediate work products like requirements specifications and design documents. A strong focus on verification criteria (frequently called "definition of done") sharpens the understanding of needed functionality, and attributes of the product that are important to the customer. Cost and Funding structures can be tailored to leverage the iterative nature of agile methods. Using optional contract funding lines or indefinite delivery indefinite quantity (IDIQ) contract structures can add flexibility in planning and managing the work of the development organization. A more detailed discussion of the considerations for contracting structures to handle this is the subject of an upcoming publication. Requirements are often expressed very differently in the context of agile development—in contrast to traditional large-scale waterfall development approaches. A detailed and complete requirements specification document (as defined in DoD parlance) is not typically viewed as a prerequisite to the start of development activities when agile methods are employed. However, the flexibility to clarify, elaborate and re-prioritize requirements, represented as user stories, may prove advantageous for many large programs. The cost of changing requirements is often seen in ripple effects across the series of intermediate work products that must be maintained in traditional approaches. The fast-paced incremental approach that typifies agile development can help reduce the level of rework. Delivery and Progress monitoring is the area where perhaps the greatest difference is seen in agile development, compared to traditional approaches. The frequent delivery of working (potentially shippable) software products renders a more direct view of progress than is typically apparent through examination of intermediate work products. Demonstrations of system capabilities allow early opportunities to refine the final product, and to assure that the development team is moving toward the desired technical performance—not just to ask whether they will complete on schedule and within budget.  Looking Ahead  We continue to learn new and inventive ways of demonstrating progress and diagnosing performance from agile implementers. The value of this approach is that it represents a narrative driven by real-world experience.  Through this research, we have also observed that agile teams don’t want to wait for data analysis. Future research needs to focus on earlier analysis of data streams. The graphical representations needed to analyze these data streams are not the traditional ones we have seen. The analysis techniques that need to be applied, as well as the available baselines, take on a different form in this context: more near-term, more immediate feedback and the intelligent use of historical baselines.  Additional Resources Acquisition researchers in the SEI’s Software Solutions Division have published several technical notes that address different topics of interest to acquisition professionals that are contemplating or are currently using agile methods as part of their acquisition:  Agile Metrics: Progress Monitoring of Agile Contractorshttp://resources.sei.cmu.edu/library/asset-view.cfm?assetid=77747 Considerations for Using Agile in DoD Acquisitionhttp://www.sei.cmu.edu/library/abstracts/reports/10tn002.cfm Agile Methods: Selected DoD Management and Acquisition Concernshttp://www.sei.cmu.edu/library/abstracts/reports/11tn002.cfm A Closer Look at 804: A Summary of Considerations for DoD Program Managershttp://www.sei.cmu.edu/library/abstracts/reports/11sr015.cfm DoD Information Assurance and Agile: Challenges and Recommendations Gathered Through Interviews with Agile Program Managers and DoD Accreditation Reviewershttp://www.sei.cmu.edu/library/abstracts/reports/12tn024.cfm Parallel Worlds: Agile and Waterfall Differences and Similaritieshttp://resources.sei.cmu.edu/library/asset-view.cfm?AssetID=62901  

  Why do so many people get so excited over an HR conference?    Because SHRM is known for producing some of the most informative and engaging conferences in the world, and the SHRM Annual Conference & Exposition, which starts at the end of June, is on every HR professional’s bucket list.  As thousands of excited attendees prepare to travel to Las Vegas for the conference, June 28-July 1, the Society is working hard to deliver another amazing experience.  Once again, SHRM will have a robust social media presence during the conference under the #SHRM15 hashtag and...