Technology Innovation for Accelerating Digital Transformation
While open source software has established itself as an industry standard in a variety of fields, notably cloud computing, obstacles in its way of use by companies include compliance with security policies and operation management. This article proposes a method for implementing cloud services using open source software by means of technologies such as software-defined networking and operation automation based on ChatOps to overcome these challenges. The effectiveness of the proposed method was confirmed by prototyping and using it in-house, including the generation of approximately 18,000 virtual machines over a period of one-and-a-half years.
Cloud Research Department, Center for Technology Innovation – Digital Technology, Research & Development Group, Hitachi, Ltd. Current work and research: Research and development of OSS-based cloud management technologies.
Cloud Research Department, Center for Technology Innovation – Digital Technology, Research & Development Group, Hitachi, Ltd. Current work and research: Research and development of OSS-based cloud management technologies. Society memberships: The Institute of Electronics, Information and Communication Engineers (IEICE).
Cloud Research Department, Center for Technology Innovation – Digital Technology, Research & Development Group, Hitachi, Ltd. Current work and research: Research and development of OSS-based cloud management technologies. Society memberships: IEICE.
Cloud Research Department (at the time of writing), Center for Technology Innovation – Information and Telecommunications, Hitachi, Ltd. Current work and research: Research and development of OSS-based cloud management technologies.
Open source software (OSS) has become an industry standard in a variety of fields, including cloud computing, data analysis, and the Internet of Things (IoT). Open innovation through the use of these OSS applications and collaborative creation with customers is essential to achieving digital transformation. While OSS is being developed by a community of developers and researchers around the world and is evolving rapidly to become technologically advanced, support is not always available to the same degree compared with commercial products, and the documentation is sometimes inadequate. This means that the utilization of OSS requires the acquisition of know-how and the development of expertise in staff through routine use of the software and by participating in and contributing to the community.
There are also significant obstacles to the corporate use of OSS. As a prerequisite of modern OSS is interoperation with Internet services, it cannot always comply with corporate security policies. There are also operation management requirements that accompany the utilization of the diverse range of rapidly evolving OSS. Practices adopted to overcome these difficulties include establishing system environments that are isolated from the rest of the company and conducting operation management manually, but these are cumbersome and have limited the adoption of OSS.
This article proposes a method for implementing cloud services utilizing OSS by using software-defined networking (SDN)(1) technology and operation automation technologies based on ChatOps(2) to overcome these challenges. It describes the results of prototyping and using the method in-house, and also presents future prospects for the method.
Interoperation with the Internet is an essential part of modern OSS, because it often requires connecting to Internet-based software repositories or web services.
However, to prevent security incidents such as the loss of confidential information or malware infection, companies and other organizations put strict security policies in place to protect their networks. In the case of interoperation with Internet services in particular, it has become common practice to control access and to restrict browsing through mechanisms such as firewalls, web filtering, and authentication proxies. Not all OSS was developed for use under such strict security policies, with the inability to operate in a proxy environment being one example of the issues that arise.
Companies and other organizations need to use a diverse range of OSS. Examples include OpenStack(3)*1, for implementing infrastructure as a service (IaaS); Cloud Foundry(4)*2, for implementing platform as a service (PaaS); GitLab(5)*3, for managing source code; Redmine(6), for project management; Jenkins(7)*4, for ongoing integration; and Rocket.Chat(8) for communication. With new software appearing on a daily basis, OSS is evolving rapidly with high turnover.
On the other hand, companies and other organizations have built their business systems and established operational practices around the use of software obtained commercially or developed in-house for which long-term support is available, leaving them ill-equipped to manage this diverse range of high-turnover OSS.
Moreover, because OSS includes both multi-user software and multi-tenant software, which uses containers and server virtualization, it is difficult to manage who is using what and to what extent. Leaving this unmanaged would result in increased costs for computer resources that are never used, and greater security risks such as the loss of information from these unmanaged computer resources.
Measures adopted to overcome the issues described above include establishing special system environments that are isolated from the rest of the company to permit interoperation with Internet services, using email-based application for access as an exception, or maintaining a ledger or other such method. Unfortunately, the cumbersomeness of such isolated system environments make them harmful to staff productivity and motivation. Issues with these isolated environments include being inaccessible from the company network routinely used by staff, the inability to exchange data, and administrative delays in granting approval for use. As a result, the adoption of OSS has been impeded.
Also problematic is manual records management of a diverse range of OSS, including the repeated creation and deletion of virtual machines and containers, and the many different communication protocols they use. This leads to operation becoming a rote process that is prone to operational mistakes and management lapses.
By using SDN technology and operation automation based on ChatOps to overcome the issues associated with past measures, Hitachi has implemented an environment for utilizing OSS as a private cloud that can interoperate with other internal and external networks and be made available on short notice (see Figure 1).
Figure 1—OSS Cloud
A private cloud capable of interoperating with internal and external networks was established as an environment for using OSS and achieving open innovation through collaborative creation with customers.
Figure 2—Technology for Interoperating with Internal and External Networks Using SDN Technology
A private cloud capable of interoperating with internal and external networks was established as an environment for utilizing OSS and for monitoring and controlling communications. It was implemented at the level of individual users through interoperation with the in-house authentication platform.
To overcome the challenges of using an isolated network, a private cloud for running OSS is established in a demilitarized zone (DMZ) located between the in-house network and the Internet, and network virtualization technologies such as virtual extensible local-area networks (VXLANs)(9) are used to provide logical separation between users, projects, and other tenants. Furthermore, SDN control of the firewall used for the DMZ provides secure on-demand interoperation with other internal and external networks (see Figure 2).
Locating the private cloud in a DMZ makes it available from the in-house network and enables interoperation between it and Internet services while at the same time making it possible to prevent direct interoperation between Internet services and the in-house network.
Moreover, the private cloud can also utilize existing in-house operation management systems such as the company's authentication platform. This means that, when members of staff use their existing identifications (IDs) to start using OSS, their ID, organizational information, and attribute information can be associated with the resources they use in the private cloud and with their network communications. These associations can then be utilized to track communication usage or detect anomalies at the level of individual users. The results of this can in turn be used as the basis for access control and monitoring of individual users through SDN control of the firewalls between the in-house network and the private cloud, and between the private cloud and the Internet.
Analyzing the payload of data packets to detect anomalous communications has become more difficult due to the growing use in recent years of encrypted communications such as hypertext transfer protocol secure (HTTPS)(10), (11), as well as its commonplace use for bidirectional communication over web protocols. Instead, anomalous communications are detected regardless of whether they are encrypted by using communication statistics as a basis for analyzing communication behavior.
Figure 3—Operation Automation Technology Based on ChatOps
This technology automates service usage using chatbots, and eliminates unused resources.
To overcome the issues associated with manual operation, ChatOps is used to operate a wide variety of OSS and to automate its operation. Specifically, this provides self-service and on-demand access to OSS by having users interact with chatbot software to invoke or terminate the OSS, and having the chatbot call the OSS's application programming interface (API) to generate resources such as accounts, instances, containers, or projects based on what the user requests (see Figure 3).
Using the ChatOps approach not only provides staff with quick access to different types of OSS, but also encourages users to collaborate and to resolve issues among themselves because it enables seamless chat-based sharing of know-how on how to use the OSS as well as other relevant information.
Furthermore, ChatOps records and manages usage at the level of individual users by associating the user ID, organizational information, and attribute information of staff who have logged into the chat software with the various OSS, accounts, and resources they use in the private cloud. It also detects and releases unused computing resources automatically based on this recorded usage.
When attempting to automatically detect unused resources, the fact that users use different OSS, or different versions of the same OSS, and have different system configurations and workloads makes it difficult to determine, based on threshold values, whether particular resources are being used or not. Accordingly, machine learning is used to make these decisions.
To assess how well the solution described above will work, Hitachi's R&D and IT divisions worked together to implement an in-house private cloud for utilizing OSS (hereinafter, “OSS cloud”) and made it available as a service within the company in September 2015. The OSS cloud was itself implemented using OSS to help accumulate know-how in the use of the software. Specifically, OpenStack was provided as an IaaS environment; Cloud Foundry as a PaaS environment; and GitLab, Redmine, Jenkins, and Rocket.Chat as a DevOps environment; all of which were made available via ChatOps. Hubot(12)*5, an OSS framework for chatbots, was used to implement ChatOps.
Prototyping also included techniques for using SDN technology for interoperation with internal and external networks, and operation automation technologies based on ChatOps, and applied to the OSS cloud.
Figure 4 shows the number of virtual machines created on the OSS cloud between July 2015 and February 2017. Usage increased rapidly since it was first made available as a private cloud for use within the company in September 2015, and a cumulative total of approximately 18,000 virtual machines have been created as of February 2017. The encouragement that the proposed method provides for people to utilize OSS was demonstrated by the fact that, apart from virtual machines, it was used by several hundred staff members at Hitachi and saw the creation of several hundred projects, of which 37% were collaborative projects spanning multiple departments.
Based on the results of prototyping the proposed method and applying it in-house, the plan for the future is to commercialize the developed technology, beginning with expanding its use to Lumada.
Figure 4—Trend in Number of Virtual Machines Created on the OSS Cloud
The graph shows the cumulative total of virtual machines created on the OSS cloud between July 2015 and February 2017.
Hitachi has proposed a method for implementing cloud services that encourage the use of OSS by using SDN technology and operation automation technology using ChatOps to resolve the issues associated with the corporate use of OSS, which include incompatibility with existing security policies and the operation management of a diverse variety of OSS software. The effectiveness of the proposed method was demonstrated by prototyping it and applying it in-house.
In the future, Hitachi will utilize open innovation through collaborative creation with customers to achieve their digital transformation.
