|
Mohan Kamath & Krithi Ramamritham Database Systems Lab Department of Computer Science University of Massachusetts Amherst MA 01003 |
Narain Gehani & Daniel Lieuwen Database Systems Research Department Lucent Technologies / Bell Labs 700 Mountain Avenue Murray Hill NJ 07974 |
The World Wide Web has been growing at a rapid rate to become a global information source. Many manufacturers, stores, banks, financial consultants, law firms and government offices publish information about their products and services - and even sell them - through the Web. Numerous studies indicate that the volume of electronic commerce on the Web will sharply increase. Thus, most information sources in a company will be Web accessible - internally for various groups within the company and externally for business partners and customers. To meet the requirements of emerging electronic commerce applications there is a need to integrate aspects from Workflow Management, World Wide Web, Transaction Management, and Warehousing technology. We will motivate this with some examples.
Consider the formation of a virtual enterprise which will make it possible to implement JIT (Just In Time) method of inventory control to save money on inventory, warehousing and handling costs. A manufacturing company can place orders with suppliers for the individual components needed to make the product immediately after it receives an order for the product. The suppliers use Web servers linked to their backend databases to process online orders. With such a system in place, the global internet can be effectively used to process orders round the clock and deliver the products in a timely manner while reducing the inventory cost. This example illustrates the benefits of automated business processes on the Web by allowing steps to automatically query/update information from Web servers located both within and outside the organization. Such processes cannot be modeled using traditional transactions [4]. They require the use of extended transaction models or transactional workflows [8]. These in turn consists of a number of individual internet transactions.
Next let us consider the operation of an electronic brokerage service. Such services are essentially data warehouses that use both the ``push'' and ``pull'' methods for obtaining information. The stored information (for example catalogs) is obtained in advance (``push'') while the rest (for example price and availability) is obtained dynamically (``pull'') as the need arises. The brokerage services allow customers to browse through statically stored / dynamically fetched information and then issue transactions. To process customer transactions, the brokerage service might have to contact a bank to get payment and then place orders with the individual supplier(s) for the required items. Here again there is a need for transactional workflows. Transactional workflows can also be used by the brokerages (warehouses) for obtaining static/dynamic information. Examples of electronic brokerage services include shopping services on the Web and stock brokerages for trading stocks from around the globe.
The above two motivating examples illustrate the utility of a system that can be used to develop global transactional workflows. Hence we have developed and implemented the WorldFlow system. This paper discusses the key issue that arises in implementing WorldFlow, namely dealing with the unreliability of the Web in the form of server/network failures and providing efficiency. Then we present the architecture of WorldFlow and discuss its implementation. The rest of this paper is organized as follows. A brief introduction to WorldFlow is provided in Section 2. Section 3 discusses problems related to implementing workflows on the Web and discusses some specific solutions. Section 4 presents the architecture for WorldFlow and discusses its implementation. Section 5 concludes the paper with a summary.
The basic principles underlying WorldFlow originate from workflow management. In this section we provide a brief overview of WorldFlow.
Conventional WFMSs consist mainly of a workflow engine, application agents and tools for modeling, administration and monitoring. The workflow engine and the tools communicate with a workflow database (WFDB) to store and update workflow relevant data. A workflow schema (workflow definition) is provided by a workflow designer with the aid of the modeling tool. The workflow schema can be used to instantiate several workflows. A workflow schema is essentially a directed graph with nodes representing the steps to be performed. The schema also specifies how data flows between the steps. Execution of a step usually corresponds to the invocation of a specified application program at a particular agent. At execution time, the agent is provided with the required inputs by the workflow engine and after the execution of the step, the output of the step is returned by the agent to the workflow engine. In case of failures, the WFDB facilitates forward recovery i.e., each workflow is continued from the state it was in before the failure occurred rather than starting from the first step.
The Web is essentially a client-server based distributed system where browsers acting as clients communicate with Web servers using the HTTP protocol [11]. As shown in Figure 1, in WorldFlow Web servers can play the role of agents and the CGI-scripts (that often execute transactions against a backend database) can play the role of programs. Note that the WorldFlow engine is now a client of the Web servers. To facilitate inter-organizational workflows (where several steps are to be performed in a different organization rather than just accessing some information), a WorldFlow engine can transfer control of a workflow to another WorldFlow engine via a Web server. This paper focuses mainly on the interaction between the workflow engine and the Web servers, i.e., business-to-business transactions.
Since customers and businesses want prompt service for their orders, it is important to ensure that workflows do not stall and make timely progress during their execution. It should be possible to track the status of a specific workflow that is in progress. Although our design incorporates a centralized scheduler (as it simplifies tracking and ensuring progress of workflows), multiple such schedulers can be used in parallel for load balancing and scalability within an organization. Transfer of control between workflow engines also ensures scalability of complicated inter-organizational workflows.
Our work assumes that a Web server assigned for each step will provide proper interfaces for performing the various functions (e.g., placing a request, enquiring about the status of a request, canceling a request, modifying a request). The Web interfaces access data from backend DBMSs using a CGI script or some other access scheme (e.g., the direct interface between HTML and SQL described in [7]). Each step in a workflow accesses data from at most one Web server.
In this section, we briefly enumerate the problems that arise when we implement WorldFlow on the Web. We also provide outlines of specific solutions.
To avoid losing potential business from customers, it is necessary for the system to be able to handle/manage all orders. Large organizations using workflows to provide and manage popular services will experience enormous loads - instantiating a few thousands workflows per day would not be uncommon. The load on an organization's internet gateway will also be high and hence it is essential to improve efficiency by reducing the number of connections and the number of messages exchanged.
Since the WorldFlow engine is a client of the Web servers, the failure of the network, large delays, and Web server failure all mean the same since all problems cause connection timeouts. All that matters is whether the failure occurred before a request was processed or after a request was processed but before the response arrived. This is crucial for update steps (e.g., steps ordering items from suppliers) of the workflow since each must be performed exactly once.
To ensure that each update operation is executed only once, a request has to be performed in an idempotent fashion. Web servers are stateless, and hence they cannot support idempotent operations, i.e.,they do not have the capability to discriminate between a new request and a duplicate request. Hence, it is necessary to have some way of testing (determining) whether the program was successfully executed or not at the remote server. If the state of the operation cannot be tested at the remote server, then it is not possible to achieve the ``exactly once'' semantics. Only after determining where the request failed, can appropriate action be taken.
To handle requests idempotently on the Web, we have developed the EONS (Exactly ONce Semantics) protocol. By assigning a unique ID to each request (analogous to a purchase order number) before the request is made it can be ensured that each request is processed exactly once in spite of failures. The EONS protocol is quite similar to standard networking retransmission schemes like Stop-and-Wait and Go-Back-n [3] where the networking software uses counters to ensure that each packet is released to higher levels of software exactly once. The main difference is that the EONS protocol requires that this exactly-once guarantee hold even if the connection between the two parties is broken for an arbitrarily long period of time. Consequently, it requires persistent storage of request information (like the unique ID), while the networking protocols do not.
The solution we propose for improving efficiency makes use of dynamic batching and utilizes enhancements proposed to the HTTP protocol [11] that support persistent connections , i.e., several URLs at a Web server can be accessed using a single connection. The workflow engine can batch steps from concurrent workflows that need to fetch or update data from the same Web server. Requests to Web servers are batched for a threshold batching period and are handled using a single persistent connection. Thus, significant savings can be achieved by avoiding several connection setup and tear down. An example is shown in Figure 3 where steps A, B, C and D are batched. The batching interval is a tradeoff and hence different types of batching schemes can be devised.
The batching schemes will benefit a lot from emerging technologies that enable the implementation of servlets [1]. Servlets can be customized to process more complex and even multiple queries/updates in a single request. The batching scheme can then group multiple queries/updates and ship them in a single request to the servlet which in turn can process the request and send back a single reply containing the results of all the queries/updates.
The architecture we have developed for WorldFlow is shown in Figure 4. Other than the standard WFMS components, it also contains a Web module which is responsible for handling steps that need to access Web servers. It uses the information provided by the workflow schema for extracting the required output from the HTML file returned by the Web server and passes the results to the workflow engine. The Web module is responsible for enforcing the EONS protocol, and the workflow engine is responsible for batching. The Web module is also responsible for managing the persistent connections to remote Web servers while performing batching of requests.
We have implemeneted a prototype of WorldFlow by extending the prototype workflow management system we have developed to study failure handling schemes for workflows [6]. This prototype supports transactional workflows and has been completely implemented in Java and is portable across heterogeneous architectures. The workflow modeling language and the compiler were enhanced to handle URL specifications and the Web module was added to the run-time support. The EONS protocol is being implemented. Batching of requests will be implemented when the enabling technologies are available, i.e., web servers using HTTP 1.1 and servlet development tools.
Using WorldFlow we have developed a sample brokerage service application called StockAdvisor. It is a tool for making smarter stock investment decisions. By selecting a company of interest, a user can get (i) the company's corporate profile information, (ii) stock performance of that company and other major players in the same sector and (iii) stock performance across all different sectors. This is achieved by accessing some local databases as well as other Web servers that provide corporate profile information and near real-time stock quotes. The EONS protocol will be used to support simulated trading.
To meet the needs of emerging electronic commerce applications there is a need for systems that can support global transactional workflows. To address this need, we have developed and implemented WorldFlow, a workflow management system that can directly access local database and online information available on the Web. In this paper we addressed issues relevant to implementing transactional workflows on the Web. Specifically we discussed problems arising from network/server failures and also focused on efficiency.
Electronic brokerages where numerous value-added services are offered by a single agency with the ability to track and account for each order placed by a customer are becoming popular [5]. Much effort is being dedicated to the development of Electronic Data Interchange (EDI) standards [5], Business APIs (e.g., SAP R/3 [2]) and Web servers to facilitate electronic commerce on the internet. These trends indicate the importance of our work in developing WorldFlow as a tool to build global cooperative information systems for conducting business over the web. The Meteor2 project [10] has implemented workflows using the Web infrastructure, but they address a different set of issues. Their focus is more on human assisted business-to-business transactions as opposed to automated ones. While we provide a general infrastructure to develop workflows that access Web information sources, they have developed a very specific solution by embedding workflow related information in CGI scripts and executing them in a cascaded fashion. This can make tracking and failure handling difficult.
Worlkflows can be used as a tool in data warehousing environments for data collection from various sites [9]. Companies can have some of their data warehouse information accessible to business partners via web servers. Thus WorldFlow can be used as a tool for building global data warehouses by connecting the Web information sources in a desired manner.