Knowledge modelling for a generic refinement framework

Abstract

Refinement tools assist with debugging the knowledge-based system (KBS), thus easing the well-known knowledge acquisition bottleneck, and the more recently recognised maintenance overhead. The existing refinement tools were developed for specific rule-based KBS environments, and have usually been applied to artificial or academic applications. Hence, there is a need for tools which are applicable to industrial applications. However, it would be wasteful to develop separate refinement tools for individual shells; instead, the KrustWorks project is developing reusable components applicable to a variety of KBS environments. This paper develops a knowledge representation that embodies a KBS's rulebase and its reasoning, and permits the implementation of core refinement procedures, which are generally applicable and can ignore KBS-specific details. Such a representation is an essential stage in the construction of a generic automated knowledge refinement framework, such as KrustWorks. Experience from applying this approach to Clips, PowerModel and Pfes KBSs indicates its feasibility for a wider variety of industrial KBSs.

Introduction

The well-known knowledge acquisition bottleneck encompasses both the original knowledge elicitation and the debugging of the knowledge while the knowledge-based system (KBS) develops. As KBSs become more routinely used in industry, their maintenance becomes a further knowledge management issue. The evolution of methodologies such as KADS [1] organises the knowledge development process, but there is a demand for knowledge refinement tools that assist with the acquisition, debugging and maintenance of the knowledge itself. A knowledge refinement tool assists a knowledge engineer by identifying places where the knowledge may need to be changed. For knowledge acquisition, it identifies potential gaps in the knowledge and incorporates missing knowledge into the KBS. For debugging, it identifies potential faults in the knowledge and suggests possible repairs. In contrast to debugging, knowledge maintenance refines the knowledge because the problem-solving environment has changed in some way; in this case, the knowledge must be updated to match the new environment.

Each of the knowledge refinement tools performs the same general steps. The tool is presented with a faulty KBS and some evidence of faulty behaviour; often this consists of examples that the KBS fails to solve correctly, together with the correct solutions. The refinement tool reacts to a piece of evidence by undertaking the following three tasks: blame allocation determines which rules or parts of rules might be responsible for the faulty behaviour; refinement generation suggests rule modifications that may correct the faulty behaviour; and refinement selection picks the best of the possible refinements. The goal of refinement is that the refined KBS correctly solves as many of the examples as possible, with the expectation that novel examples will also have an improved success rate.

Most knowledge refinement systems are designed to work with KBSs developed in a single language [2], [3], [4], or a particular shell [5]. However, it is wasteful to develop refinement tools for individual languages and shells. We prefer to investigate reusable refinement components that can be applied to a variety of KBS environments. This paper concentrates on the knowledge representation issues, and the rest of this section investigates the knowledge demands of the core refinement processes. Section 2 considers how these can be satisfied by generic structures that organise the key roles adopted by components of rules. In 3 Applying the knowledge skeleton, 4 Extensions to the hierarchy we describe our experience of applying these generic knowledge structures to various KBSs. Section 5 investigates the approaches used by other refinement tools. In Section 6, we draw some conclusions about the usefulness of these structures and their utility in our long term goal of providing a framework of refinement components in the KrustWorks project.

We base this paper on experience with Krust refinement system [6]. Fig. 1 shows a KrustTool¹ performing the operations highlighted above. The KBS's problem-solving for one training example is analysed, and blame is allocated to the knowledge that has taken part in the faulty solution, or which failed to contribute to the solution as intended. The experiment toolset generates repairs that correct this faulty behaviour. A rule is prevented from firing by making its conditions harder to satisfy, or by preventing rules that conclude the knowledge required by the original rule's conditions. Conversely, a rule is encouraged to fire by making its failed conditions easier to satisfy, or by encouraging rules that conclude the knowledge required by the original rule's failed conditions. Knowledge specific refinement operators implement these repairs on the rules. KrustTools are unusual in proposing many faults and in generating many repairs initially, and hence a KrustTool applies filters to remove unlikely refinements before any refined KBSs are implemented. It then evaluates the performance of these refined KBSs on the training example itself and on other examples that are available. A detailed description of the execution of an early Krust appears in Ref. [7].

The refinement process relies on determining what knowledge was applied to solve the given training example and what knowledge might have been applied instead (Blame Allocation), and how to alter the knowledge (Refinement Generation). A refinement tool, therefore, reasons about the actual and potential interactions within the KB, and in particular rule chaining behaviour. An important result of our work on knowledge refinement is the conclusion that the necessary information for the refinement of any KBS may be represented in two structures.

The Knowledge Skeleton is an internal representation of the rules in a KBS. The knowledge skeleton allows the refinement tool to determine what knowledge is applied and how rules can chain. The creation of a knowledge skeleton requires the existence of a common knowledge representation language, which can represent any feature in any shell to which KrustWorks is to be applied. The skeleton itself is built by a shell-specific translator. The remaining sections of this paper concentrate on the representation language for the knowledge skeleton and the use of the knowledge skeleton in KrustTools.

The Problem Graph represents the KBS's problem-solving for an incorrectly solved example. Details of the problem graph and its use by the KrustTools will appear in a later paper. Fig. 2 shows a problem graph when the KBS solution is ‘−’, but the correct answer is ‘+’. The initial facts are shown as circular leaf nodes, provable knowledge is also shown as circles, and square nodes represent knowledge that is currently not provable but would help to correct the error. Each rule is labelled by a diamond, linked to its conditions beneath it and its conclusion above. A circular arc indicates that a rule has two or more conditions forming a conjunction. For example, rule R7 has condition G∧H and conclusion B. Rule R8 has condition H and conclusion C. The positive part of the problem graph is shown as solid lines and represents the knowledge that was applied during the problem-solving. The negative part of the problem graph is represented with dotted lines and highlights those parts of the rules which may be changed to deduce the correct answer ‘+’. The negative part of the graph is constructed with reference to the desired goal and the knowledge skeleton, as it cannot be derived from the observed behaviour of the KBS. We represent the control strategy by organising the sub-graphs in a left-to-right order, where the leftmost rules are those chosen earliest for execution. The chain of reasoning that leads to the KBS's conclusion is, therefore, the leftmost sub-graph containing circled nodes only; nodes are circled because the conditions of these rules were satisfied, and they appear as the leftmost of the satisfied rules because they were selected in preference to other satisfied rules. This chain is highlighted using bolder lines in Fig. 2. Blame allocation identifies square nodes leading to ‘+’ that should be altered to allow them to fire (leftmost sub-graph), circular nodes leading to ‘−’ that should be prevented from firing (middle sub-graph), and circular nodes leading to ‘+’ that should be altered to make them more competitive in the control strategy (rightmost sub-graph).

It is important to note that the positive half of the problem graph is derived from the actual observed behaviour of the KB, and not from an internal simulation, hence, it is guaranteed correct. On the other hand, the negative half of the graph does require a simulation of potential KB behaviour, and hence may introduce inaccuracies. However, any consequent incorrect repairs will be detected during testing; and if the ideal repair is missed, then the best of the many other refinements that the KrustTool generates will be applied instead.

The core refinement procedures adopted by all KrustTools are shown in Fig. 1. Fig. 3 places the refinement procedures in context, and shows how a KrustTool interacts with a KBS to create the knowledge skeleton and the problem graph. These provide the information needed to carry out refinement. The KrustTool performs the following steps:

• The tool translates the KBS's rules into the knowledge skeleton, representing the static knowledge in the KBS.

• The tool is given a training example, for which the KBS gives an incorrect solution, together with information about how the KBS reaches its conclusions. This information may be provided either in the form of an execution trace, or via queries submitted to the KBS. The tool uses the information to build a problem graph: an internal structure representing the reasoning of the KBS for the particular training example.

• The refinement algorithm analyses the problem graph and the knowledge skeleton to determine where changes may be made to correct the errors made by the KBS. In general, the correct fix cannot be uniquely determined, hence the tool generates a number of alternative refinements, which are then filtered, implemented and tested. Testing consists of translating the modified knowledge skeletons back into the language of the KBS, and then executing them on the training example and others.

One distinguishing feature of KrustTools is that the refinement algorithm and the KBS run as separate processes. In contrast, Either [2] is written in Prolog and refines only Prolog KBSs, using a single process for both tool and KBS. The KrustTools approach is necessary for a generic refinement tool, as a separate refinement tool and KBS are necessary to allow the refinement of a KBS written in any language.