IBM Embedded Business AI Framework

Learn

Integrate

Debugging EBA

EBA was created with transparency in mind – we did not want to provide users with another AI black box. As such, the assistant provides users a variety of open debug capabilities to inspect every relevant aspect of its own understanding. You will find these introspection capabilities useful in different steps of the assistant’s reasoning pipeline.

Pipeline overview

Before diving into an exposition of the different debug capabilities available, you will want to ensure that you understand how EBA works. That is, you should have a good grasp of our collaborative reasoning pipeline. We provide an brief enumeration below.

With this pipeline in view, lets walk through the some of the common bugs we have seen associated with each step.

Ontology

An ontology in EBA represents a set of concepts and a set of relationships between them. A developer should encode his business domain as an ontology. The main features an ontology should capture are showability, singular-plural references, attribute definition, spelling, and any other custom predicates aligned to a particular use case. In the course of development, it is possible to create an ontology which is incomplete or not well defined.

Common Bugs:

Annotation

After asking the assistant a question, hover over your question to see an information icon appear in the top right portion of the message. Clicking this icon will reveal a tokenized representation of your original question. This tokenized representation will contain a part of speech (PoS) breakdown of the questions along with the concept annotations selected by the assistant.

syntatic-parsing.png

In the tree above, you find a topological breakdown of a user’s question, including the lemmas, part of speech, and dependency relationship of each token. It is important to consider that this tree serves as a starting point for the rest of the reasoning process and that EBA is sensitive to the features of this tree. A side effect of this is that EBA does not recognize concepts on a keyword basis, but rather it considers the context in which the word is used. Consider the following distinct usages of the word ‘weather’ in the sentences ‘how is the weather?’ and ‘can we weather the storm’. The keyword ‘weather’ is used differently in each sentence, and EBA is intelligent enough to distinguish each instance. Annotations, which developers supply via NL patterns, are introduced into the syntax tree only in cases where their respective topologies match.

Common Bugs:

Reasoning

Based on the existing conceptual knowledge derived from the syntax and annotation process, EBA will perform reasoning to decide which set of actions can be taken to resolve a particular question. EBA will iteratively evaluate what actions can be applied in a sequence to resolve the user’s question, by outputting the variant with the best concept coverage, i.e. the variant which utilizes the maximum understanding.

Common Bugs:

Execution

Once EBA reasoning core determines the optimal understanding variant, only that variants will be executed. This means that all code, including any side effects, will be executed. Any time runtime errors will be detected and logs will be shown within the ‘Debug’ tab inside the development lab.

Built-in Debug Tooling

Information space

After the assistant responds to your question, you will be able to view an outline of its reasoning in the information space. Select the graph icon in the top right corner of the content tab to view the information space. The information space is represented as a set of nodes and links which contain drill-down information. Each question begins with a question node and is linked to other nodes which represent your original question. These nodes are subsequently linked to concept nodes (solid-filled colors) and data nodes (white-filled colors). Concept nodes reveal the concepts associated with your tokens while data nodes contain the raw data the assistant associated with their respective concept. Data is gathered after executing actions. From this view, you can inspect the annotations, links, actions, and data selected and produced by the assistant for each question.

information-space.png

Common Bugs: If you notice an unsatisfying result from the assistant, you should first ensure that data nodes were actually generated for your domain concepts. Furthermore, you should ensure that the data returned from your assistant is correct. If your question does not contain data nodes, it is likely due to an invalid configuration within the assistant, e.g. undeclared patterns, invalid action signatures, invalid ontology, etc. If you have data nodes, but the data is incorrect, the problem lies within the action that produced this data. Consult the Debug Tab within our lab to inspect the action itself. If you have both data nodes and the correct data, then the issue likely lies in showability. Ensure that your domain entity is showable by subclassing it from the :Showable concept within the ontology tab or else ensure that your frontend visualization assets correctly render your data.

Auxiliary nodes

The information space also contains a set of auxiliary nodes, viz. token, meta, and variants. Token provides detail into each token of your sentence, including a full CoNNL breakdown as well as synonyms that the assistant can substitute in place of the current token. Meta provides meta data of the entire question and answer exchange. Variants will show the understanding variants that the assistant reasoned about along with their respective scores.

Reasoning in debug mode

To bring alternative understandings into the information space, simply asking your question and append ||debug to the end of it, e.g. "show me all invoices||debug".

alternative-intrepretations.png

The information space will also now display all alternative variants. As a fair warning, the assistant is able to reason about many thousands of alternatives at a time, so this view can become too large for more complex questions. In order to view one particular path in isolation, you should double click the message node associated with that variant. Double clicking will highlight this specific variant path and hide the remaining paths for easier introspection.

alternative-path.png

Debug tab

Actions allow the assistant to perform api requests and other operations to produce real data. The Debug tab is created to provide insight into the FaaS code invocations, timings, and logs. Consequently, this tab is the best place to debug any runtime errors hindering the assistant.

Common Bugs: If you see a 401 error after asking a question, it is likely due to an invalid api credential. Supply valid credentials using the Secrets or Settings tab. If you see a 500 error after asking a question, the problem lies in the api you are connecting to. If you see that null data is produced within the information space, there may likely be a runtime error within your action body itself.

Browser console

The development lab allows developers to apply custom frontend styles when rendering their data. These styles run within the browser and can viewed within the inspector itself.

edit this article