Category: whitepaper

The DataNeuron Pipeline

DataNeuron helps you accelerate and automate human-in-loop annotation for developing AI solutions. Powered by a data-centric platform, we automate data labeling, the creation of models, and end-to-end lifecycle management of ML.

Ingest

Upload Visualization

Users can upload the entire data available with them without performing any filteration to remove out of scope paragraphs.

The data can be uploaded in various file formats supported by the platform.

The platform has an in-built feature that can handle out-of-scope paragraphs and separate them from the classification data. This functionality is optional and can be toggled on or off anytime during the process.

Structure

Structure Visualization

The next step is the creation of the project structure.

Instead of a simple flat structure with just the classes defined, we provide the user with the option to create a multi-level (hierarchical) structure so that he can extract the data grouped into domains, subdomains, and indefinitely continue dividing into further subparts depending on his needs.

Any of the defined nodes can be marked as a class for the data to be classified into irrespective of the level on which it is in the hierarchy. This provides flexibility to create any level of ontology for classification.

Validate

Validate Visualization

User does not have to go through the entire dataset to sort out paragraphs that belong to a certain class and label them to provide training data for the model, which can be a tedious and difficult task.

We propose a validation based approach:

• The platform provides the users with suggestions of paragraphs that are most likely to belong to a certain category/class based on an efficient context-based filtering criterion.
• The user simply has to validate the suggestions, i.e, check whether or not the suggested class is correct.

This reduces the effort put in by the user in filtering out the paragraphs belonging to a category from the entire dataset by a large margin.

The strategic annotation technique allows the user to adopt a ‘one-vs-all’ strategy, which makes the task far easier than having to take into consideration all the defined classes, which can be a large number depending on the problem at hand, while tagging a paragraph.

Our intelligent filtering algorithm ensures “edge-case” paragraphs, i.e paragraphs that do not have obvious correlation with a class but still belong to that class, are not left out.

This step is broken down into 2 stages:

• The validation done by the user in the first stage is used for determining the annotation suggestions offered in the second stage.
• Each batch of annotation is used to improve the accuracy of the filtering algorithm for the next batch.

The platform also provide a summary screen after each batch of validation which provides the user with an idea as to many more paragraphs he might need to validate per class in order to achieve a higher accuracy.

It also helps determine when to stop the validation for a specific class and focus more on a class for which the platform projects low confidence.

Train

Train Visualization

User invests virtually no effort into the model training step and the model training can be initiated with the simple click of a button.

The complete training process is automatic which performs preprocessing, feature engineering, model selection, model training, optimization and k-fold validation.

After the final model is trained, the platform shows a detailed report of the trained model is presented to the user which includes the overall accuracy of the model as well as the accuracy achieved per class.

Iterate

Iterate Visualisation

Once the model has been trained, we provide the user with 2 options:

• Continue to the deployment stage if the trained model matches their expectations.
• If the model does not achieve the desired results, the user can choose to go back and provide more training paragraphs (by validating more paragraphs or uploading seed paragraphs) or alter the project structure to remove some classes and then retrain the model to achieve better results.

Deploy (“No-Code” Prediction Service)

Deploy Visualisation

Apart from providing the final annotations on the data uploaded by the user using the trained model, we also provide a prediction service which can be used to make a prediction on any new paragraphs in exchange for a very minimal fee.

This does not require any knowledge of coding and users can utilize this service for any input data from the platform.

This can also be integrated into other platforms by making use of the exposed prediction API or the deployed Python package.

No Requirement for a Data Science/Machine Learning Expert

The DataNeuron ALP is designed in such a way that no prerequisite knowledge of data science or machine learning is required to utilize the platform to its maximum potential.

For some very specific use cases, a Subject Matter Expert might be required but for the majority of use cases, an SME is not required in the DataNeuron Pipeline.

Experiment

We compare DataNeuron‘s Unsupervised learning algorithms to the most popular Language Models in benchmark experiments. This benchmarking is primarily intended to compare models capable of data annotation without any prior domain knowledge or pre-training (Zero-Shot).

Dataset

3 domain-specific datasets were used to compare the algorithms’ accuracy:

1. Ohsumed: is a medical abstracts from MeSH categories from 1991 (https://paperswithcode.com/dataset/ohsumed)
2. Contract Understanding Atticus Dataset (CUAD): is a legal contract dataset with extensive expert annotations (https://www.atticusprojectai.org/cuad)
3. Emotions: Dataset with classes such as “sadness, anger, love, surprise, fear, and joy.” (https://www.kaggle.com/datasets/parulpandey/emotion-dataset)

Open-source Large Language Models (LLMs)

We have selected three of the most popular LLMs that are capable of performing Zero-Shot learning:

1. Nb-bert-base-mnli: Trained on MNLI dataset (https://huggingface.co/NbAiLab/nb-bert-base-mnli)
2. Bart-large-mnli: Trained on the MultiNLI (MNLI) dataset (https://huggingface.co/facebook/bart-large-mnli)
3. Nli-roberta-base: Trained on the SNLI and MultiNLI datasets (https://huggingface.co/cross-encoder/nli-roberta-base)

GPT 3.5 Turbo

The most capable GPT-3.5 model, optimized for chat, at one-tenth the cost of the text-davinci-003. Supports up to 4,096 tokens with training data until September 2021.

We have picked this GPT version for the benchmarking since ChatGPT hosts the same model version, which is available to the general public.

Colab Link

1. API Setup

Using pip install OpenAI, install the OpenAI library in your Python environment. After installation, create an OpenAI API key. Then use Openai.api_key = ‘key-extracted-from-openai’ to authenticate the API key.

2. ChatCompletion

Use the Chat Completion Class to call the “create” function, which serves as a model for sending requests and receiving API responses.

chat_completion = openai.ChatCompletion.create(model="gpt-3.5-turbo", 
messages=[{"role": "user", "content": "Hello world"}])

Request Message

[{“role”: “user”, “content”: “Hello world”}]

Content: Prompt Example

Please label each of the following 100 paragraphs strictly into one of these 6 classes: 1. sadness 2. anger 3. love 4. surprise 5. fear 6. joy. Separate answers with , for each of the paragraphs [“paragraph1”,”paragraph2”,………,”paragraph100”]

4. Benchmarking

Classification Report was generated by comparing the ground truth to the GPT 3.5 Output after creating classes on paragraphs with OpenAI libraries.

DataNeuron Platform (Stage 1)

Stage 1 models predict paragraphs based on the user-defined Masterlist/ Taxonomy. is equivalent to providing prompts or scope of the classification to the Zero-Shot LLMs. Stage 1 consists of proprietary Unsupervised models for annotation and DSEAL algorithms for strategic data sampling.

Result

Conclusion

DataNeuron Stage 1 performed better than pre-trained LLMs BERT BASE and ROBERTA across all the 3 datasets. Further it has comparable accuracies to BART LARGE for OSHUMED and CUAD datasets while outperforming it significantly on EMOTIONS dataset. DataNeuron Stage 1 models also outperformed GPT 3.5 in the benchmarking on CUAD and EMOTIONS dataset.

It is critical to note DataNeuron Stage 1 model was not given any sample paragraphs for pre-training, implying that Stage 1 models can automatically annotate with high accuracy without any prior domain knowledge.

Since DataNeuron models are light-weight it scales much better for the large data annotation workflows when compared to LLMs. At the same time DataNeuron is able to achieve comparable/ better accuracies with the proprietary Unsupervised models and DSEAL algorithms when compared to pre-trained LLMs at lesser cost/ time.

The term Artificial General Intelligence (often abbreviated “AGI”) has no precise definition, but one of the most widely accepted ones is the capacity of an engineered system to display intelligence that is not tied to a highly specific set of tasks or generalize what it has learned, including generalization to contexts qualitatively very different from those it has seen before and take a broad view, interpret its tasks at hand in the context of the world at large and its relation thereto.

In essence, Artificial General Intelligence can be summarized as the ability of an intelligent agent to learn not just to do a highly specialized task but to use the skills it has learned to extract insight from data originating in multiple contexts or domains.

How does DataNeuron achieve Artificial General Intelligence?

The DataNeuron platform displays Artificial General Intelligence as it has the ability to perform well on:

• NLP tasks belonging to multiple domains.
• Text data originating from multiple contexts.

Masterlist: Machine Learning is not binary so we don’t rely on rules or predefined functions, we rely on the simpler structure which is the Masterlist where we allow classes to have overlap. Further, we support taxonomy or hierarchical ontologies on the Masterlist. The platform uses intelligent algorithms to assign paragraphs for each class making the data annotation process automated.

Advanced Masterlist: We are also launching Advanced Masterlist to support subjective labeling of datasets (where clear class distribution is missing).

Apart from the ability to perform auto-annotation on data, the platform also provides complete automation for model training including automatic data processing, feature engineering, model selection, hyperparameter optimization, and cross-validation of results.

The DataNeuron Platform automatically deploys the algorithm and provides APIs which can be integrated to build any application with real-time no-code prediction capabilities. It also provides a continuous feedback and retraining framework for updating the model for achieving the best performance. All these features make it one step closer to achieving Explainable AI.

The DataNeuron platform has produced exceptional results in extremely specialized domains like Document or Text classification in the Tax & Legal, Financial, and Life Sciences use cases, as well as general tasks like Document or Text Clustering in any given context. DataNeuron reduces the time and effort by ~95% required to label and create models, allowing users to extract up to ~99.98% insights. DataNeuron is an Advanced platform for complex data annotations, model training, prediction & lifecycle management. We have achieved a major breakthrough by fully automating data labeling with comparable accuracy to state-of-the-art solutions with just 2% of labeled data when compared to human-in-loop labeling on unseen data.

The impact created by DataNeuron’s General Intelligence

We observe that the DataNeuron platform can decrease the annotation time by up to ~98%. This vastly decreases the time and effort spent annotating huge amounts of data and allows teams to focus more on the task at hand by automating the process of data annotation and easing research.

Additionally, it can also help reduce the SME effort up to 96%, while incurring a fraction of the cost. Our platform also significantly reduces the overall cost of the project, by nearly eradicating the need for data labeling/annotation teams. In some cases, the need for an SME is also diminished as the process of annotation is much simpler and anyone with knowledge of the domain can be able to do it properly unless the project is too complex.

Testing On Various Datasets

How DataNeuron performs for various use cases

The difference in Paragraphs is Annotated. We observe it is possible to reduce annotation effort by up to 96%.

The difference in Time Required. We observe it is possible to reduce time required for annotation by up to 98%.

The above visualizations showcase the platform’s ability to perform extraordinarily in different domains. As opposed to the specialized systems that tend to perform well on only one type of task or domain, the DataNeuron platform breaks boundaries by performing exceptionally for a diversified set of domains.

What does it mean for the Future of AI?

As AI adoption has picked up among enterprises, the need for labelled and structured data has dramatically increased in order to remove the bottleneck in developing the AI solutions.

DataNeuron, powered by a data-centric platform provides a complete end-to-end platform from training to Ensemble Model APIs for faster deployment of AI.

Our research continues to be focused on the area of Artificial General Intelligence and further automation of Data Labeling / Validation and provide better explainability of AI.

DataNeuron’s Active Learning to NLP data labeling outperforms the competitors’ Sequential Learning approach by 20% with 50% less data Active learning resembles conventional supervised learning more closely. Given that models are trained to utilize both labeled and unlabeled data, it is a form of semi-supervised learning. The concept behind semi-supervised learning is that classifying a small sample of data may produce results that are as accurate as or even more accurate than training data that has been fully labeled. Finding that sample is the only difficult part. In active learning, data is incrementally and dynamically labeled during training so that the algorithm can determine which label would be most helpful for it to learn from.

Experiment

Let’s compare the two classifiers, Sequential and Active Learning, through this easy experiment.

We’ll be using the Contract Understanding Atticus Dataset (CUAD) for the purpose of this experiment. The dataset contains 4414 paragraphs. We have randomly selected 10 types of legal clause classes for the text classification task.

Download the dataset: https://www.atticusprojectai.org/cuad

Loading data & preprocessing

We’ll start by loading the data. The features and labels are then extracted from the data, and a train-test split is made, allowing the test split to be used to assess how well the model trained using the train split performed.

#train test split

df_train, df_test = train_test_split(df_total, test_size=0.2, random_state = 0)

# The resulting matrices will have the shape of (`nr of examples`, `nr of word n-grams`)

vectorizer = CountVectorizer(ngram_range=(1, 5))

X_train = vectorizer.fit_transform(train_df.para)

X_test = vectorizer.transform(df_test.para)

X_train.shape[Out] : (4082, 1802587)

X_test.shape[Out] : (1021, 1802587)

With Sequential Learning

The Sequential Learning classifier will be trained twice only, once with 100 paragraphs and the next time with 1000 paragraphs. For the next 1000 paragraphs, the total number of correct and wrong predictions is calculate

#Defining the classifier
classifier_seq = MultinomialNB()

#preprocessing of training set
x_100_train = vectorizer.transform(df_train[:100]['para'])
y_100_train = df_train[:100]['label']

#preprocessing of testing set
x_1000_test = vectorizer.transform(df_train[100:1100]['para'])
y_1000_test = df_train[100:1100]['label']

#training the model
classifier_seq.fit(X=x_100_train, y=y_100_train.to_list())

#calculating the accuracy on the test set
classifier_seq.score(X=x_1000_test, y=y_1000_test.to_list())
[Out] : 0.681

Code for calculating the accuracy of suggestion for every 10 paras sequentially

#the number of paragraphs in validation set for each iteration
n_instances = 10

#first 100 paragraphs
n_start = 100 
n_para = 100

#defining the classifier
classifier_seq = MultinomialNB()
#preprocessing of training set
x_100_train = vectorizer.transform(df_train[:100]['para'])
y_100_train = df_train[:100]['label']

#preprocessing of testing set
x_1000_test = vectorizer.transform(df_train[100:1100]['para'])
y_1000_test = df_train[100:1100]['label']

#training the classifier
classifier_seq.fit(X=x_100_train, y=y_100_train.to_list())
#calculating the accuracy on the testing set
acc_1000 = classifier_seq.score(X=x_1000_test, y=y_1000_test.to_list())

accuracies_seq = []
seq_correct = 0
seq_wrong = 0
y_test_list = []

while n_para <= 1100:
    
    x_test =  vectorizer.transform(df_train[n_start:n_start+n_instances]['para'])
    y_test = df_train[n_start:n_start+n_instances]['label']
    y_test_list.append(y_test.to_list())
    
    y_pred = classifier_seq.predict(X=x_test)
    y_pred = list(y_pred)
    
    #calculation of total number of correct and wrong prediction for the   next 1000 paras
    if len(y_test_list) <= 100:
        for ele_pred, ele_test in zip(y_pred, y_test.to_list()):
            if ele_pred == ele_test:
                seq_correct += 1
            else:
                seq_wrong += 1
    
    score = classifier_seq.score(X=x_test, y=y_test.to_list())
    accuracies_seq.append(score)
    
    n_start = n_start + n_instances
    n_para += n_instances
   
#total number of correctly classified and incorrectly classified samples
seq_correct, seq_wrong
[Out] : (681, 319)

With Active Learning

The Active Learning classifier will be trained iteratively.

After training using the first 100 paragraphs, accuracy on the next 10 paragraphs is predicted. We take note of correct and incorrect paragraphs predicted on unseen data. These 10 paragraphs will now be used in training. This will happen subsequently until all 1000 paragraphs are used in training. Similar to the sequential learning classifier, the total number of correct and wrong predictions for the next 1000 paragraphs are checked.

#the number of paragraphs in validation set for each iteration
n_instances = 10

#first 100 paragraphs
n_start = 100 

#defining the classifier
classifier = MultinomialNB()

#preprocessing of training set
x_train = vectorizer.transform(df_train[:100]['para'])
y_train = df_train[:100]['label']

#training the classifier
classifier.fit(X=x_train, y=y_train.to_list())

accuracies = []
test_accuracies = []

correct = 0
wrong = 0

n_para = 100
y_test_list = []

while x_train.shape[0] <= 1100:
    
    #calculating test accuracy for each iteration
    test_score = classifier.score(X=X_test, y=df_test.label.to_list())
    test_accuracies.append(test_score)
    
    #calculating validation accuracy for every next 10 paras
    x_test = vectorizer.transform(df_train[n_start:n_start+n_instances]['para'])
    y_test = df_train[n_start:n_start+n_instances]['label']
    y_test_list.append(y_test.to_list())
    y_pred = classifier.predict(X=x_test)
    y_pred = list(y_pred)
    
    ##calculation of total number of correct and wrong prediction for the next 1000 paras
    if len(y_test_list) <= 100:
        for ele_pred, ele_test in zip(y_pred, y_test.to_list()):
            if ele_pred == ele_test:
                correct += 1
            else:
                wrong += 1

    score = classifier.score(X=x_test, y=y_test.to_list())
    accuracies.append(score)
    #sequentially increasing the number of para
    x_train = vectorizer.transform(df_train[:n_start+n_instances]['para'])
    y_train = df_train[:n_start+n_instances]['label']
    classifier = MultinomialNB()
    classifier.fit(X=x_train, y=y_train.to_list())
    
    n_start = n_start + n_instances
    n_para += n_instances

#total number of correctly classified and incorrectly classified samples
correct, wrong
[Out] : (844, 156)

Results

Let’s take a look at the results we obtained from the experiments:

The first graph indicates that with more paragraphs used for training, the accuracy of the active learning classifier increases. Whereas, for the Sequential Learning algorithm the accuracy does not improve.

The second graph shows the accuracy of the two classifiers after the addition of every 10 paragraphs over the initial 100 paragraphs of the unseen testing dataset.

Conclusion

The results show that the Active Learning Classifier outperforms the Sequential Learning Classifier. This is because Active Learning learns through intermediate validations, whereas the Sequential Learning classifier only learns at 100 and 1100 paragraphs only and does not understand anything in between. This is how the active Learning classifier can achieve higher accuracy while using less labeled data.