Automatic Data Annotation: Next Breakthrough

Data Validation through the DataNeuron ALP

Teams in nearly all fields, spend a majority of their time on research and finding chunks of important information from the huge bulk of unfiltered data and documents that is present within the organization. This process is very time consuming and tedious.

In fields like data science and machine learning, getting annotated data is one of the biggest hurdles and one that the teams tend to spend the most time on.

Apart from this, data annotation can often prove to be expensive as well. Multiple human annotators might need to be hired and this can increase the overall cost of the project.

The DataNeuron platform enables organizations to get accurately annotated data, while minimizing the time, effort and cost expenditure.

DataNeuron’s Semi-Supervised Annotation

What does the platform provide?

The user is provided with an option to define a project structure, which is not limited to a flat classification hierarchy but can incorporate a multilevel hierarchical structure as well with indefinite levels of parent-child relationships between nodes.

This aids research, since the data is essentially divided into groups and further sub-groups depending on the user preference and defined structure which enables the team to adopt a “top-down” approach for getting to the desired data.

The platform takes a semi-supervised approach to data annotation in the sense that the user is required to annotate only about 5–10% of the entire data and the platform annotates the remaining data automatically for the user by detecting contextual similarity and patterns in the data.

How the semi-supervised approach works?

Even for the 5–10% of the total data that still needs to be annotated, the time and effort spent is reduced by a large margin by adopting a suggestion-based validation technique.

The platform provides auto-labeling to the users and suggests the paragraphs that are likely to belong to a specific class based on label heuristics and contextual filtering algorithm; users have to accept or reject at the validation stage.

The semi-supervised approach for validation is broken down into stages:

• In the first stage, the user is provided with suggestions based on an intelligent context-based filtering algorithm. The validations done by the user in the first stage are used to improve the accuracy of the filtering algorithm used to provide suggestions for validations.
• In the second stage the validation is then further broken down into ‘batches’. This process is repeated for each batch of the second stage, i.e. the validations done in each batch are used to increase the accuracy of the filtering algorithm for the succeeding batch.

This breaks down the problem of annotating a data point into a “one-vs-all” problem which makes it far easier for the user to arrive at an answer(annotation) than if they had to consider all the classes (which might be a huge number depending on the complexity of the problem) for making each individual annotation.

Our platform is a “No-Code” platform and anyone with basic knowledge of the domain they are working on can use the platform to its maximum potential.

Testing On Various Datasets

The platform chooses from among multiple models trained on the same training data, to provide the best possible results to the users.

The average K-Fold accuracy of the model is presented as the final accuracy of the trained model.

We incur a relatively small drop in accuracy as a result of the decreased size of the training data as highlighted. This dip in accuracy is within 12% and can be controlled by the user by annotating more data, or choosing to add seed paragraphs during the validation or feedback and review stage.

Comparisons with an In-House Project

Difference in Paragraphs Annotated. We observe it is possible to reduce annotation effort by up to 96%.

Difference in Time Required. We observe it is possible to reduce time required for annotation by up to 98%.

Difference in Accuracy.

We observe that the DataNeuron platform can decrease the annotation time up to 98%. This vastly decreases the time and effort spent annotating huge amounts of data, and allows teams to focus more on the task at hand.

Additionally it can also help reduce the Subject Matter Expert effort up to 96%, while incurring a marginal cost. Our platform also helps reduce the overall cost of the project by a significant margin, by nearly eradicating the need for data labeling/annotation teams.

In most cases, the need for appointing an SME is also diminished, as the process of annotation is made much simpler and easier and anyone with knowledge of the domain and the project they are working on can be able to perform the annotations through our platform.