Custom translation risk prediction

Training, evaluating and deploying a ModelFront custom model

ModelFront provides both generic and custom risk prediction.

The ModelFront custom training process is a thorough case study - we'll guide you through the setup and dataset selection, review and train and retrain on your feedback, deploy a production-ready model, run an accuracy evaluation on your data and provide a report.


Should you customize?

Customization is as important for translation risk prediction as it is for machine translation.

Different use cases - filtering, comparison and hybrid translation - require different handling, and different content types, clients and projects have different style guidelines and quality bars.

For example, for hybrid translation in production, catching segments with source-side risk is key, and it's a good idea to catch all very out-of-domain or out-of-distribution segments - no matter what the translation is.

But for filtering training data, it's ideal to keep segments with source-side risk and from any domain or distribution, assuming the translation is good.

For basic filtering, effort estimation or validation on generic content, you can try the ModelFront default generic risk prediction model. For hybrid translation, a custom model is necessary, and for other production use cases like effort estimation and validation, we also recommend it.

ModelFront is massively multilingual - one custom model can support many language pairs - so a typical large enterprise needs only one or two models for a use case like hybrid translation.


The customization process

The customization process is transparent to the client and driven by the ModelFront team from end to end.

Assessment

  1. The client delivers the training data and any special instructions to ModelFront.
  2. ModelFront formats, evaluates and inspects the client datasets for custom requirements, noise and conflicts.
  3. ModelFront consults with the client.

Training

  1. ModelFront shapes the dataset - for example filtering out noise - and selects the best configuration.
  2. ModelFront trains and deploys an initial model.
  3. ModelFront runs an evaluation on a holdback set and shares it with the client.

The client sees the results and can also access the model to provide feedback. ModelFront will retrain the model iteratively.

Evaluation

  1. ModelFront runs an evaluation on a holdback set - data that the model has never seen before.
  2. ModelFront compiles accuracy metrics.
  3. ModelFront delivers the evaluation and a report with accuracy metrics and recommendations.

By the end of the process, the client understands how to proceed.

Deployment, integration and retraining

The custom model is transferred to the client before evaluation.

It is deployed for production and accessible to the client in both the API and console.

If a model is undeployed, it will generally be kept by ModelFront for months and redeployable at the client's request.

The API documentation describes the details for development and production integration. We recommend starting with a low risk threshold.

Periodic retraining is supported, to update with new client data and take advantage of future improvements in the ModelFront training system.


Dataset recommendations

Data for training should be similar to data in production. Each segment should correspond to approximately one sentence. The distribution of the dataset should be a realistic representation of the production content stream.

For hybrid translation - a production use case that requires high accuracy - we recommend at least 100K segments of post-edited data per language pair.

We recommend a separate model per logical grouping with consistent style and quality, typically a client or content type.

For models with many language pairs, it is possible to train on less data for some of the language pairs. For use cases that require lower accuracy, it is possible to train on less data or less consistent data.

Configurations

For hybrid translation, by default, a custom model is trained to predict whether or not a post-edit will be made. Note that there are other possible translation metrics, like post-editing distance and quality.

Clients can choose between two basic modes, strict and tolerant. In strict mode, any edit is consider an edit. In tolerant mode, edits that involve only casing, punctuation or whitespace are ignored. Enforcement of locale preferences is supported, depending on the data available.

Beyond that, it's possible for the ModelFront team to adjust the behavior of the model by adjusting the balance of the dataset or adding in-house curated datasets. It's also possible to order additional dataset curation, human labelling and logic for localization like unit conversions.

General strictness can be adjusted at any time by setting a lower or higher risk threshold, so it does not need to be decided before customizing.

Commitment requirements

Custom training, including an accuracy evaluation report and 4 months of deployment, requires a minimum upfront commitment of $12K, which covers the first 15M characters per month under the standard plan.

Keeping a model deployed after that requires a minimum commitment of $3K per month, which covers the first 15M characters per month under the standard plan.

Multiple custom models, advanced customization and advanced evaluation and require additional commitments.

That is, basic custom models are effectively provided at no additional cost to those clients using ModelFront at significant volume. For clients with no volume, a basic custom model effectively costs $12K.

Partner-level options, like human labeling, volume discounts and direct access to the training API, are also available.