End-to-End OCR-Free Image Translation

Gujarati ↔ English Translation Directly from Images

Revolutionary approach to translate text in images without relying on OCR, specifically designed for low-resource languages like Gujarati using Vision Transformers.

View on GitHub Learn More
Gujarati Text Image
English Translation

Key Highlights

Low-Resource Translation

Extended the model to Gujarati, a language with limited OCR and parallel resources, enabling translation in resource-constrained environments.

Vision Transformer Upgrade

Replaced ResNet with Vision Transformer (ViT) for better visual-text alignment and improved image understanding capabilities.

Hybrid Training Data

Combined 100k real OCR-English pairs with synthetic image-text data to improve model generalization and robustness.

Direct Translation

Enables direct translation from images without OCR preprocessing, reducing error propagation from traditional OCR+MT cascades.

Multi-Task Learning

Incorporates auxiliary MT (text-to-text) and OCR tasks with shared encoder-decoder parameters for better alignment.

CLIP Integration

Uses CLIP Vision Encoder for robust image feature extraction and better cross-modal understanding.

Model Architecture

Image Input

224x224 RGB Images

↓

CLIP Vision Encoder

Feature Extraction

↓

Transformer Architecture

Encoder-Decoder with Cross-Attention

↓

Text Output

Translated Text

Technical Specifications

  • Image Encoder: CLIP Vision Encoder for robust visual feature extraction
  • Text Processing: Transformer-based sequence-to-sequence architecture
  • Model Dimension: 768-dimensional embeddings
  • Attention Heads: 8 multi-head attention mechanisms
  • Layers: 6 encoder layers, 6 decoder layers
  • Training Strategy: Multi-task learning with MT, OCR, and TIT tasks

Dataset Information

Real Data

100,000 Gujarati text images with corresponding English translations

  • Handwritten and printed text
  • Various fonts and styles
  • Real-world image conditions

Synthetic Data

Generated text images with diverse backgrounds and fonts

  • Multiple background textures
  • Various font families
  • Improved generalization

OCR Augmentation

OCR-extracted Gujarati text aligned with English translations

  • Supervised training data
  • Text-level alignment
  • Multi-task learning support

Research Foundation

This project is based on the research paper:

"Improving End-to-End Text Image Translation From the Auxiliary Text Translation Task"

Ma, Cong and Zhang, Yaping and Tu, Mei and Han, Xu and Wu, Linghui and Zhao, Yang and Zhou, Yu

arXiv preprint arXiv:2210.03887, 2022

Read Paper