Model Card for boltuix/bert-mobile

The boltuix/bert-mobile model is a mobile-optimized BERT variant designed for natural language processing tasks requiring efficient performance on resource-constrained devices like mobile phones and edge hardware. Pretrained on English text using masked language modeling (MLM) and next sentence prediction (NSP) objectives, it is optimized for fine-tuning on a range of NLP tasks, including sequence classification, token classification, and question answering. With a size of ~140 MB, it can be quantized to ~25 MB with no major loss in performance, making it ideal for mobile and edge applications needing strong performance with minimal resource usage.

Model Details

Model Description

The boltuix/bert-mobile model is a PyTorch-based transformer model derived from TensorFlow checkpoints in the Google BERT repository. It builds on research from On the Importance of Pre-training Compact Models (arXiv) and Generalization in NLI: Ways (Not) To Go Beyond Simple Heuristics (arXiv). Ported to Hugging Face, this uncased model (~140 MB) is engineered for mobile-optimized NLP applications, such as sentiment analysis, named entity recognition, and natural language inference, making it suitable for developers and researchers targeting efficient deployment on mobile and edge devices.

Developed by: BoltUIX
Funded by: BoltUIX Research Fund
Shared by: Hugging Face
Model type: Transformer (BERT)
Language(s) (NLP): English (en)
License: MIT
Finetuned from model: google-bert/bert-base-uncased

Model Sources

Repository: Hugging Face Model Hub
Paper: BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

Model Variants

BoltUIX offers a range of BERT-based models tailored to different performance and resource requirements. The boltuix/bert-mobile model is optimized for mobile and edge devices, offering strong performance with the ability to quantize to ~25 MB without significant loss. Below is a summary of available models:

Tier	Model ID	Size (MB)	Notes
Micro	boltuix/bert-micro	~15 MB	Smallest, blazing-fast, moderate accuracy
Mini	boltuix/bert-mini	~17 MB	Ultra-compact, fast, slightly better accuracy
Tinyplus	boltuix/bert-tinyplus	~20 MB	Slightly bigger, better capacity
Small	boltuix/bert-small	~45 MB	Good compact/accuracy balance
Mid	boltuix/bert-mid	~50 MB	Well-rounded mid-tier performance
Medium	boltuix/bert-medium	~160 MB	Strong general-purpose model
Large	boltuix/bert-large	~365 MB	Top performer below full-BERT
Pro	boltuix/bert-pro	~420 MB	Use only if max accuracy is mandatory
Mobile	boltuix/bert-mobile	~140 MB	Mobile-optimized; quantize to ~25 MB with no major loss

For more details on each variant, visit the BoltUIX Model Hub.

Uses

Direct Use

The model can be used directly for masked language modeling or next sentence prediction tasks, such as predicting missing words in sentences or determining sentence coherence, delivering strong accuracy for mobile applications.

Downstream Use

The model is designed for fine-tuning on a variety of downstream NLP tasks optimized for mobile and edge devices, including:

Sequence classification (e.g., sentiment analysis, intent detection)
Token classification (e.g., named entity recognition, part-of-speech tagging)
Question answering (e.g., extractive QA, reading comprehension)
Natural language inference (e.g., MNLI, RTE) It is recommended for developers and enterprises deploying NLP solutions on mobile devices or edge hardware where efficiency and performance are critical.

Out-of-Scope Use

The model is not suitable for:

Text generation tasks (use generative models like GPT-3 instead).
Non-English language tasks without significant fine-tuning.
Applications requiring maximum accuracy (use boltuix/bert-large or boltuix/bert-pro instead).

Bias, Risks, and Limitations

The model may inherit biases from its training data (BookCorpus and English Wikipedia), potentially reinforcing stereotypes, such as gender or occupational biases. For example:

from transformers import pipeline
unmasker = pipeline('fill-mask', model='boltuix/bert-mobile')
unmasker("The man worked as a [MASK].")

Output:

[
  {'sequence': '[CLS] the man worked as a engineer. [SEP]', 'token_str': 'engineer'},
  {'sequence': '[CLS] the man worked as a doctor. [SEP]', 'token_str': 'doctor'},
  ...
]

unmasker("The woman worked as a [MASK].")

Output:

[
  {'sequence': '[CLS] the woman worked as a teacher. [SEP]', 'token_str': 'teacher'},
  {'sequence': '[CLS] the woman worked as a nurse. [SEP]', 'token_str': 'nurse'},
  ...
]

These biases may propagate to downstream tasks. While the model’s size (~140 MB, quantizable to ~25 MB) makes it suitable for mobile devices, its performance may be limited for complex tasks compared to larger variants.

Recommendations

Users should:

Conduct bias audits tailored to their application.
Fine-tune with diverse, representative datasets to reduce bias.
Apply quantization to reduce the model size to ~25 MB for ultra-efficient mobile deployment.

How to Get Started with the Model

Use the code below to get started with the model.

from transformers import pipeline, BertTokenizer, BertModel

# Masked Language Modeling
unmasker = pipeline('fill-mask', model='boltuix/bert-mobile')
result = unmasker("Hello I'm a [MASK] model.")
print(result)

# Feature Extraction (PyTorch)
tokenizer = BertTokenizer.from_pretrained('boltuix/bert-mobile')
model = BertModel.from_pretrained('boltuix/bert-mobile')
text = "Replace me by any text you'd like."
encoded_input = tokenizer(text, return_tensors='pt')
output = model(**encoded_input)

Training Details

Training Data

The model was pretrained on:

BookCorpus: ~11,038 unpublished books, providing diverse narrative text.
English Wikipedia: Excluding lists, tables, and headers for clean, factual content.

See the BoltUIX Dataset Card for more details.

Training Procedure

Preprocessing

Texts are lowercased and tokenized using WordPiece with a vocabulary size of 30,000.
Inputs are formatted as: [CLS] Sentence A [SEP] Sentence B [SEP].
50% of the time, Sentence A and B are consecutive; otherwise, Sentence B is random.
Masking:
- 15% of tokens are masked.
- 80% of masked tokens are replaced with [MASK].
- 10% are replaced with a random token.
- 10% are left unchanged.

Training Hyperparameters

Training regime: fp16 mixed precision
Optimizer: Adam (learning rate 1e-4, β1=0.9, β2=0.999, weight decay 0.01)
Batch size: 256
Steps: 900,000
Sequence length: 128 tokens (90% of steps), 512 tokens (10% of steps)
Warmup: 9,000 steps with linear learning rate decay

Speeds, Sizes, Times

Training time: Approximately 180 hours
Checkpoint size: ~140 MB (quantizable to ~25 MB)
Throughput: ~110 sentences/second on TPU infrastructure

Evaluation

Testing Data, Factors & Metrics

Testing Data

Evaluated on the GLUE benchmark, including tasks like MNLI, QQP, QNLI, SST-2, CoLA, STS-B, MRPC, and RTE.

Factors

Subpopulations: General English text, academic, and professional domains
Domains: News, books, Wikipedia, scientific articles

Metrics

Accuracy: For classification tasks (e.g., MNLI, SST-2)
F1 Score: For tasks like QQP, MRPC
Pearson/Spearman Correlation: For STS-B

Results

GLUE test results (fine-tuned):

Task	MNLI-(m/mm)	QQP	QNLI	SST-2	CoLA	STS-B	MRPC	RTE	Average
Score	83.9/82.7	71.5	89.9	92.7	51.8	85.0	88.0	66.2	79.0

Summary

The model delivers strong performance across GLUE tasks for a mobile-optimized model, with notable results in SST-2 and QNLI. It outperforms smaller variants like boltuix/bert-mid in tasks such as RTE and CoLA, making it a robust choice for mobile applications.

Model Examination

The model’s attention mechanisms were analyzed to ensure effective contextual understanding optimized for mobile deployment, with no significant overfitting observed during pretraining. Ablation studies validated the training configuration for efficient performance.

Environmental Impact

Carbon emissions estimated using the Machine Learning Impact calculator from Lacoste et al. (2019).

Hardware Type: 4 cloud TPUs (16 TPU chips)
Hours used: 180 hours
Cloud Provider: Google Cloud
Compute Region: us-central1
Carbon Emitted: ~130 kg CO2eq (estimated based on TPU energy consumption and regional grid carbon intensity)

Technical Specifications

Model Architecture and Objective

Architecture: BERT (transformer-based, bidirectional)
Objective: Masked Language Modeling (MLM) and Next Sentence Prediction (NSP)
Layers: 8
Hidden Size: 512
Attention Heads: 8

Compute Infrastructure

Hardware

4 cloud TPUs in Pod configuration (16 TPU chips total)

Software

PyTorch
Transformers library (Hugging Face)

Citation

BibTeX:

@article{DBLP:journals/corr/abs-1810-04805,
  author    = {Jacob Devlin and Ming{-}Wei Chang and Kenton Lee and Kristina Toutanova},
  title     = {{BERT:} Pre-training of Deep Bidirectional Transformers for Language Understanding},
  journal   = {CoRR},
  volume    = {abs/1810.04805},
  year      = {2018},
  url       = {http://arxiv.org/abs/1810.04805},
  archivePrefix = {arXiv},
  eprint    = {1810.04805}
}

APA: Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018). BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. CoRR, abs/1810.04805. http://arxiv.org/abs/1810.04805

Glossary

MLM: Masked Language Modeling, where 15% of tokens are masked for prediction.
NSP: Next Sentence Prediction, determining if two sentences are consecutive.
WordPiece: Tokenization method splitting words into subword units.

More Information

See the Hugging Face documentation for advanced usage details.
Contact: boltuix@gmail.com

Model Card Authors

Hugging Face team
BoltUIX contributors

Model Card Contact

For questions, please contact boltuix@gmail.com or open an issue on the model repository.