CATIE-AQ/LFM2-1.2B_french_dpo

Description

This model is an initial test conducted in the summer of 2025 shortly after the release of Liquid.AI's LFM2 model.
The idea was to quickly get an opinion on this hybrid convolutional SSM (StripedHyena) & attention architecture (we really love SSMs and especially convolutional SSMs, which we believe don't get enough attention compared to recurrent SSMs).
In particular, the resources required to train the compact base model (the 1.2B parameter model can be finetuned on one 80GB A100 without quantization, according to our observations).

As part of this initial test, we fine-tuned the model using the notebook provided by Liquid.AI on 195,583 non-synthetic French DPO data.

Usage

See original LFM-1.2B model card.

Planned improvements

• Addition of 150,000 more non-synthetic DPO data
• Addition of synthetic data
• Propose models in 350M and 700M versions

Environmental Impact

Carbon emissions were estimated using the Machine Learning Impact calculator presented in Lacoste et al. (2019). The hardware, runtime, cloud provider, and compute region were utilized to estimate the carbon impact.

• Hardware Type: A100 PCIe 40/80GB
• Hours used: 13h
• Cloud Provider: Private Infrastructure
• Carbon Efficiency (kg/kWh): 0.020kg (estimated from electricitymaps ; we take the carbon intensity in France between August 1, 11 p.m. and August 2, 12 p.m.
• Carbon Emitted (Power consumption x Time x Carbon produced based on location of power grid): 0.07 kg eq. CO2

License

LMF1.0