Parag Ekbote

AINovice2005

https://paragekbote.github.io/

AI & ML interests

ML Engineer passionate about taking models from research to production. 1 year supporting tech startups. Active OSS contributor.

Recent Activity

posted an update 2 days ago

Excited to share the release of dagster-hf-datasets: A Dagster-native integration that brings Hugging Face Datasets into Dagster's asset-oriented orchestration model The integration enables: • 🤗 Dataset and DatasetDict assets • 🐙 Dagster asset lineage and observability • 📦 Parquet-backed materialization via HFParquetIOManager • 🚀 Publishing curated datasets back to the Hugging Face Hub • 📝 Automatic dataset card generation from pipeline metadata As the Hub continues to grow beyond 1M+ datasets, orchestration, reproducibility, and observability are becoming increasingly important parts of the dataset lifecycle. I'm also working on a longer article covering the architecture and data pipelines enabled by the integration. More Soon! https://github.com/dagster-io/community-integrations/tree/main/libraries/dagster-hf-datasets https://github.com/dagster-io/community-integrations/tree/main/libraries/dagster-hf-datasets/docs

updated a dataset 3 days ago

AINovice2005/golden-glue-qqp

updated a Space 6 days ago

the-hf-stack/README

View all activity

Organizations

Excited to share the release of dagster-hf-datasets: A Dagster-native integration that brings Hugging Face Datasets into Dagster's asset-oriented orchestration model

The integration enables:

• 🤗 Dataset and DatasetDict assets
• 🐙 Dagster asset lineage and observability
• 📦 Parquet-backed materialization via HFParquetIOManager
• 🚀 Publishing curated datasets back to the Hugging Face Hub
• 📝 Automatic dataset card generation from pipeline metadata

As the Hub continues to grow beyond 1M+ datasets, orchestration, reproducibility, and observability are becoming increasingly important parts of the dataset lifecycle. I'm also working on a longer article covering the architecture and data pipelines enabled by the integration.

More Soon!

https://github.com/dagster-io/community-integrations/tree/main/libraries/dagster-hf-datasets

https://github.com/dagster-io/community-integrations/tree/main/libraries/dagster-hf-datasets/docs