Choosing a Data Pipeline Orchestrator: Cron Jobs to Dagster

It started with a single cron job, then grew to a dozen, across different schedules, servers, and parameters, with no reliable way to verify whether any of them were working. This is a common trigger for adopting a pipeline orchestrator, and it’s what led us to evaluate one while building 7Seers, our AI-powered employability platform for engineering and management colleges.

Struggling with the same brittle cron jobs and silent failures? 47Billion’s Data Analytics team helps engineering teams design and migrate to modern orchestration layers like Dagster. Talk to our data engineers.

Where It Started

As 7Seers’ microservices grew, our background job setup started with one cron job to scrape job listings. Then came job expiry, marking listings stale every two to three days. Then credit expiry, monthly. Then nudge jobs for user notifications. And the scraper itself multiplied, different locations, different titles, each on its own schedule.

Before long, we had four job categories in production, each running multiple times with different configs. The cron table became unmanageable.

Then We Moved to Celery Beat

We moved to Celery with Beat Scheduler, a distributed task queue with scheduling built on top.

The problem: Celery is built for per-record, event-driven tasks. Process this document. Send this email. One record in, one result out. Our jobs didn’t look like that. Scraping wasn’t processing one listing, it was fetching thousands in bulk, through multiple stages. Credit expiry wasn’t reacting to events, it was scanning all users on a schedule.

When something went wrong, we had no visibility: no run history, no alerts, no way to know if a job had completed or silently failed. We only found out once a downstream effect surfaced.

Three problems, repeating:

•  No visibility – no single view of what’s running or what it produced

•  No alerting – failures silent until something broke downstream

•  No recovery – mid-pipeline failure meant re-running everything

Worth being precise here: Dagster and Celery aren’t strictly either/or. Dagster ships its own dagster-celery executor, so you can run Celery underneath Dagster for distributed step execution. We didn’t go that route, we wanted Celery out of the critical path entirely, but it’s a legitimate setup if you just need more parallelism and already like Dagster’s scheduling and visibility layer.

Figure 1: The journey from cron jobs → Celery Beat → Dagster

Looking at the Other Options

Apache Airflow has the biggest ecosystem of the three and is genuinely battle-tested, but it’s heavy for a team starting fresh. DAGs are plain Python, not YAML like people assume, though the scheduling config, retries, and connections still end up scattered across Python files, Airflow’s metadata database, and environment variables. Running the scheduler, webserver, and workers as separate pieces is really what slows a new team down, not the language.

Prefect is Python-native and easy to start, and has changed significantly since we first evaluated it. Back then it had no real way to tell you what a run produced, only that it ran. Prefect has since added an @materialize decorator for asset lineage, which closes some of that gap. But when we were evaluating tools, Dagster already had the asset model built in from the ground up instead of added onto a task-first design. That was the deciding factor.

Dagster solved both, out of the box.

Update, since we first wrote this: In July 2026, Prefect announced it is acquiring Dagster Labs. Both keep running independently under their own names, licenses, and pricing, so nothing here changes if you’re already using either one. Still worth knowing, since it says something about where this market’s headed.

What Dagster Actually Does

Dagster’s core design principle: a pipeline should produce something inspectable, traceable, and trustworthy, not just run. The framework has changed since its early releases. It started as ops wired into graphs, and has moved toward software-defined assets as the primary abstraction. Both terms still appear in the docs and in production code, so both are worth knowing.

Ops: the actual Python functions doing the work, fetching, deduplicating, classifying. Every asset is backed by one or more ops under the hood. Functionally similar to Celery tasks, but designed to chain into a graph.

Jobs: what actually runs and gets monitored, a slice of your op or asset graph. Our scraping job goes raw scrape, deduplicate, classify, post to DB. Each step feeds off the last one’s output, and if one fails, nothing after it runs.

Assets: the named, tracked output a pipeline is responsible for, a table, a Parquet file, a trained model. Most people writing Dagster today start with assets: you say what the asset is and how to build it, and Dagster figures out the order to run things in. The dashboard tells you what the last run of each asset actually produced, when, and how many records came out.

Schedules: live in the same Python file as the job itself. No separate cron table, nothing extra to maintain.

Sensors / webhooks: trigger pipelines off events instead of time. A new file landing in a bucket or an external webhook can kick off a run.

Declarative Automation: the newer way to automate materializations, on top of schedules and sensors. Instead of a cron string, you write a condition, on_cron(), any_deps_updated(), or a mix, and Dagster re-evaluates it roughly every 30 seconds and only runs what actually needs to run. It replaced the older AutoMaterializePolicy API, which is now deprecated. We haven’t moved everything over to it yet, but it’s the direction new Dagster code is heading.

Figure 2: Ops chain into a Job, which produces a tracked Asset

What Changed for Us

Visibility. One dashboard shows every job, every run, every op: success, failure, timestamps. All four job categories, all in one place.

Alerting. Built-in failure notifications. Route to Slack, email, or webhook. We know when something breaks, not when a user reports it.

Recovery. Pipeline fails on op 4? Re-run from op 4. Earlier ops’ data is preserved. No need to re-run everything or write custom retry logic.

Concurrency. Running multiple scraping instances in parallel, different locations, same pipeline, just works. No workarounds needed.

Local development. dagster dev spins up the whole UI and daemon on your own machine, so you can materialize an asset and watch it run without touching a shared or production environment. This cut down the iteration time needed just to verify whether a job ran correctly.

Cron, Celery, and Dagster Side by Side

Why Bulk Data Changes Things

Celery is optimised for per-record work: one event, one task, one result. That’s right for transactional emails or per-document inference.

Dagster is optimised for the bulk model: take a large dataset, transform all of it at each stage, pass the result forward. Scraping is bulk. Recommendation generation is bulk too, you don’t compute one student’s recommendations, you process all students’ activity together and output recommendations for everyone in a single pass.

It also changes where the data actually lives. Raw scraped records, pre-aggregated activity logs, none of that intermediate stuff needs to touch Postgres. It sits in Parquet files instead, passed cheaply between stages through a Dagster I/O manager, and only the final, filtered output hits the database. Storage costs go down, the DB stays lean, and the pipeline itself becomes the record of what actually happened.

This is the same pattern we see across data engineering work at 47Billion: once you split the cheap, disposable, columnar storage from the database your product actually queries, costs go down and it’s just clearer what’s authoritative.

When Dagster Actually Makes Sense

Reach for Dagster when:

•  More than a handful of scheduled jobs and tracking them has become a daily chore

•  Multi-step pipelines where mid-failure recovery matters

•  Bulk data, processing sets of records, not individual ones

•  Audit-ability, knowing what each run actually produced

•  Schedule and logic in one place, no separate config files to fall out of sync

If your cron jobs are simple and working, leave them as is. Once job count grows, pipelines become multi-step, or you lose visibility into what a run actually produced, Dagster becomes the more direct choice.

Frequently Asked Questions

1. What is Dagster and how is it different from cron jobs?

Dagster is a data pipeline orchestration platform for Python. Unlike cron jobs, it gives you run history, per-op logging, retry logic, asset lineage tracking, and a real-time dashboard, so you always know what ran, what it produced, and what failed. This matters most once you have more than a handful of scheduled jobs to keep track of.

2. When should I use Dagster instead of Celery?

Use Celery for per-record, event-driven tasks in near real-time, like sending an email or processing one uploaded document. Use Dagster for multi-stage bulk pipelines where each stage transforms a full dataset and passes it to the next, like scraping thousands of listings or generating recommendations for every user in one pass.

3. How does Dagster compare to Airflow?

Airflow’s the most mature of the three, with by far the biggest ecosystem of integrations, and its DAGs are Python, not YAML. The overhead comes from running separate scheduler, webserver, and worker components, not from how you write the config. Dagster’s usually faster to get a new team up on, and comes with asset lineage built in; with Airflow you’d bolt on something like OpenLineage to get the same thing.

4. What are Dagster assets?

Assets are the named, tracked outputs a pipeline is responsible for: a database table, a Parquet file, a trained model. Ops are the Python functions underneath that compute those things. Most teams writing Dagster today start from assets; ops are still there doing the work, just one layer down.

5. Does Dagster work with Parquet?

Yes. Through Dagster’s I/O manager abstraction, assets can be Parquet files, CSVs, or objects in S3/GCS just as easily as database rows. Keeping intermediate data in Parquet and writing only final results to Postgres is a common, cost-effective pattern.

6. Is Prefect a good alternative to Dagster now that it has asset tracking too?

Prefect’s @materialize decorator, added in 2025, added asset lineage onto its existing task model, reducing that gap. Dagster’s asset model is still more baked into scheduling, partitioning, and the UI, since it was built asset-first from day one. If you’re already happy with Prefect’s lighter flow/task setup, this is worth a look. If lineage and backfills are what you actually need as core features, Dagster’s still the more direct route.

7. Are Dagster and Prefect merging?

In July 2026, Prefect agreed to acquire Dagster Labs. Both products keep running independently under their own names, licenses, and pricing, so it doesn’t change which tool actually fits your use case. It’s a business story, not a technical one. Read the announcement.

Migrating off cron jobs or Celery and evaluating orchestration tools for your own data platform? 47Billion’s Data Analytics and AI/ML teams help engineering teams design, migrate, and operate production data pipelines. Get in touch to talk through your setup.