One SQL Command vs. 13 Hours of Python

In tech, we reach for Python to solve almost any data problem. It's a versatile tool — but sometimes we overuse it and miss a more efficient solution already built into the database. Over two decades in cloud and database operations, I've learned that knowing when to lean on the database instead of code can make all the difference.

A recent example made the point cleanly: a 13-hour data load that finished in seconds once it was handled by a single SQL command instead of a script. Here's how the inefficiency surfaced, what changed, and what it says about how we pick tools.

The problem: a slow Python-based data load

We were loading somewhere between 100k and 250k rows into an RDS MySQL instance. A Python script handled the insertion, using executemany() to batch the inserts. That was faster than inserting one row at a time, but still slow and far from optimal.

At first it looked like a Python problem — surely there was a better way to batch the inserts. I suggested optimizing the script further. But after watching the job stretch past 13 hours, I started questioning whether Python was even the right tool for the task.

The switch: thinking like a DBA

I didn't want to just patch the Python code; I wanted to approach the problem from a database-first perspective. So I asked two questions:

• Why are we using Python when MySQL has built-in tools designed specifically for bulk loads?
• How do we let the database do what it does best — load large volumes of data efficiently?

That reframing brought it back to something familiar: Oracle's SQL*Loader, a trusted tool for bulk loading that handles these operations with exceptional speed. MySQL has an equivalent, highly efficient command — LOAD DATA INFILE.

The solution: LOAD DATA INFILE

A quick pass through the MySQL documentation confirmed it was exactly what we needed — a highly optimized command for loading data directly from a file into a table. I tested it on a local example, adjusted for the security settings and file permissions on the RDS instance, and ran it. What had taken over 13 hours in Python now completed in a few seconds.

LOAD DATA INFILE '/path/to/data.csv'
INTO TABLE my_table
FIELDS TERMINATED BY ','
LINES TERMINATED BY '\n'
IGNORE 1 LINES;

The strength of LOAD DATA INFILE is that the database performs the load natively, without looping through each row in application code — a significant performance gain for bulk inserts.

The lesson: don't overuse Python for bulk loads

This was a strong reminder that:

• Databases are built for bulk operations, and using them well often means relying on native features rather than custom code.
• Python isn't always the best choice for tasks like bulk inserts, especially when the database has a more efficient path.
• A database-first perspective can save hours of development and execution time.

Python is powerful and flexible, but it isn't always the right tool. Here it was slowing us down unnecessarily, and the database had the answer all along.

The value of perspective in operations leadership

This wasn't only about fixing a slow load — it reinforced how much perspective matters in leadership. Technical expertise is part of it; so is knowing when to step back and reassess the problem.

• Knowing the tools you have lets you spot inefficiencies early. While the team focused on optimizing Python, the database already had a built-in solution.
• Shifting from a Python-centric mindset to a database-first one was what actually solved it. Leadership here meant encouraging the team to challenge assumptions and look for a better path.
• The goal wasn't one faster task — it was a culture of continuous improvement, and helping the team find smarter ways to work even when that means leaving a familiar tool behind.