EF Core was not designed for bulk data operations
But I had to insert data from 50,000 IoT devices
When building data-intensive applications with Entity Framework Core, you will eventually face a critical performance challenge:
- inserting, updating, or deleting thousands of records efficiently.
The standard EF Core approach using SaveChanges() works perfectly for small datasets or even a few hundred rows.
But it becomes a significant bottleneck when you need to process thousands of records.
To understand why EF Core struggles with bulk operations, you need to understand what happens when you call SaveChanges().
Step 1: Change Detection
Step 2: SQL Statement Generation
Step 3: Parameter Binding
Step 4: Database Execution
Step 5: Identity Value Retrieval
Step 6: Change Tracker Update
After detecting changes, EF Core generates SQL statements for each entity that needs to be persisted.
For inserts, EF Core generates individual INSERT statements.
In SQL Server, EF Core uses a MERGE statement to insert multiple rows at once.
However, EF Core is limited by the SQL Server parameter limit (2,100 parameters per batch), which restricts the number of rows or columns in a single batch. Performance drops quickly when entities have many columns.
You might think about using raw SQL with a single INSERT statement.
This approach is faster than using SaveChanges(), but it has serious problems:
❌ SQL injection risk - Building SQL strings from user input is dangerous
❌ No type safety - You lose compile-time checking
❌ No navigation properties - You can't work with related entities
❌ Maintenance burden - You need to update SQL strings when your model changes
If you try to introduce parameters into the raw SQL string, Microsoft(.)Data(.)SqlClient will generate multiple insert statements (❌ ) instead of a single batch. And performance will drop even more than with EF Core SaveChanges.
I had to insert 50,000 telemetry records from IoT devices into the database every few minutes. Each device sends temperature readings, humidity levels, and status updates.
So I looked for a better solution. And I found the 𝗘𝗻𝘁𝗶𝘁𝘆 𝗙𝗿𝗮𝗺𝗲𝘄𝗼𝗿𝗸 𝗘𝘅𝘁𝗲𝗻𝘀𝗶𝗼𝗻𝘀 library.
Entity Framework Extensions provides high-performance bulk operations that can process thousands of records in less than a second. You still use the same code you have in EF Core with a new set of DbContext extension methods.
Tomorrow, I'll share with 21,320 people:
Why Every EF Core Developer Needs to Try Entity Framework Extensions.
This library saved me days of development and performance improvements. And the license paid for itself within the first month of use.
📌 If you want to join us, subscribe to my weekly newsletter:
↳
antondevtips.com/?utm_source…
——
♻️ Repost to help others use bulk operations in EF Core
➕ Follow me (
@AntonMartyniuk ) to improve your .NET and Architecture Skills