Handling large datasets efficiently is a critical challenge in mobile app development, particularly as applications scale in complexity. This article examines advanced techniques for managing substantial data volumes in Flutter, with a focus on optimizing performance and scalability.
Challenges of Poor Data Management
Many Flutter applications need to work with extensive datasets, ranging from thousands to millions of records. This presents several key challenges:
- Excessive Memory Consumption: Loading large datasets into memory can lead to out-of-memory errors, especially on low-end devices.
- Wasted System Resources: Inefficient processing can cause high CPU usage and unnecessary disk or network I/O, draining battery life and overheating devices.
- App Crashes and Stability Issues: Poorly managed data can result in frequent crashes, unexpected shutdowns, and data corruption.
- Performance on Low-End Devices: High resource demands can render the app unusable on less powerful devices, leading to slow startup times and sluggish performance.
- Sluggish User Interface: Blocking the UI thread with heavy data processing causes freezes, slow scrolling, and choppy animations.
- Network-Related Issues: Inefficient data fetching can lead to excessive network usage, slow load times, and frustration for users with limited data plans.
- Scalability Problems: As datasets grow, performance can degrade significantly, complicating maintenance and feature additions.
- Battery Drain: Constant processing or network operations contribute to rapid battery depletion.
- Compliance and Security Risks: Improper handling of sensitive data can expose vulnerabilities and lead to regulatory non-compliance.
Pros and Cons of Different Approaches for Managing Large Datasets in Flutter
When managing large static datasets, it’s crucial to evaluate various approaches to determine the best fit for your Flutter application. Each method has its strengths and weaknesses, which can significantly impact performance, user experience, and overall app functionality.
A) Static Const Models
Static const models embed data directly into the app’s code, providing the fastest access but at the cost of increased app size and difficulty in updates.
Pros: Static const models provide the fastest runtime access, are type-safe, and have no parsing overhead.
Cons: They significantly increase app binary size, consume memory for the entire app lifecycle, and make updates difficult without redeployment.
B) JSON Strings with Runtime Parsing
This approach stores data as JSON strings, offering flexibility in updates but requiring runtime parsing, which can impact performance.
Pros: JSON strings result in a smaller app binary size, allow easier updates without changing code, and offer flexibility for dynamic data structures.
Cons: They incur slower access due to runtime parsing, consume memory once parsed, can lead to parsing errors at runtime, and may block the UI thread during large dataset processing.
C) Web Services
Web services (Firebase, Supabase, etc.) allow for real-time updates and scalability but require internet connectivity and may incur ongoing costs.
Pros: Web services enable real-time updates, reduce app size, and are scalable for large datasets.
Cons: They require internet connectivity, may incur ongoing costs for hosting and data transfer, can affect initial user experience due to set up times, and may be limited by data caps.
D) Assets File Storage
Assets file storage packages data with the app, ensuring offline availability and good performance, but requires encryption and complicates updates.
Pros: Assets file storage ensures offline availability and good performance by packaging data with the app.
Cons: It requires encryption to protect data, complicates the build process, necessitates app redeployment for updates, and increases app size (though less than static const models).
Choose Assets File Storage
Striking a balance between performance and manageability, we are opting for assets file storage for our Flutter application. This approach lowers memory use by loading data on demand, simplifies management by packaging data with the app, ensures offline access for a reliable user experience, and provides faster performance without network delays.
Architecture of Assets File Storage
Tech Stack
Choosing the right data formats and compression methods is essential for maintaining a robust and efficient design. Getting this wrong will undermine the project.
- Storage: JSON is Ideal for Flutter as it is lightweight, easy to read, and well-supported in various programming languages. It allows efficient data exchange between the Flutter app and backend services, making it perfect for dynamic content. Flutter’s libraries also simplify the process of parsing and generating JSON.
- Compression: Zip is a mature, fast, cross-platform, and efficient way to reduce file sizes. Using ZIP allows us to keep files separate for efficient iteration, enabling optional sequential or parallel processing.
- Encryption: ZIP offers a basic level of encryption that deters unauthorized access. A password length of 20 characters is sufficient to provide strong security. For example:
P@ssw0rd!123Secure#456. - Passwords: Passwords will be stored in a
.envfile (usingenvied, and also in plain text within the batch file. - Processing: 7-Zip is an open-source tool for command-line file compression and encryption —
p7zipon macOS. - Database: Select a disk-based database, such as
Isar-Communityor SQLite (sqflite), or reusing an existing database — provided it is not in-memory. - Source Control — The JSON files can be stored in source control, but the zip files will be excluded in the
.gitignore
Data Preparation
This phase focuses on organizing and securing data before it is used in the application.
- Source Folder Setup: Developers place JSON files in a designated project folder.
- Compile-Time Processing: The contents of the source folder are processed during the app’s compile phase.
- Zipping and Encrypting: JSON files are zipped and encrypted for security and efficiency.
- Target Folder Storage: Processed zip files are saved to a target folder within the application.
Data Importation
Choosing the right storage solution for unzipped data is critical; selecting an inappropriate method can negate the benefits of our data management efforts.
- Runtime Access: The application accesses the preprocessed data at runtime.
- Efficient Extraction: Implement non-blocking extraction methods to ensure data retrieval does not block the main thread.
- Version Control: Implement a version control mechanism to track changes in the data over time, allowing for updates and rollbacks as necessary.
- Separation: JSON files will contain dynamic content that can change over time, while static types and enums will be defined in the source code to maintain consistent core data types.
Recommended Folder Structure
The folder structure will clearly organize the project’s assets and scripts.
project_root/
│
├── assets/
│ ├── database_assets_raw/ # Original JSON files
│ └── database_assets_publish/ # Processed (zipped) JSON files for import
│
├── scripts/
│ └── zip_raw_assets_for_publish.ps1 # PowerShell for zipping JSON
│
└── pubspec.yaml # Flutter project configurationpubspec.yaml
flutter:
assets:
- assets/database_assets_publish/ # only publish processed assets.gitignore
# Ignore processed (zipped) JSON files for import
assets/database_assets_publish/env.dart
import 'package:envied/envied.dart';
part 'env_config.g.dart';
// Run the build_runner: Run the build_runner to generate the necessary code:
// dart run build_runner build --delete-conflicting-outputs
@Envied(path: '.env')
abstract class EnvConfig {
@EnviedField(varName: 'importPassword')
static final String importPassword = _EnvConfig.importPassword;
}Compression & Encryption PowerShell (Windows)
# Define relative paths and password
$sourceFolder = "database_assets_raw" # Source folder for raw assets
$destinationFolder = "database_assets_publish" # Output folder for ZIP files
$password = "P@ssw0rd!123Secure#456" # *YOUR* Password for ZIP encryption
# Check if the source folder exists
if (-Not (Test-Path $sourceFolder)) {
Write-Host "Source folder does not exist: $sourceFolder"
exit
}
# Get all JSON files in the source folder (not including subfolders)
$jsonFiles = Get-ChildItem -Path $sourceFolder -Filter *.json -File
# Create the destination folder if it does not exist
if (-Not (Test-Path $destinationFolder)) {
New-Item -ItemType Directory -Path $destinationFolder | Out-Null
}
# Loop through each JSON file and create an individual ZIP file
foreach ($file in $jsonFiles) {
# Define the output ZIP file path
$zipFileName = [System.IO.Path]::ChangeExtension($file.Name, ".zip")
$destinationZip = Join-Path -Path $destinationFolder -ChildPath $zipFileName
# Create a ZIP archive of the JSON file with password protection using 7-Zip
# Use 7z.exe without specifying the full path by adding the directory
# to your system's PATH environment variable
& "C:\Program Files\7-Zip\7z.exe" a -p$password -mhe=on $destinationZip "$($file.FullName)"
Write-Host "Created ZIP file: $destinationZip"
}Verifying the zips
Be aware of cached passwords, but otherwise you should be able to confirm the contents without unzipping:
Phase 2: Importing
Import Workflow and Design
[Asset Path] --> [Load JSON from asset]
|
+--> [JSON File] --> [Extract JSON from file]
|
+--> [ZIP File] --> [Extract JSON from ZIP]
|
v
[Parse JSON to map]Static Data Import Class
1. Load JSON from asset
• Determines file type from asset path (ZIP or JSON)
• Route to appropriate extraction method
2. Extract JSON from file
• Handles regular JSON files
• Loads and decodes file content
• Parses JSON to data structure
3. Extract JSON from ZIP
• Decompresses ZIP archive
• Locates JSON file within archive
• Extracts and parses JSON data
4. Load asset byte data
• Reads raw byte data from asset
• Creates mutable copy for processing
5. Parse JSON to map
• Converts JSON string to map structure
• Performs type checking on resultDatabase Data Addition
Workflow Class Diagram:
[Data Objects List] --> [Validate Data Objects]
|
+--> [Prepare Data for Insertion]
|
+--> [Perform Delta Check]
|
+--> [Delta Check Passed]
|
v
[Choose Insertion Method]
|
+-------------+-------------+
| |
[Bulk Insert] [Delta Update]
| |
[Clear Records] [Identify Changes]
| |
[Insert All] [Update Modified]
|
[Insert New]
|
[Delete Removed]Pseudocode:
1. Load Data Objects
• Validate the list of data objects.
2. Prepare Data for Insertion
• Map each data object to the corresponding database model.
• Create a list of these database models.
3. Perform Delta Check
• Compare the list with existing records to check for differences.
4. Bulk Insert (Original Approach)
• Clear the existing data set to ensure a fresh insertion.
• Insert all entries from the prepared list in a single transaction.
OR
4. Delta Update (Alternate Approach)
• Identify records that have been modified, added, or deleted.
• Update only the modified records in the database.
• Insert new records into the database.
• Delete records from the database that are not in the prepared list.Loading Data Phase
There is not much to say here, except you will read the static data like any other database fields. It is recommended to avoid writing to the static table and instead let the import JSON methods control content.
Static data displayed live in Saropa Contacts
Concluding Notes:
- Importing JSON: we avoid factory methods so that we can validate the import for correctness and ensure mandatory fields are provided. On error, we log warnings (with context) so that the data can be cleaned up.
- Zip Decoding: The ZipDecoder provides the file content as an unmodifiable list, but the decryption process requires a mutable list to function properly. Without this copy, we would encounter the error: “Unsupported operation: Cannot modify an unmodifiable list”.
- Static Database Design: Establish primary keys to prevent duplication, and consider version control issues. Sometimes a simple, performing solution for static data may involve complete deleting and bulk-adding static data.
- Obfuscation and Assets: The
— obfuscateflag primarily affects Dart code, not asset files. Assets (e.g., images, JSON files) remain unchanged, unencrypted, and easily accessible after extraction from the compiled app, leaving any sensitive information in these files vulnerable.
References & Packages
Compressing
- 7zip https://7-zip.org/download.html
- p7Zip https://p7zip.sourceforge.net/
- 7zip Command Line Version User’s Guide https://web.mit.edu/outland/arch/i386_rhel4/build/p7zip-current/DOCS/MANUAL/syntax.htm
- Archive package https://pub.dev/packages/archive
Database
- Isar Community https://isar-community.dev/
- Sqflite https://pub.dev/packages/sqflite
With 30 years in the tech industry, we’ve handled everything from coding to leading international projects. Our passion is sparking curiosity and deepening understanding of complex topics.
We’re dedicated to small-scale emergency preparedness, disaster planning, and reducing stress and financial burdens. Our app, Saropa Contacts, is more than just an address book — it’s your personal network navigator, linking people, companies, and emergency groups.
Our mission is to empower you and reduce the impact of crises.
If you have any suggestions or thoughts on this article, we invite your feedback, here or app.feedback@saropa.com.
Learn more at saropa.com
