| Title: | File Reading with Pre-Filtering, Pattern Searching, and Distributed Files |
| Version: | 0.1.0 |
| Description: | Provides high-performance tools for out-of-core text processing and data ingestion by leveraging system 'AWK' utilities. Allows users to count records, filter rows, and compute streaming aggregations—such as group-by means, streaming medians, standard deviations, and correlations—directly on disk prior to reading data into R. By delegating line-by-line filtering and summarization to system-level 'AWK' commands and streaming results back through data.table::fread(), the package significantly reduces memory footprint and execution times when working with large individual files or multi-file directory structures. |
| License: | MIT + file LICENSE |
| Encoding: | UTF-8 |
| RoxygenNote: | 7.3.3 |
| Suggests: | knitr, rmarkdown, testthat (≥ 3.0.0) |
| VignetteBuilder: | knitr |
| Config/testthat/edition: | 3 |
| Imports: | data.table |
| NeedsCompilation: | no |
| Packaged: | 2026-08-23 16:23:18 UTC; akshat |
| Author: | David Shilane [aut], Akshat Maurya [aut, cre], Jason Livingston [aut], Chung-Woo (Caffrey) Lee [aut], Mayur Bansal [aut], Srivastav Budugutta [aut] |
| Maintainer: | Akshat Maurya <codingmaster902@gmail.com> |
| Repository: | CRAN |
| Date/Publication: | 2026-09-03 12:20:11 UTC |
Fast Stream Aggregation of Delimited Files via AWK
Description
High-performance aggregation and streaming of multiple tabular data files using an
optimized AWK engine pipeline. This function parses columns, groups multi-file inputs,
and calculates summarizations (like sum(), mean(), and sd()) natively in the shell
before pulling structured aggregates back into R. It safely evaluates scalar mathematical transformations
passed as strings (e.g., "sqrt(col)").
Usage
aggregated.fread(
the.files,
value.code = "data",
delim = ",",
file.pattern = NULL,
recursive = FALSE,
num.batches = 1,
num.files.per.batch = 1000,
summarize.with = NULL,
return.as = "result",
group.by = NULL,
path.to.awk = "awk",
file.header = "file",
include.filename = FALSE,
skip = 0,
show.warnings = TRUE,
nrows = Inf,
sample.size.median = -1,
return.data.table = TRUE
)
Arguments
the.files |
A character vector containing paths to the targeted delimited text datasets. |
value.code |
A internal character tracking string for code generation mappings. Defaults to |
delim |
A character string defining the field separator delimiter within files. Defaults to |
file.pattern |
Optional character string to filter file names when |
recursive |
Logical. Should directory searches recurse into subdirectories when |
num.batches |
An integer defining the processing grouping structure. Defaults to |
num.files.per.batch |
An integer specifying the maximum threshold of files targeted per concurrent process execution loop. Defaults to |
summarize.with |
A named list specifying operations and target columns. Names must match supported functions:
|
return.as |
A character string defining what data is returned to the environment. Options include |
group.by |
A character vector or list containing column names to act as the aggregation grouping dimensions. Defaults to |
path.to.awk |
A character string declaring the binary location execution path. Defaults to |
file.header |
A character string tracking column identification titles when filenames are tracked. Defaults to |
include.filename |
A logical value indicating whether the tracking origin file string column should append to final structures. Defaults to |
skip |
An integer, list, or character regex pattern. Controls metadata/line skipping configuration routines before row parsing evaluations trigger. Defaults to |
show.warnings |
A logical value determining whether underlying structural data warnings should surface during operation execution cycles. Defaults to |
nrows |
A numeric ceiling threshold limiting output aggregation allocations. Defaults to |
sample.size.median |
Integer. Maximum number of observations per group used to compute streaming medians via the P-Square algorithm. A positive integer caps the sample size per group for
faster execution on large datasets. Set to |
return.data.table |
A logical value. If |
Value
Depending on the configuration argument passed to return.as, returns either a
data.table (data.frame if return.data.table = FALSE), a character vector representing full
system executable command configurations, or a combined metadata list package object.
Examples
# Create a sample CSV file using base R mtcars dataset
tmp_file <- tempfile(fileext = ".csv")
write.csv(mtcars, tmp_file, row.names = FALSE)
# Aggregate data over multiple columns grouped by cylinders
agg_res <- aggregated.fread(
the.files = tmp_file,
group.by = "cyl",
summarize.with = list(
mean = list("hp", "mpg"),
sd = "wt"
),
return.as = "all"
)
print(agg_res$result)
# Clean up temporary file
unlink(tmp_file)
Fast Batch Combining of Flat Files via AWK
Description
A streamlined wrapper around filtered.fread() designed to quickly
read, process, and combine multiple flat files into a single dataset without applying row filters.
Usage
combined.fread(
the.files,
path.to.awk = NULL,
header = TRUE,
the.variables = ".",
file.pattern = NULL,
recursive = FALSE,
include.filename = TRUE,
skip = 0,
file.header = "file",
num.files.per.batch = 1000,
return.as = "result",
envir = parent.frame(),
show.warnings = FALSE,
return.data.table = TRUE,
nrows = Inf,
drop = NULL,
...
)
Arguments
the.files |
A character vector of file paths to process. Non-existent files are automatically filtered out. |
path.to.awk |
A character string specifying the path to the AWK binary. If |
header |
A logical value indicating whether the target files contain a header row. Default is |
the.variables |
A character vector specifying which columns to retain. Use |
file.pattern |
Optional character string to filter file names when |
recursive |
Logical. Should directory searches recurse into subdirectories when |
include.filename |
A logical value indicating whether to include a source file tracking column
in the returned dataset. Default is |
skip |
A numeric offset, a character regex pattern, or a structured list indicating lines to bypass. If a list is used, it must follow dot notation:
Default is |
file.header |
A character string defining the column name for the tracked file origin.
Only utilized if |
num.files.per.batch |
An integer specifying how many files to aggregate per AWK system pipeline call.
Default is |
return.as |
A character string specifying the desired return object. Options are |
envir |
The environment context in which evaluation variables are evaluated. Default is |
show.warnings |
A logical value determining whether underlying |
return.data.table |
A logical value indicating whether to return a |
nrows |
An integer specifying the maximum total rows to parse out from the batch pipeline. Default is |
drop |
A character or numeric index vector specifying columns that should be explicitly excluded from the final output. |
... |
Extra parameters forwarded to underlying internal setup routines inside |
Value
A data.table (or data.frame), or a character vector containing the raw shell commands,
depending on the value passed to return.as.
Examples
# Create sample CSV files with metadata rows
f1 <- tempfile(fileext = ".csv")
f2 <- tempfile(fileext = ".csv")
writeLines(c("# Title: Jan Log", "# Status: Active", "id,val", "1,10", "2,20"), f1)
writeLines(c("# Title: Feb Log", "# Status: Active", "id,val", "3,30", "4,40"), f2)
# Combine multiple log files while skipping the 2 metadata rows
all_logs <- combined.fread(
the.files = c(f1, f2),
skip = list(skip.metadata.rows = 2, skip.data.rows = 0)
)
print(all_logs)
# Clean up temporary files
unlink(c(f1, f2))
Execute AWK Script over Batches of Files (Internal Engine)
Description
An internal helper function that splits file processing into batches, constructs
systemic shell commands to invoke AWK, and streams the processed text strings
back into R via data.table::fread.
Usage
execute.awk.stream(
awk.script.content,
the.files,
value.code,
header.names,
include.filename,
num.batches,
num.files.per.batch,
path.to.awk,
total.files,
show.warnings,
nrows,
file.header,
return.as
)
Arguments
awk.script.content |
A character string containing the raw body of the AWK script logic. |
the.files |
A character vector of normalized paths to the target files. |
value.code |
A character string indicating the identifier for code-only return mode. |
header.names |
A character vector of column names to assign to the resulting dataset. |
include.filename |
A logical value indicating whether to append the source filename tracking column. |
num.batches |
An integer specifying the total number of batches to chunk files into. |
num.files.per.batch |
An integer specifying the maximum number of files processed per AWK execution window. |
path.to.awk |
A character string designating the system path or command name for the AWK binary. |
total.files |
An integer tracking the total count of valid files to process. |
show.warnings |
A logical value. If |
nrows |
A numeric value restricting the maximum number of rows to read per batch chunk. |
file.header |
A character string establishing the column header name for file origin logging. |
return.as |
A character string controlling the return type format ( |
Value
A named list containing two elements:
list.data |
A list of data tables containing parsed chunk outputs. |
expanded.statements |
A character vector containing the raw shell strings passed to the system command pipeline. |
Fast, Filtered Reading of Multiple Files via AWK
Description
Translates R-style filtering statements into highly efficient AWK commands to process, filter, and select specific columns from multiple flat files simultaneously. Batched outputs are fast-loaded and bound together into a single dataset.
Usage
filtered.fread(
the.files,
path.to.awk = NULL,
file.pattern = NULL,
recursive = FALSE,
header = TRUE,
delim = ",",
the.filter = NULL,
the.variables = ".",
include.filename = TRUE,
skip = 0,
file.header = "file",
num.files.per.batch = 1000,
return.as = "result",
envir = parent.frame(),
and.symbol = "&",
or.symbol = "|",
in.symbol = "%in%",
nin.symbol = "%nin%",
show.warnings = FALSE,
return.data.table = TRUE,
nrows = Inf,
drop = NULL,
...
)
Arguments
the.files |
A character vector of file paths to process. Non-existent files are automatically filtered out. |
path.to.awk |
A character string specifying the path to the AWK binary. If |
file.pattern |
Optional character string to filter file names when |
recursive |
Logical. Should directory searches recurse into subdirectories when |
header |
A logical value indicating whether the target files contain a header row. Default is |
delim |
A character string specifying the column separator within the files. Default is |
the.filter |
A character string or unquoted expression outlining the filtering logic to pass to AWK.
Supports implicit translation of basic math and symbolic calls. Default is |
the.variables |
A character vector specifying which columns to retain. Use |
include.filename |
A logical value indicating whether to include a source file tracking column
in the returned dataset. Default is |
skip |
A numeric offset, a character regex pattern, or a structured list indicating lines to bypass. If a list is used, it must follow dot notation:
Default is |
file.header |
A character string defining the column name for the tracked file origin.
Only utilized if |
num.files.per.batch |
An integer specifying how many files to aggregate per AWK system pipeline call.
Default is |
return.as |
A character string specifying the desired return object. Options are:
|
envir |
The environment context in which evaluation characters or external metadata strings are parsed.
Default is |
and.symbol |
A character replacement flag for logical AND statements. Default is |
or.symbol |
A character replacement flag for logical OR statements. Default is |
in.symbol |
A character replacement flag for inclusion tests. Default is |
nin.symbol |
A character replacement flag for exclusion tests. Default is |
show.warnings |
A logical value determining whether underlying |
return.data.table |
A logical value indicating whether to return a |
nrows |
An integer specifying the maximum total rows to parse out from the batch pipeline. Default is |
drop |
A character or numeric index vector specifying columns that should be explicitly excluded from the final output. |
... |
Extra parameters forwarded to underlying internal setup routines. |
Value
A data.table (or data.frame), or a character vector containing the raw shell commands,
depending on the value passed to return.as.
Examples
# Create a sample CSV file using base R mtcars dataset
tmp_file <- tempfile(fileext = ".csv")
write.csv(mtcars, tmp_file, row.names = FALSE)
# Standard usage with a numeric filter and dot-notation row skipping
my_data <- filtered.fread(
the.files = tmp_file,
the.filter = "hp > 100",
skip = list(skip.metadata.rows = 0, skip.data.rows = 2)
)
print(my_data)
# Clean up temporary file
unlink(tmp_file)
Intelligent Auto-Detection for System AWK Binary
Description
An internal helper that searches the system environment and common Windows installation paths (Rtools, Git Bash) to find a working AWK interpreter.
Usage
find.awk.binary()
Value
A character string representing the absolute path to the AWK binary.
Pattern-Based Subsetting and Reading of Multiple Files via AWK
Description
Reads and aggregates multiple flat files concurrently, filtering rows based on regular expression patterns processed directly via AWK before parsing the data into R.
Usage
pattern.fread(
the.files,
path.to.awk = NULL,
header = TRUE,
the.patterns = NULL,
file.pattern = NULL,
recursive = FALSE,
tf = TRUE,
delim = ",",
connectors = "or",
the.variables = ".",
include.filename = TRUE,
skip = 0,
file.header = "file",
num.files.per.batch = 1000,
return.as = "result",
envir = parent.frame(),
show.warnings = FALSE,
return.data.table = TRUE,
nrows = Inf,
drop = NULL,
...
)
Arguments
the.files |
A character vector of file paths to process. Non-existent files are automatically filtered out. |
path.to.awk |
A character string specifying the path to the AWK binary. If |
header |
A logical value indicating whether the target files contain a header row. Default is |
the.patterns |
A character vector containing the regular expression patterns to match against data rows. Default is |
file.pattern |
Optional character string to filter file names when |
recursive |
Logical. Should directory searches recurse into subdirectories when |
tf |
A logical value determining whether to include rows that match the patterns ( |
delim |
A character string specifying the column separator within the files. Default is |
connectors |
A character string defining how multiple patterns should be combined logically.
Options are |
the.variables |
A character vector specifying which columns to retain. Use |
include.filename |
A logical value indicating whether to include a source file tracking column
in the returned dataset. Default is |
skip |
A numeric offset, a character regex pattern, or a structured list indicating lines to bypass. If a list is used, it must follow dot notation:
Default is |
file.header |
A character string defining the column name for the tracked file origin.
Only utilized if |
num.files.per.batch |
An integer specifying how many files to aggregate per AWK system pipeline call.
Default is |
return.as |
A character string specifying the desired return object. Options are |
envir |
The environment context in which evaluation characters are parsed. Default is |
show.warnings |
A logical value determining whether underlying terminal messages should be shown. Default is |
return.data.table |
A logical value indicating whether to return a |
nrows |
An integer specifying the maximum total rows to parse out. Default is |
drop |
A character or numeric index vector specifying columns to explicitly exclude. |
... |
Extra parameters forwarded to internal setup routines. |
Value
A data.table (or data.frame) containing rows satisfying the pattern configurations.
Examples
# Create sample log files
log1 <- tempfile(fileext = ".log")
log2 <- tempfile(fileext = ".log")
writeLines(c("INFO: System started", "Error: Disk full"), log1)
writeLines(c("INFO: User login", "Critical: Database unreachable"), log2)
# Extract rows matching "Error" or "Critical" across log files
errors <- pattern.fread(
the.files = c(log1, log2),
the.patterns = c("Error", "Critical"),
connectors = "or"
)
print(errors)
# Clean up temporary files
unlink(c(log1, log2))
Efficient Record and Row Counting via AWK
Description
Computes the total number of records matching specific logical filtering criteria across multiple large files using AWK. This performs counting at the shell level, eliminating the overhead of loading whole datasets into memory.
Usage
record.count(
the.files,
path.to.awk = NULL,
delim = ",",
the.filter = NULL,
file.pattern = NULL,
recursive = FALSE,
the.variables = ".",
include.filename = TRUE,
skip = 0,
file.header = "file",
num.files.per.batch = 1000,
return.as = "result",
envir = parent.frame(),
and.symbol = "&",
or.symbol = "|",
in.symbol = "%in%",
nin.symbol = "%nin%",
show.warnings = FALSE,
nrows = Inf,
drop = NULL,
...
)
Arguments
the.files |
A character vector of file paths to scan. Non-existent files are automatically filtered out. |
path.to.awk |
A character string specifying the path to the AWK binary. If |
delim |
A character string specifying the column separator within the files. Default is |
the.filter |
A character string or unquoted expression outlining the filtering logic to pass to AWK.
Default is |
file.pattern |
Optional character string to filter file names when |
recursive |
Logical. Should directory searches recurse into subdirectories when |
the.variables |
A character vector specifying active evaluation columns. Default is |
include.filename |
A logical value indicating whether to retain tracking metrics grouped by individual files. Default is |
skip |
A numeric offset, a character regex pattern, or a structured list indicating lines to bypass. If a list is used, it must follow dot notation:
Default is |
file.header |
A character string defining the tracking header index. Default is |
num.files.per.batch |
An integer specifying how many files to aggregate per pipeline call. Default is |
return.as |
A character string specifying the desired return format. Options are |
envir |
The environment context in which variables are parsed. Default is |
and.symbol |
A character replacement flag for logical AND statements. Default is |
or.symbol |
A character replacement flag for logical OR statements. Default is |
in.symbol |
A character replacement flag for inclusion tests. Default is |
nin.symbol |
A character replacement flag for exclusion tests. Default is |
show.warnings |
A logical value determining whether terminal messages are displayed. Default is |
nrows |
An integer specifying the maximum total matching record sets to count. Default is |
drop |
A character or numeric index vector specifying columns to exclude from mapping. |
... |
Extra parameters forwarded to underlying internal setup routines. |
Value
A data.table summarizing record counts per file (or overall), or raw shell string arrays.
Examples
# Create a sample CSV file with price data
tmp_file <- tempfile(fileext = ".csv")
write.csv(data.frame(id = 1:3, price = c(500, 15000, 20000)), tmp_file, row.names = FALSE)
# Get matching transaction counts without importing full rows
total_expensive_items <- record.count(
the.files = tmp_file,
the.filter = "price > 10000"
)
print(total_expensive_items)
# Clean up temporary file
unlink(tmp_file)
Core Translation Pipeline for Filtering Expressions
Description
Orchestrates the parsing and conversion of R-style filtering strings
or logical statements into valid, executable AWK syntax expressions. Maps variables
to their positional column indices (e.g., $1, $2) and handles
operating-system-specific quote adjustments.
Usage
translate.filtering.statement(
the.filter,
the.variables,
envir = parent.frame(),
and.symbol = "&",
or.symbol = "|",
in.symbol = "%in%",
nin.symbol = "%nin%",
equation.symbols = c(">=", ">", "<=", "<", "!=", "==")
)
Arguments
the.filter |
A character string or unquoted expression containing the R filtering statement. |
the.variables |
A character vector containing the full ordered variable names from the data header. |
envir |
The environment context in which variable evaluations are evaluated. Default is |
and.symbol |
A character tracking key for logical AND operations. Default is |
or.symbol |
A character tracking key for logical OR operations. Default is |
in.symbol |
A character tracking key for membership matching operations. Default is |
nin.symbol |
A character tracking key for excluded membership operations. Default is |
equation.symbols |
A character vector listing the recognized relational comparison operators.
Default is |
Value
A character string representing the compiled logic statement formatted for direct injection into an AWK pipeline execution.
Translate R Vector Membership Filters to AWK Logic
Description
Parses explicit vector membership statements containing the %in% operator,
translating them into structurally matching compound matching loops or array validations in AWK syntax.
Usage
translate.in.statement(
in.statement,
the.variables,
nin.symbol = "%nin%",
in.symbol = "%in%",
envir = parent.frame()
)
Arguments
in.statement |
A character string representing an isolated membership statement fragment. |
the.variables |
A character vector matching data frame column names to map to column indexes. |
nin.symbol |
A character structural definition string representing negative matches. Default is |
in.symbol |
A character structural definition string representing positive matches. Default is |
envir |
The evaluation context environment frame. Default is |
Value
A character string containing the mapped structural subset layout for the AWK statement block.
Global Scope Evaluation Framework for Membership Vectors
Description
Evaluates membership variables that point directly to larger active environment structures (like global character vectors) rather than internal row variables, building optimized inline conditional maps.
Usage
translate.in.statement.global(
in.statement,
the.variables,
in.symbol,
envir = .GlobalEnv
)
Arguments
in.statement |
A character expression fragment mapping lookup requirements. |
the.variables |
A character vector matching the positional target mapping structure. |
in.symbol |
The specific string structure pattern to catch. |
envir |
The active R runtime environment containing the target variable content array. Default is |
Value
A resolved logical text mapping line compatible with text extraction shell commands.
Parse and Translate Standard R Logical Operators
Description
Evaluates low-level logical connectors within a segment of text, converting
R's boolean layout terms (such as & or |) into corresponding AWK relational syntax blocks.
Usage
translate.logical.statement(
the.statement,
the.variables,
envir = parent.frame()
)
Arguments
the.statement |
A character string isolating a single logical operation block. |
the.variables |
A character vector containing the recognized file variable headers. |
envir |
The environment context in which the evaluation elements reside. Default is |
Value
A processed character string containing translated logical characters compatible with AWK.
Translate Negated R Vector Membership Filters to AWK Logic
Description
Special-case handler designed to invert membership matching statements containing
the %nin% operation rules, converting them cleanly into negated validation criteria strings for AWK.
Usage
translate.nin.statement(
nin.statement,
the.variables,
nin.symbol = "%nin%",
in.symbol = "%in%",
envir = parent.frame()
)
Arguments
nin.statement |
A character string representing an isolated negative matching fragment. |
the.variables |
A character vector identifying known variable column allocations. |
nin.symbol |
The target string pattern matching an exclusion declaration. Default is |
in.symbol |
The target string pattern matching an inclusion declaration. Default is |
envir |
The system environment lookup layer for evaluating vectors. Default is |
Value
An isolated, translated character block containing negated relational checks.