Skip to contents

Fork of inborutils::csv_to_sqlite to save a delimited text table into a single table sqlite database that can then be munged w dplyr/ dbplyr / sqldf

csv_to_sqlite.Rd

The table can be a comma separated (csv) or a tab separated (tsv) or any other delimited text file. The file is read in chunks. Each chunk is copied in the same sqlite table database before the next chunk is loaded into memory. See the INBO tutorial Handling large files in R to learn more.

Usage

csv_to_sqlite(
  csv_file,
  sqlite_file,
  table_name,
  delimiter = ",",
  pre_process_size = 1000,
  chunk_size = 50000,
  show_progress_bar = TRUE,
  ...
)

Arguments

csv_file

Name of the text file to convert.

sqlite_file

Name of the newly created sqlite file.

table_name

Name of the table to store the data table in the sqlite database.

pre_process_size

Number of lines to check the data types of the individual columns (default 1000).

chunk_size

Number of lines to read for each chunk (default 50000).

show_progress_bar

Show progress bar (default TRUE).

...

Further arguments to be passed to read_delim.

delim

Text file delimiter (default ",").

Value

a SQLite database

Remark

The callback argument in the read_delim_chunked function call refers to the custom written callback function `append_to_sqlite` applied to each chunk.