CSV example files and ingest tool flags

This page provides examples of CSV files and CSV ingest tool flags that you can use to test the system.

Table of contents

Before you begin
CSV source files
- header-defined.csv
- header-undefined.csv
CSV ingest tool setup
- CSV ingest tool flags for header-defined.csv
- Connect securely over TLS and define header flags

Before you begin

CSV source files

header-defined.csv

This CSV file defines the header row as follows:

Field	Data type	Additional
`username`	`String`	This will be used as the primary key
`salary`	`Int`
`created_at`	`Timestamp`	Timestamp field with 1 second granularity and parse dates with this layout 2006-01-02T15:04
`balance`	`Decimal`	Scale of 2 (store two decimal places)
`account_types`	`StringArray`	Values are comma separated list enclosed in double quotes

username__String,salary__Int,created_at__Timestamp_s_2006-01-02T15:04,balance__Decimal_2,account_types__StringArray
erin,85000,2019-05-22T13:44,334.43,"checking,savings"
will,90000,2019-05-22T13:44,111.13,"savings"
john,35000,2019-05-22T13:44,999.22,"savings"
paul,50000,2019-05-22T13:44,444.53,"checking,savings"
mary,40000,2019-05-22T13:44,555.63,"checking,savings"
jack,95000,2019-05-22T13:44,888.73,"savings"
dogg,85000,2019-05-22T13:44,777.83,"checking,savings"
zoom,15000,2019-05-22T13:44,222.93,"checking"

header-undefined.csv

Header data types need to be defined by the ingest tool.

asset_tag,fan_time,fan_val
ABCD,2019-01-30,40%
BEDF,2019-01-08,10%
BEDF,2019-01-08,20%
ABCD,2019-01-04,30%

CSV ingest tool setup

CSV ingest tool flags for header-defined.csv

The required header is defined in the source file

./molecula-consumer-csv \
    --batch-size=10000 \
    --primary-key-fields=username \
    --index=users \
    --files=header-defined.csv

Connect securely over TLS and define header flags

Use this method to:

ignore CSV headers and define them at the command line
define FeatureBase server tls certificates to securely connect to a remote server.

./molecula-consumer-csv \
    --featurebase-hosts=https://localhost:10101
    --tls.certificate=featurebase.local.crt \
    --tls.key=featurebase.local.key \
    --tls.skip-verify \
    --batch-size=10000 \
    --auto-generate \
    --header=asset_tag__String,fan_time__RecordTime_2006-01-02,fan_val__String_F_YMD \
    --ignore-header
    --index=csv-ingest-tls \
    --files=header-defined.csv,header-undefined.csv \