make_chunks(file_name: str)

What it does

  1. Memory-maps the entire file.
  2. Divides the file into N chunks (where N = CPU cores).
  3. Adjusts each end to align with the next \\n (newline).

Why

Important note


agg_initializer()

What it does

Why


process_chunk(start, end, queue)

What it does

  1. Takes a slice mm[start:end] of the file.