mmap

How it works:

• mmap (memory map) lets you map a file’s bytes directly into memory, creating a view on the file as if it were a byte array.
• The operating system handles reading data into memory pages on demand.
• No explicit read() or string allocations for each line are needed.

Why it’s used here:

• Instead of reading line by line and allocating Python strings, the entire file is mapped once.
• Each process can slice a chunk (mm[start:end]) without extra memory overhead.
• Very fast: zero-copy slices, handled by the OS.

fork

How it works:

• On Unix (Linux, macOS), multiprocessing defaults to using fork, which means the child processes inherit memory from the parent.
• The mmap object in mm is directly accessible to all child processes without duplication.

Why it’s used here:

• The big advantage is that the large memory-mapped file is not copied.

• All processes can read from the same mmap without extra RAM.

• This is why the script explicitly does:

  mp.set_start_method('fork', force=True)
  

  to ensure fork (important on macOS or future-proofing on systems that might default to spawn).

multiprocessing