Ada 4.1.1
Fast spec-compliant URL parser
Loading...
Searching...
No Matches
pack_tables Namespace Reference

Functions

dict[str, Any] _parse_blob_meta (str text)
dict[str, list[int]] load_blob_sections (Path path=BLOB_PATH)
None write_blob (dict[str, Any] sections, Path path=BLOB_PATH)
None write_mapping_constants (*, str version, int block_bits, int block_size, int block_mask, int sentinel_valid, int sentinel_disallowed, int ignored_idx, int bool_flag, int low_range_end, int high_ignored_start, int high_ignored_end, list[int] stage1, list[int] mixed_data, list[int] bool_words, list[int] utf8_table, Path path=MAPPING_CPP)
None write_id_tables_stub (*, str version, list[list[int]] id_continue, list[list[int]] id_start, Path path=ID_CPP)
None update_mapping_in_blob (list[int] stage1, list[int] mixed_data, list[int] bool_words, list[int] utf8_table)
None update_id_in_blob (list[list[int]] id_continue, list[list[int]] id_start)
int main (list[str]|None argv=None)

Variables

 ROOT = Path(__file__).resolve().parents[1]
str BLOB_PATH = ROOT / "src" / "table_blob.inc"
str MAPPING_CPP = ROOT / "src" / "mapping_tables.cpp"
str ID_CPP = ROOT / "src" / "id_tables.cpp"
list SECTION_ORDER
dict ALIGN = {"u8": 1, "u16": 2, "u32": 4, "u64": 8}
dict PACK = {"u8": "B", "u16": "H", "u32": "I", "u64": "Q"}
dict WIDTH = {"u8": 1, "u16": 2, "u32": 4, "u64": 8}

Detailed Description

Pack IDNA/Unicode tables into a raw-DEFLATE blob (src/table_blob.inc).

Runtime tables are stored compressed and expanded once by ensure_tables()
(see src/table_store.hpp). This script is the only writer of table_blob.inc.

Typical workflows
-----------------
Regenerate mapping tables from Unicode (IdnaMappingTable.txt) and repack:

  python3 scripts/idna_table.py --write

Regenerate identifier tables from DerivedCoreProperties.txt and repack:

  python3 scripts/derived_table.py --write

Regenerate normalization and validity (bidi/combining/joining) tables from
UnicodeData.txt, CompositionExclusions.txt and DerivedJoiningType.txt and
repack:

  python3 scripts/unicode_data_table.py --write

Keep every generator on the same Unicode version as IdnaMappingTable.txt.

Repack using whatever is already in the blob (no-op rebuild / verify):

  python3 scripts/pack_tables.py

Update only some sections programmatically:

  from pack_tables import load_blob_sections, write_blob, write_mapping_constants
  sections = load_blob_sections()
  sections['idna_stage1'] = (...)  # list of ints
  ...
  write_blob(sections)
  write_mapping_constants(...)

Section layout (all present in the blob)
----------------------------------------
  Mapping:     idna_stage1, idna_stage2, idna_bool_blocks, idna_utf8_mappings
  Norm:        decomposition_*, ccc_*, composition_*
  Identifier:  id_continue_flat, id_start_flat
  Validity:    dir_start, dir_final, dir_value, combining_flat,
               joining_start, joining_final, joining_value

load_blob_sections() skips sections that the current blob does not have yet,
so a generator can add a new section; write_blob() requires every section.

Function Documentation

◆ _parse_blob_meta()

dict[str, Any] _parse_blob_meta ( str text)
protected

Definition at line 93 of file pack_tables.py.

Referenced by load_blob_sections().

◆ load_blob_sections()

dict[str, list[int]] load_blob_sections ( Path path = BLOB_PATH)
Load every table section from the current compressed blob.

Definition at line 121 of file pack_tables.py.

References _parse_blob_meta().

Referenced by main(), update_id_in_blob(), unicode_data_table.update_in_blob(), and update_mapping_in_blob().

◆ main()

int main ( list[str] | None argv = None)

Definition at line 366 of file pack_tables.py.

References load_blob_sections(), main(), and write_blob().

Referenced by main().

◆ update_id_in_blob()

None update_id_in_blob ( list[list[int]] id_continue,
list[list[int]] id_start )

Definition at line 351 of file pack_tables.py.

References load_blob_sections(), and write_blob().

Referenced by derived_table.write_derived().

◆ update_mapping_in_blob()

None update_mapping_in_blob ( list[int] stage1,
list[int] mixed_data,
list[int] bool_words,
list[int] utf8_table )

Definition at line 337 of file pack_tables.py.

References load_blob_sections(), and write_blob().

Referenced by idna_table.write_idna().

◆ write_blob()

None write_blob ( dict[str, Any] sections,
Path path = BLOB_PATH )
Write sections (name -> list[int]) as src/table_blob.inc.

Definition at line 144 of file pack_tables.py.

Referenced by main(), update_id_in_blob(), unicode_data_table.update_in_blob(), and update_mapping_in_blob().

◆ write_id_tables_stub()

None write_id_tables_stub ( * ,
str version,
list[list[int]] id_continue,
list[list[int]] id_start,
Path path = ID_CPP )
Write slim src/id_tables.cpp (arrays live in the blob).

Definition at line 307 of file pack_tables.py.

Referenced by derived_table.write_derived().

◆ write_mapping_constants()

None write_mapping_constants ( * ,
str version,
int block_bits,
int block_size,
int block_mask,
int sentinel_valid,
int sentinel_disallowed,
int ignored_idx,
int bool_flag,
int low_range_end,
int high_ignored_start,
int high_ignored_end,
list[int] stage1,
list[int] mixed_data,
list[int] bool_words,
list[int] utf8_table,
Path path = MAPPING_CPP )
Write constants-only src/mapping_tables.cpp (arrays live in the blob).

Definition at line 233 of file pack_tables.py.

Referenced by idna_table.write_idna().

Variable Documentation

◆ ALIGN

dict pack_tables.ALIGN = {"u8": 1, "u16": 2, "u32": 4, "u64": 8}

Definition at line 88 of file pack_tables.py.

◆ BLOB_PATH

str pack_tables.BLOB_PATH = ROOT / "src" / "table_blob.inc"

Definition at line 59 of file pack_tables.py.

◆ ID_CPP

str pack_tables.ID_CPP = ROOT / "src" / "id_tables.cpp"

Definition at line 61 of file pack_tables.py.

◆ MAPPING_CPP

str pack_tables.MAPPING_CPP = ROOT / "src" / "mapping_tables.cpp"

Definition at line 60 of file pack_tables.py.

◆ PACK

dict pack_tables.PACK = {"u8": "B", "u16": "H", "u32": "I", "u64": "Q"}

Definition at line 89 of file pack_tables.py.

◆ ROOT

pack_tables.ROOT = Path(__file__).resolve().parents[1]

Definition at line 58 of file pack_tables.py.

◆ SECTION_ORDER

list pack_tables.SECTION_ORDER
Initial value:
= [
("idna_stage1", "u16"),
("idna_stage2", "u16"),
("idna_bool_blocks", "u64"),
("idna_utf8_mappings", "u8"),
("decomposition_index", "u8"),
("decomposition_block", "u16"),
("decomposition_data", "u32"),
("ccc_index", "u8"),
("ccc_block", "u8"),
("composition_index", "u8"),
("composition_block", "u16"),
("composition_data", "u32"),
("id_continue_flat", "u32"),
("id_start_flat", "u32"),
("dir_start", "u32"),
("dir_final", "u32"),
("dir_value", "u8"),
("combining_flat", "u32"),
("joining_start", "u32"),
("joining_final", "u32"),
("joining_value", "u8"),
]

Definition at line 64 of file pack_tables.py.

◆ WIDTH

dict pack_tables.WIDTH = {"u8": 1, "u16": 2, "u32": 4, "u64": 8}

Definition at line 90 of file pack_tables.py.