Organizing information in the AI age:  plus ça change, plus c'est la même chose

After AI first emerged in the second half of the 20th century, it depended on explicit handcrafted human knowledge represented symbolically as definitions and rules because computers needed help in understanding meaning.  Subsequent breakthroughs in computational power and statistical learning enabled AI systems to outperform semantic representations created by humans for search and classification, their dominant uses at the time.  More recently, the remarkable capabilities of large language models spawned the belief that “scaling” mattered the most; vast quantities of unstructured and unorganized data were sufficient to discover the statistical regularities that enabled plausible response generation.

But this assumption turned out to be mistaken.  Today’s AI systems attempt to produce knowledge, explain relationships, and use context to adapt their responses.  Models that lack explicit semantic infrastructure spend unnecessary computation processing duplicated, contradictory, or irrelevant information that all increase the possibility of error while bloating the cost.  Hitting this “economic wall” now that the cost of models is increasingly based on token counts has rapidly led to efforts to make information “AI-ready” in ways that are familiar to information architects and knowledge management specialists. These include controlled description vocabularies, identifiers, taxonomies, ontologies, and provenance records.  It is a bit amusing to see these concepts being rebranded by AI as “semantic compression,” “ontology-grounded retrieval-augmented generation” “context engineering,” and other neologisms because the field ignored the need for information organization and semantic infrastructure for so long.

‍