The Semantic Spreadsheet

Retrofitting semantics onto an existing symbolic systems is problematic, but it's probably unavoidable. Almost all organizational knowledge already exists in symbolic form. Electronic spreadsheets are symbolic systems.  Relational databases are symbolic systems.  Data warehouses are symbolic systems.

To understand the different types of systems, please have a look at this blog post, Five Types of Systems.

Tableau has added a semantic layer. Microsoft Excel can now get a semantic model via Power BI (and this). Microsoft Fabric IQ allows for the creation of a semantic layer. Databricks gets a semantic model from Unity Catalog. You can add a semantic layer to Google Sheets. Everybody and their brother is providing a semantic layer these days.

The real challenge is not adding the semantics with a semantic layer. The challenge is recovering semantics that were never explicitly captured in the first place.  All that data in those spreadsheets and databases never captured the semantics. Ironically, symbolic systems frequently succeed because they allow ambiguity. The ambiguity is hidden. But now all this ambiguity will be brought into the open.

Another big challenge. For the past 50 years business professionals have been taught to think in terms of "tables".  Databases have tables that information is queried from, spreadsheets have what are thought of as tables.  The problem is that the meaning, those semantics, are implicitly buried in the representation of the table.

Now, if everyone building a spreadsheet built the spreadsheet using the exact same terms and the exact same structures this would be less of a problem.  However, business professionals build spreadsheets and technical professionals build databases using arbitrary terms and arbitrary structures.

And now we want artificial intelligence to understand those spreadsheets and databases. But all those arbitrary terms and arbitrary structures make this problematic.  And even if the terms and structures are made consistent; you still have to add that semantic layer.

And so, "everybody and their brother" is building a semantic layer. They're not really trying to improve reporting. They're trying to make legacy systems intelligible to AI.

But working with a semantic layer can be challenging for business professionals taught to think in terms of tables.  Working with these semantics in this semantic layer needs to be made easy enough for a business professional to get their heads around. This can be challenging.

Is there another way?

The notion of The Semantic Spreadsheet is not new and it is not my idea.  The way I propose that this be done in this blog post, Status Quo -||- Augmented Symbolic -||- Grounded, Model Driven?, I propose three alternatives:

• Status quo: That will not work because we cannot leverage AI.
• Augmented symbolic: That can work, but that involves building that semantic layer and it will involve fixing all that data in the spreadsheets and databases that hold all that data.
• Grounded,  Model Driven: That can work, but it involves building software that assists business professionals in creating the semantics correctly, for fixing the existing data, and for projecting the semantic system in a manner that is approachable to business professionals.

The situation is this: you cannot fix a bad foundation.  That is what we have, a bad foundation. There are many problems with the current foundation, not just that it does not have a semantic layer. The biggest problem is really the "document" nature of the foundation. As Georg Philip Krog pointed out, the document is a "costume" the information that we need to work with wears. We need to think beyond the document.

As Himanshu Gaurav explains in this article under the section "The bigger idea" where he says, "Maybe the better goal is to remove the limitations that made spreadsheets problematic in the first place." Now, I don't agree with all the items in his list but I do agree with some.  I have started with Gaurav's list and added some additional items:

• Keep the interface. I agree that the spreadsheet paradigm is easy for the average business professional to relate to.
• Keep the appropriate amount of flexibility, but institute a capability to also apply control. The biggest strength of the spreadsheet, that flexibility, is also its biggest weakness. Users don't need total flexibility, they need a precise set of flexibility.
• Keep the formulas, I agree with that, but there needs to be a capability to separate the formulas from the information and share those formulas across many spreadsheets.
• Keep the modelling; yes, but model the information, not model the way you want to visualize that information in ONE DOCUMENT because someone else will want to use that same information in different ways.
• Make the information global, not local to the one spreadsheet; people don't seem to grasp the problem of  taking data out of one system and trying to use that data in another system.
• Use a global open industry standard model; don't lock business professionals into one software vendor; get Microsoft, Tableau, Google, Databricks, and others to all use the same logical conceptualization (model meaning goes into) and a handful of representations (physical syntax). See the Triangle of Meaning. See XBRL Semantic Model. See Reconciling Seattle Method, OIM,  SBRM, XBRL.
• Provide a global open industry standard intelligence engine; while some use cases are not deterministic, there are absolutely other use cases that are deterministic and therefore must be addressed. See Essence of Accounting.
• Make the solution industrial strength; you cannot scale the current spreadsheet paradigm, people have tried for years; use the proper foundation and build something industrial strength; the traditional electronic spreadsheet will be used for years to come, but we need a solution for all those operational spreadsheets, mission critical spreadsheets, shadow spreadsheets.  See, Connecting Industrial Engineering, Informatics, Pragmatic Knowledge Management Systems
• Semantic spreadsheets will fail loudly; those annoying errors that are hard to find can be very problematic; semantic spreadsheets will fail loudly.
• Enable the creation of other capabilities such as provenance, traceability, trackability, Git type versioning, etc.; the possibilities are endless...if the right foundation is established.
• Work with information rather than data; we can make all this work by separating the "data" layer and let the IT people deal with that and the "information" layer and enable business professionals to work with that.  See Atomic Design Methodology.

Another term for the semantic spreadsheet is the logic spreadsheet. Another is deductive spreadsheet.  I will use the term semantic spreadsheet. The semantic spreadsheet should be based on the pure relational model and be a pure logic system. Mixing the model causes big problems in terms of debugging software code, maintaining software code, and the maintenance of business rules.

Everything is relative. A spreadsheet feels simpler than a graph; and that is true. But that simplicity comes from limited expressive power of the spreadsheet. A graph can represent far more than a spreadsheet ever could.

“Simple” and “simplistic” are not the same. Simple means making something easy to use. Simplistic means stripping away essential complexity and pretending it is not there.

But you cannot make complexity disappear simply by ignoring it. That complexity just moves. Approaches like the Atomic Design Methodology help one move complexity. Think digital information Legos.

A graph is richer than a spreadsheet in all but the most trivial use cases. But a graph is also more complex; and that complexity exists for a reason. A graph carries additional information, structure, and metadata. A graph makes explicit the assumptions that spreadsheets leave unstated.

Simplicity tends to favor accountants who want to finish their task quickly, without considering how their work affects the next person in the chain of work. Richness favors accountants who want accuracy, transparency, and accountability across the entire chain of work.

A “shortcut” is a matter of perspective. When viewed holistically, the graph; though slightly more complex at the start; makes the full workflow dramatically easier for the system as a whole. Today accountants have new work partners in their chain of work, artificial intelligence.  Be conscious of those shortcuts. They might not really be a shortcut all things considered. If something is not represented explicitly; it will need to be implied by artificial intelligence.

The world's first spreadsheet was written on a clay tablet. The clay tablet evolved to papyrus; then from papyrus to linen; then linen to paper; then paper to e-paper.

Additional Information:

• The Enterprise Has a Hidden Operating System
• Using logic programming for theory representation and scientific inference
• The Deductive Spreadsheet
• Five Costs and Perils of Using Spreadsheets for Business Analytics
• AI-First, Digital-First, Graph-First
• The Birth of Spreadsheets: Humanity’s Long Effort to Organize the World in Tables
• A data table thousands of years old
• Data on clay
• 5,000 Years Ago, We Invented Writing… To Count Sheep
• Universal Global Standard Logical Spreadsheet (Professional Spreadsheet)
• Pragmatic Knowledge Management System (PKMS)
• Accounting & Audit by Design (A&AD) Framework
• Logic Programming and Theories