Lucidity is a new way of understanding what programs are doing as they execute.

I've been working on it off and on since 2014, but it's looking like I won't be able to continue. So, I'm sharing what I have and telling the story of the project's origins.


At Lucidity's core is a flexible way of mapping data into interactive representations of:

Maps
Trees
Lists
Graphs
Tables
It works like:
  1. Tell Lucidity what to monitor
  2. Run your program
  3. Use Lucidity to observe the changing (internal) state of your program

Here's a demo:

This video is an overview of features, but there's another video below demonstrating practical usage.

Part of why I'm posting this is to meet new people and to get feedback on what to do with the project. See What's next / get in touch for details. And I'm looking for work!


I.
II.
III.
IV.
V.
VI.

I. What is Lucidity and what is it good for?

Concretely:

How Lucidity would be used in ordinary, real-world software development:

In my latest project (Disk Atlas) I ran into a situation where I really wanted to use Lucidity—but it's not ready for use with JavaScript projects yet. So, I ended up rebuilding a small subset of Lucidity's features into Disk Atlas to use as a debugger. Here is a video that shows how the "mini-Lucidity" worked with Disk Atlas, and how its features correspond with Lucidity's:

Fullscreen pretty much necessary

In the past I've also used Lucidity-like visualizations for more complex data structures. Here's a dynamic tree visualization I used to show operations being performed on the AST of a structure editor I was building.

debugger screenshot
Click for animation

My sister and I started building a "code tutor" app using Lucidity. The idea was for it to contain lesson sequences connected to a live coding environment where Lucidity would auto-visualize the behavior of the user's code. This way, instead of just giving the binary feedback 'correct/incorrect', the user could see exactly what their code was doing to the data it operated on.

excuse the lorem ipsum

Analogously:

Lucidity is a pretty unique piece of software, but some of its motivations and approaches can been seen in existing, widely used software. For example, consider the Chrome or Firefox dev tools, and the console.log / console.table features they provide for 'printing' tree widgets and table views, representing JavaScript objects and tabular data. Much like Lucidity: they allow users to inspect program data rendered in a manner that reflects the data's structure.

Chrome object explorer
Chrome console.log object explorer

Firefox console.table
Firefox console.table

Another angle: the core is like a Graphviz that instead of static images produces interactive/explorable widgets capable of conveying time-evolution of the data structures being modeled.

On the educational side, Lucidity could be used in building software like: http://pythontutor.com/

Distinguishingly:

Other tools exist that let us analyze program state—so what makes Lucidity unique?

1. Comprehensible time-evolution. Many tools work well for exploring snapshots of data, but not for understanding what's happening to the data in time. Lucidity lets you passively watch data changing in time, with clear indications not only of primitive value changes (which might be rendered with e.g. line graphs), but of changes in relationships between values or even between entire data structures.

2. Passivity. Task switching in humans is costly. Stepping debuggers, for example, are very costly in this regard; println statements fare better. Most of the time we just need a quick glance at some values we're modifying, and do not want to introduce a new problem of controlling a separate system. Lucidity's aim, for most usecases, is to require the level of involvement of println statements (if not less): I imagine it always running on the side while developers test their programs, only occasionally needing to actually pause or step through operations (and those controls are dead simple: press spacebar or left/right arrow keys).

3. Representation of structure. When we work with data as programmers, that data has some kind of structure to it. With very few exceptions, contemporary tools for viewing program data offer no means of viewing its structure. Instead we're forced to lose the forest for the trees with stepping debuggers, or to get creative with println formatting—or to rely solely on our imaginations.

4. Universality: i.e. loose coupling with programming language / environment. The bulk of the program's features have no tie to any particular language or environment. In fact, it could even be used to monitor a wide range of systems outside the context of programming languages (e.g. anything usefully modeled by some kind of graph + time-evolution, e.g. FSMs).

Statefully :)

Program state isn't evil, it's just incomprehensible without clear feedback on what's happening to it. There are problem domains where strictly ruling out state may be optimal—but it's far from all of them.

Rather than getting rid of state, let people understand the state they're working with.

If state is going to be there—as I think it should be in probably most cases—it seems to me like a significant concern to figure out better how to work with it. I believe Lucidity—or at least the concept behind it—is a meaningful step in that direction.

Metaphorically:

The intention here is to illustrate a specific type of feedback in software development tools which Lucidity aims to improve: there is increasingly well-known value in improving feedback loops that clarify the relationship between program source and final program output; Lucidity's approach is to provide continual, low-effort feedback on the relationship between source and the actual computations taking place (in terms of how they affect data). Its power both as a debugger and as an instructional tool derives from this.

As a metaphor for ordinary programming, imagine a setup where two people work together to build a castle out of legos. Jill comes up with instructions for how to assemble it, Jack follows the instructions in a separate room with no windows. So Jill hands off the instructions, then after a while Jack returns with the lego creation that resulted from following them.

As programmers will intuit, it would require many iterations of this before a valid set of instructions were arrived at.

The assertion implied by the Lucidity concept is: if Jill were able to watch Jack following the instructions and assembling the legos, their ability to effectively improve the instructions at each iteration would be significantly increased.

The flawed output itself carries less information than the full set of specific errors and violated expectations that gave rise to it.

Theoretically:

Computer programs operate on data, but computer programmers generally cannot see that data being operated on. We are in need of an instrument like a microscope: we need something that automatically transforms the object of our observations into something which human minds and senses can actually use. The transformation required in software development is not magnification, however. Our problem is that we have too much data, too rapidly evolving—and we need ways of automatically eliminating less relevant subsets of that data, retaining only the most essential parts.

The name I've used to refer to this type of instrument is an 'abstractoscope'. Lucidity is one example of an abstractoscope—even if only a primordial one.

II. The dream (of what Lucidity would one day become)

[Short story where a programmer tries out a hypothetical future version of Lucidity, using it as a first step in learning the git codebase]

March, 2022

I sat down at my desk, excited for my first foray in contributing to a major open source project: the git version control system. I'd heard it was using some new technology to visualize high-level system behavior—a way of teaching new developers how the major parts worked together. And apparently their developers used it for debugging, too. I was intrigued.

I downloaded and launched the visualizer—and had no idea what to make of what I saw: it appeared to be a large assembly of molecular machines working together, maybe within a cell. I found the controls to zoom in and soon spotted the familiar shapes of graphs, trees, tables, lists, and maps—so it did have something to do with software.

I pressed pause and moved the cursor around to examine: names appeared over the objects in the scene, some of them familiar, some not. At the center was a large hashmap labeled "git object database", and I could see SHA values on the left (keys) and tree icons on the right (values). When I clicked one of the icons, a tree expanded outward from the hashmap: it was a commit object, mainly representing a repo's directory structure, but I could also see "commit message" among a host of unfamiliar properties.

Eventually I figured out I was interacting with a pre-made demo, meant to illustrate the basics of branches and commits, building up to some example 'merge' and 'rebase' operations.

But I wanted to see what happened to the data structures as I performed operations in git myself, so I switched into 'real-time' mode.

Now that it was reflecting an instance of git running locally, the data structures in the scene were less populated. I hadn’t paused it, but everything was still. I navigated my terminal to a provided sample repo, changed some files, and made a commit with the message, "test commit".

Exactly as I pressed ‘enter’ in the terminal, the visualized scene began to shift. The new commit popped into view as a tree with its message property reading, "test commit".

I noticed a text-field for filtering data structures from the scene; I typed 'stash', and all but a single stack data structure disappeared; the camera zoomed in on the stack automatically. I switched to the terminal and `git stash`-ed my changes a few times. With each stash operation a new tree was added to the stack in the scene.

I used the visualizer to move backwards and forwards in time, stepping through individual operations for branching, committing, and merging. I wanted to make sure I understood all the changes taking place. First I would focus on one data structure, then move on to another. I was beginning to get a feel for it.

Another command came to mind: `git rm --cached`. I would explore it next.

data structures in Lucidity

III. How I came to build it

In 2014, a year or so after wrapping up my Tiled Text project and moving to Berkeley (from Colorado where I grew up), I was working part-time for a small startup, and producing new and impractical ideas at a breakneck pace. Probably the most developed of those ideas, prior to Lucidity, was for a programming language I was impudently naming IA, for IntellAgent.

Lucidity came out of my determination that I would need it in order to build IA.

There's a lot that could be said on IA, but I'll sum it up as a project that came out of:

1. My Tiled Text project being well-received, reinforcing the idea that I might do something important after all and should try.
2. My spotty, eclectic knowledge of subjects in CS, overly lacking in connection with both AI and programming languages.
3. The fact that as a teenager I'd read Society of Mind and a couple others and tried e.g. writing a (Java :) program for AGI, and had since developed an ingrained habit of observing my own thinking for feedback on a handful of topics in cognition I'd been refining models for. I was very interested in how thinking works, and thought I might have some unique insight into it.
4. The fact I'd developed an RSI when I was 19 that prevented me from viewing computer programming as something I would reliably be able to do as a career, forcing me to explore a wider range of subjects. At this particular time I'd been absorbed in topics within philosophy and general linguistics, in part because of new friends I'd been making around Berkeley.

The result of all this was, unsurprisingly, a very ambitious and almost certainly doomed to fail concept, based in a heady mixture of ideas which could be individually categorized as a) interesting b) mistaken or c) already well-established (my own ignorance of them notwithstanding).


'Analogy' was to be the first of a number of built-in types used to bootstrap cognition using a set of innate relations (including 'relation' itself, of course), partially inspired by questionable interpretations of Kant and Lakoff & Johnson.

The idea behind that definition of analogy: Generalize and re-Particularize the original in such away that its structure stays the same, though the things related through the structure are allowed to change.

The two blocks are prefixed with `+` and `-`, because the one with a list of parameters adds things into the Type, whereas the second block's purpose is to remove things via constraints and other means. The first block provides the stone, the second carves it up.


I produced 44 pages of handwritten notes full of modeling trials, syntax experiments, general musing—and a fairly detailed design for the language's runtime, expressed primarily in terms of algorithms on a pile of... interesting data structures:


"Parse Operation" —IA was going to parse (and generate) things that weren't necessarily sequences, since that's just one type of relation.

"Per-Structure node FSM" —IIRC this was related to allowing more complex constraints on parameters to Types (e.g. quantification: a Chair must have >= 1 leg)

From prior experience and my own methodological preferences in writing programs (another example), what appeared to me as the main problem, after settling on a design for the runtime, was that I'd need a means of visualizing the dynamic state of a wide range of data structures.

It would be a monumental task to build each of these visualization utilities separately—but if I could create a general system that captured all their requirements, the problem might become managable. Lucidity was supposed to be that general system.

The image I had in mind for it bears a strong resemblence to the story from the previous section, though I'd be using it to get easy/rich feedback on my in-development algorithms rather than to learn a new system from scratch.


As for the IA project: an important realization for any ambitious intellectual working outside academic institutions is that the set of apparently beautiful ideas is much larger than the set of beautiful ideas of real value. It's easy to judge an idea as valuable because it has the marks and feel of something 'important,' before the critical step of practical verification has been completed.

I realized a couple years later (while reading "The Art of Prolog") that some of the ideas I liked best in my language were probably just poor approximations to Prolog. There is plenty unique about it still, but my bet on the odds of it having the significance I originally imagined is not optimistic.

IV. What I was going to do with it

Eventually Lucidity became more interesting in my mind than IA, partly because of the value I saw in the ideas behind it—but more significantly (to me), I saw that I could build a business around it and finally be able to pursue one of my projects with full focus, rather than it being a 'side' project.

So while my focus in Lucidity remained on the extremely general core, I began searching for a more specialized product I could build on top of it. The possibilities there seemed endless (I even submitted a poorly thought out application to YC at this time, motivated by the foregoing hopeful line of thinking, while the product-concept was still vague at best).

Some business/product ideas:

  • Dynamic database visualization: a way of easily seeing what's happening to your database as certain actions are performed in your app. It would optionally provide auto-assembled tree views using foreign key references, rather than needing to pieces things together across tables. This way you can easily see what's happening to some composite entity as the database is acted on. (I tried Asking HN about this. I don't have much experience with databases.)
  • 'code tutor' style app for early CS (possibly high school) students that gives algorithm/data structure problem sets and auto-visualizes the students' code. So when they, e.g., write a broken sorting algorithm, they can see what's actually happening to the list their code is acting on.
  • Quick method of building instructional algorithm material for presentations. Could be used in lectures, or e.g. youtubers describing algorithms in general, or for companies/individuals teaching others how their software works.
  • Something like Graphviz, but for data structures that evolve in time, or that contain too much data and need to be interactive/explorable.
  • Consulting on integrations into platform-style projects, which would alllow it to become available without additional integration work for anyone using the platform. For instance, if it were integrated into Unity or three.js, there could be a powerful, always available scenegraph debugging utility ('integrating' only involves the monitoring portion; visualizer communicates with monitors over TCP).
  • Specializing either as an FSM or tree visualizer, for monitoring any kinds of systems usefully modeled by those structures. As an example on the tree side: it would resemble the Chrome dev tools object explorer, but easier to get info at a glance, and much better at showing changes in time, and capable of being attached to new programming languages/environments—e.g. for Java or C++ development.

I also had in mind that after building some of these specialized products on top of Lucidity, Lucidity itself could be licensed to other companies who had products they wanted to build.


Independent of any business-related hopes for Lucidity, I retain a persistent curiousity to just see the thing built and functioning as originally envisioned. My most powerful motivation in building software comes from being in the position of a Dr. Frankenstein, anticipating the first steps and sentences of their creation. The software I want to spend my life working on is the type that inspires that kind of feeling.

There's also a more specific hypothesis about Lucidity that I remain very interested in testing, which I believe to be true: Lucidity could make the difference in whether certain sufficiently deep or complex software projects are feasible to construct or not.

Once I'd settled on a design for IA's runtime, my sense that I could realistically build it was "no" without Lucidity, "yes" with. I wonder how many other cases are out there where people would be enabled by Lucidity to raise the threshold on what they see as maximum feasible system complexity.

V. Struggles and potential abandonment

The biggest problem with Lucidity is that I thought it would be reasonable for me to turn it into a production ready piece of software on my own, using only the relatively little free time I had for it. Now that I've built a vastly simpler commercial software product, which is still not quite ready to sell, I'm aware of how ludicrous that idea was.

More recently I've been facing the reality that I'm not going to have enough time to finish it after all.

It's been about two years since I've been able to work on it. That time has been spent either working to support myself or trying to bootstrap my work on Lucidity using another project. Now I've got to face the fact that I'm completely out of resources and it will likely be a significant amount of time before I've earned the free time to focus on it again—and I'm not sure to what end. The amount of time I could scrape together would almost certainly still be inadequate: the project is too big for a single person to do in sporadic bits of free time.

So now what I'm looking at is cutting my losses, abandoning the project, and hoping it will at least be a useful source of ideas for other people's projects.

Open sourcing:

I don't know whether it would be worthwhile to open source it or not, and would like feedback on that. I can't afford to put much more time into it. I could write up some nice documentation, slap an MIT license on it, and that'd be about it. I guess I'd probably consult on some things now and again too—but there's no way I could provide consistent leadership without getting paid.


If I did have the resources to continue working on it, my first step would be prototyping the remaining difficult UI problems, and finding a better 'dogfooding' integration than what I have at the moment. Then I'd finish up a rough version of the general core, and make it available to communicate with through an IntelliJ and/or VSCode plugin or something, so people could start using it in some form right away. I'd collect feedback and refine the core. If it needed to earn money, I'd explore more particular product ideas to build on top of it in parallel. Otherwise, I'd just keep honing the core and working on language integration tech until the whole thing sang.

VI. What's next / get in touch

While working on Lucidity was a great experience in many ways, I don't think I'll be self-funding this kind of work again without establishing some relatively passive income first. I have hopes I'll be able to do that at some point, but for now: I'm at about my limit of scrambled egg dinners, and it'd be great to have health insurance again.

So now I'm ready for something completely different—and, I don't really know what to do with Lucidity.

My hope is to tap into the wider knowledge of the tech and academic community here and see if there's some place for Lucidity and/or myself that I'm overlooking. If something occurs to you—shoot me a message!

I'm happy to hear from anyone, but here are some things I already know I'm interested in:

  • People with complementary skillsets who might want to work with me either on:
    1. Building a business designed to optimize high likelihood of sucess over unicorn status, focused on developing something simple, high-quality, and non-exploitative. (Disk Atlas is my first attempt, and I could potentially use help there too, especially from any veterans of this class of software business.)
    2. Designing and building things based on outlandishly interesting and unexplored ideas—but with a practical bent: not daydreaming, but with geniune intent to build something new. Especially if you live in Tucson—because let's face it: you can't force "something new" to come about, so we might just be hanging out around town some meanwhile.
  • Part-time or contract work doing something related to interactive computer graphics or data visualization. It would be especially cool to take on some work making system behavior comprehensible/visible. Also happy to do pretty much whatever three.js work, or maybe even more vanilla web dev. Thinking of becoming a React specialist (but I'm not one now).
  • Anyone wanting to hire me to do research and build software prototypes in domains from programming tools to biology, manufacturing, robotics, CAD, VR/AR UI systems, etc.
  • People looking for a good generalist to help build some new interesting software and do various things that need doing around the business, probably in the context of a startup.
  • People who want to advise me about open-sourcing Lucidity.
  • Idealistic billionaires looking to improve the course of humanity by donating to idealistic philosopher-programmers.

Stay in touch:



Email newsletter:
Updates on Lucidity or other weird software I've made or will make. Maybe some thoughts on future programming tools. Frequency: sending out 4 - 6 per year would be ambitious for me.

powered by TinyLetter