In my profession - as in many, many others as well - knowledge is very valuable. Growing knowledge, as in continuous learning, is the common response. I don’t want to make that sound like a chore! For it is not! I love to learn. I love to read. I love to explore and experiment. There are few things that are more fun, in my opinion.
However, a part of that process is actually a chore, or at least not easy. That part is storing the learned knowledge in a way, that it is easily accessible in the future. Well, if you have perfect memory you can stop reading here, and forget you ever found this blog (pun intended). My memory is terrible. I often deal with complex systems, that have highly detailed information. I also engage with a broad range of knowledge domains. In short: I am in dire need of an artificial crutch, that helps me organize all the interesting nuggets of knowledge that I come across.
Now we are getting to the crux of this post. Such a tooling, to manage my own personal knowledge, has been a pet peeve of mine since I first used a computer as a tool for learning. I never found a good software that really fits my needs. What I came across in the past either had bad usability (makes me sad), was only available for some of my devices (a no-go), or forced me to give up ownership over the content (not helpful). Thanks, but no thanks.
For a long time I was bouncing in between tools. Creating data graves as I went. While I certainly did not research the topic continuously, I occasionally looked around and sometimes even put some time into research - without real success, to my great chagrin. Recently, I had some bigger life changes, which also left me with some time on my hand. So I took the opportunity and studied a lot. Hence my urge to find a tooling to preserve that new knowledge surfaced again - and I think I found something that actually checks all the boxes for me this time! Read on if you are interested.
What is personal knowledge?
I think I need to clarify what I mean with personal knowledge, or my solution for personal knowledge management may not make sense to you.
On the Wikipedia page about personal knowledge base (PKB) I found the following paragraph, that deeply resonates with me:
[…] It differs from a traditional database in that it contains subjective material particular to the owner, that others may not agree with nor care about. Importantly, a PKB consists primarily of knowledge, rather than information; in other words, it is not a collection of documents or other sources an individual has encountered, but rather an expression of the distilled knowledge the owner has extracted from those sources. […]
So in short: Personal knowledge is distilled information, that makes sense to the owner who extracted it from other sources.
It is, in a way, a personal shorthand. Information, retained in such a way, that when I read it again, it will make sense to me - but maybe not to someone else (or maybe it will, I frankly don’t care - however, I do for my blog posts).
What is personal knowledge management?
With the above understanding of personal knowledge, the management part means that there needs to be a process to build that knowledge and then make it usable. In practice that means there needs to be a tooling, that implements or at least supplements that process.
For the process, I started out with the Zettelkasten (German for: note box) method, which is in (very) short: an organization system with a prescribed method how to record and interlink knowledge that is written on notes (either analog or digital). I won’t go into much detail how that exactly works, as there are awesome tutorials to be found:
- https://zettelkasten.de/posts/overview/
- https://zenkit.com/en/blog/a-beginners-guide-to-the-zettelkasten-method/
- https://www.levelupfun.com/articles/writing-to-understand-the-zettelkasten-system-of-smart-notes
- search yourself
Also, I am not really following Zettelkasten (anymore). It didn’t work out for me, alas when trying to adhere strongly to it. For now I am using a “dirty derivate”, that is something in between Zettelkasten and a Wiki. If you roughly know how Wikipedia works, you are fine for the rest of this article. So far that system works out for me. Maybe I will come back to a more formalized method in the future. Maybe I will document my method and make it formalized. Maybe neither.
There are also two other documented methodologies I came across. None of them spoke to me, so, aside from superficial research and reading, I haven’t tried them out:
- GTD - Getting Things Done
- P.A.R.A. - seems to be bundled with proprietary software
Anyway. Whatever methods works out for you is the right one. I recomend to start with Zettelkasten and iterate from there, but that’s just because that was my journey.
What is a personal knowledge management tool?
Within the realm of software engineering, you usually start to answer a question about the properties a software needs to have by interviewing the intended users and extracting their requirements. The intended user is me, so that is easily done. However, I think I can put the requirements in a form that is much more universal and will make sense to you as well.
Here we go then. In my opinion, any personal knowledge management tool needs to address four topics:
- Storing personal knowledge, so that the knowledge can endure and grow over a long time
- Accessing personal knowledge, so that the knowledge is available when needed
- Browsing personal knowledge, so that the knowledge can be navigated and consumed easily
- Editing personal knowledge, so that building and maintaining the knowledge is easy
The order of the topics is of little consequence. This is just the way I thought of them. Let’s have a deeper look:
Storing personal knowledge
Storage is cheap these days. Every service, it seems to me, throws a gigazillion bytes of free storage at you. For personal knowledge management you actually don’t need much - comparatively speaking. We are talking mainly text files here. Maybe some odd diagrams, images or other, small binary formats. This won’t need terrabytes, not even gigabytes, even for very large knowledge databases.
Consider Google Keep, Apple Notes, Microsoft OneNote, Evernote and other “notes-as-a-service” offers. At a first glance, they seem to offer at least part of what a personal knowledge management tool would need. Also they are either free or available for a small coin. However, they - and all tools like them - are categorically unusable, in my opinion, for a simple reason: Storage is about ownership.
What does that mean? Any tool like the above allows you to store data within. However, getting the data out, so that you can put it into another tool, is a whole different story. Even if you can get it out - through begging, legal threats or dark magic - the data also needs to be in a format, so that the new tooling can understand it.
I think this is a very important point, so please allow me elaborate on that: Think of any software that you are using right now (that goes for anything, also outside of knowledge management). Chances are high that it won’t be around in five years - or if that, then what about ten years? Or twenty? We’re talking about your personal knowledge database here, that - ideally - is around as long as you are around.
There are a million reasons, why the software won’t be around: Maybe a competitor buys up the vendor, discontinues the software. Maybe the vendor decides to pivot and discontinues the software themselves. Maybe the vendor is a SaaS that suffers a fatal data-breach and is wiped of the map. Maybe the software is open source and the lone contributer gives up coding in favor of ice-skating. And so forth. I think you get the gist: Software is short-lived. Data is not. Software is just an interface. Data is the knowledge. Own your data, or lose your knowledge.
Excuse my ranting. I have been burned in the past. I usually don’t mind about storing data with the vendor. I use a lot of tools that are exclusively online and that do create data. However, for this specific use-case of a personal knowledge database, it’s immensely frustrating if you find your data unusable or inaccessible, after collecting it for a good while.
Requirements:
- I can access the raw data representation of my knowledge
- My knowledge is preserved in an open format
Accessing personal knowledge
Access is, of course, key. If I cannot access my knowledge, it has no use to me.
These days, using multiple devices is commonplace. Also switching devices is far from unheard of. And I am not only talking about getting the next model. I was using Android phones - until about two years ago, now I am using an iPhone. I might switch back next year. I was using exclusively Linux on my Deskop, until six months ago, now I am using a Mac. I might switch back again sometime. Maybe AR and VR becomes actually a part of our lives soon. Who knows what OS will power that?
While I would really like not to be bound to a specific vendor, hardware or subscription - at the very least: my personal knowledge needs to be accessible from all devices that I am currently using. Possibly more: I might need “some” access to my knowledge, even without my devices at hand. Let’s call that a nice-to-have.
Access from multiple devices also means that the data must be synchronized somehow in between the devices. Otherwise it becomes very hard to maintain at best and suffers actual data-corruption at worst. Wrong knowledge may be worse than no knowledge.
Requirements:
- My knowledge is accessible from all my devices
- My knowledge is synchronized between all my devices
Nice to have:
- My knowledge accessible without my devices (but then protected with authorization?)
Browsing personal knowledge
Assuming the knowledge is stored and accessible, then another important question arises: How can I navigate it, how can I interrogate my personal knowledge database in a convenient way? This must be really easy and intuitive, or all of the above is for naught.
To get to the requirements, let me spell out what I need to be able to do with the tooling. Basically: What are my user stories(‘ish) that describe how I will interact with the tooling when I am browsing my knowledge? Here we go:
- I have something that I want to add and need to know whether it’s already there (i.e. extend or create)
- I have a specific thing / term / word in mind, that I want to look up
- I have a broad issue / topic / subject in mind and want more info about it i.e. I want to see what I collected about that
- I already found something and now want to dig deeper
- I already found something, it’s not what I wanted, but close / related
Ok, that makes it easy to formulate the requirements. The first two come down to a base full text search. The third requires the knowledge to be categorized; actually things could be in multiple categories, so I’m gonna call that: taggable. Once they are that, they can be listed by tag - and tags can be listed as well. For the last two there need to be navigable links, like good ol’ hyperlinks in the interwebs.
Requirements:
- I can search my knowledge (full text)
- I can view all my knowledge in a topic (aka tag)
- I can follow links in between my knowledge
Editing personal knowledge
Last, because it references all topics, we need to talk about input of knowledge. Input is adding of new knowledge and of course also correcting or extending existing knowledge. Also deleting of wrong or re-factored knowledge contents must be possible. Let’s just extend that to the standard CRUD (create, read, update, delete) operations.
Looking at the previous topic about browsing, there is also linking and tagging that needs to be possile. It is equally important that I can create relations in between the knowledge. Aside from being required for navigation a lot of information is in the connection itself.
And finally, I want to be able to group knowledge. This is differnt from linking. More like categorizing
And all of the above needs to be doable in an easy way. Easy, I guess, is relative. What is easy to you, might be hard for me and vice versa. Hence we both might judge it differently. You will see below when I present the solution I found, that works pretty well for me.
Requirements:
- I can create, read, update and delete (CRUD) my knowledge
- I can tag my knowledge
- I can create links in between my knowledge
What is the solution then?
Having all the requirements spelled out, is the hard part. At least for me it becomes much clearer what a solution needs to look like.
So without further ado: I am using Obsidian, which styles itself as a second brain and is basically a text editor with Markdown syntax highlighting and rendering support. It provides a text search and various visualizations of the contents.
There are native apps for the usual mobile and desktop operating systems (Android, iPhone, Linux, Mac, Windows).
It’s free for personal use. On top of that, it has a large amount of community provided plugins, that add more visualization, help integrate with 3rd party and much more.
To be clear: I am in no way, shape or form involved with the developers or company. I am just speaking from my own experience over the last three or so months.
If that peaked your interest, here are the most noteworthy getting started guides and tutorials I came across:
- Marc Littlemore: A beginner’s guide to note taking using Obsidian - a quick overview, good intro
- Productive Engineer: Beginners Guide to Obsidian - visual explanation of the user interface
- Youtube for audio-visual learners - especially the ones from Martin Adams are very detailed
Or you just get started, as I mostly did, and see where that leads you. It’s quite intuitive.
Also, because I came up with that requirement list and feel now obligated to do the visual check:
| Requirement | Fulfilled | Solution |
|---|---|---|
| access to raw data | ✔ | Stores files on local disk |
| open format | ✔ | Markdown |
| access from all devices | ✔ | iOS & MacOS App (as well as Android, Windows, Linux) |
| synchronized betwee devices | ✔ | iCloud or Git(hub) or paid sync |
| full text search | ✔ | search text files |
| add tags | ✔ | front matter |
| view tagged | ✔ | community plugin |
| knowledge links | ✔ | internal link format |
| CRUD | ✔ | text editor |
As you can tell I am really happy with Obsidian (so far). Especially the easy access to my files in a format that I can re-use is appreciated. Even if Obsidian should not work out for any reason, I will still have all my knowledge stored in universally usable Markdown files. That gives me confidence that I will not create another data grave again and motivates me to put in the time to document what I learn.
The power of open formats (Easteregg)
Apropos universally usable. You might have noted that the nice-to-have about access without any of my devices did not show up in the above table. And yes, Obsidian cannot help much here. However, I am using iCloud to sync my files between my phone and laptop. I also use the Git plugin to automatically create regular backups to a (private) Github repository. Hence, I can access my knowledge without my devices, as long as I have Github access.
Even more so. This blog you are reading, is published via Hugo, which is an open source static site generator that can be easily used for blogging. Coincidently, my blog pages are generated from Markdown documents that I write.
So I figured: why not make (some part of) my knowledge database available to me and potentially others through my personal homepage? This way I can can link to entries from other publications and especially have it accessible from anywhere. It might also do good for my *cough* otherwise non-existing SEO efforts.
So I thought and so I did. Here you can find an excerpt of my personal knowledge database: https://ulrichkautz.com/zettel.
And here is the tooling that exports (and slightly converts) the Markdown files from my local Obsidian directory to my local Hugo contents directory: https://github.com/ukautz/obsidian-meets-hugo.
I really am happy how I can just re-use my knowledge database, now that I truly own the data that makes it up.
Alternatives to Obsidian
While researching the matter I also came across other tools, which did not fit my needs as nicely as Obsidian did. However, I might get back to them in the future and they might work for you now:
- Joplin - Open Source, sync via Dropbox / OneDrive / Self-hosted / as-a-service, Markdown, very active community - not for me, because not using Dropbox/OneDrive and dislike SQLite local storage
- Zettlr - Open Source, only desktop, Markdown, active community - not for me, because no mobile support (nor sync tooling)
- 1Writer - Paid, Markdown, Mac-only, sync via Dropbox / WebDAV, beautifully simple - not for me since not using Dropbox
- Update (2024-01-18) Acreom - Limited free for personal use, looks fantastic, desktop & mobile & web, focus on developers and tasks, many integrations with typical OSS & tooling - not for me since I currently work in proprietary environment
And some mentions that I looked into, but that are unusable for me as a personal knowledge management tooling due to the data being not or only very hard to extract:
- Apple Notes - Same use-case as Google Keep for me
- Evernote - Used in the past for same as Keep
- Google Keep - I use if for shopping lists, cooking recipes, ..
- Microsoft OneNote - Used at work for meeting notes and such
- Notion - good usability
- Roam - looks nice, pay-to-try
- SimpleNote - web only
Fin
Hope this helps someone else to setup their own personal knowledge management. That’s it for now. Cheers from London.
