Why doesn't Emacs' Org Mode just use Markdown? (2021)
About a year ago, someone in the Emacs subreddit asked why Org mode invented its own syntax instead of "just using markdown". This was my reply. It's not 100% accurate, but I did a bit of research to generate it, so I thought I'd reproduce it here.
Org and Markdown were created roughly concurrently, so "just going with markdown" wasn't an option, since markdown didn't exist. But they are both structured text formats, which makes them seem similar enough that one might be able to use one where the other is used. So I think I get where your question is coming from. In fact, I wished for "markdown everywhere" just a few years ago, but after struggling with achieving in Markdown what I could do in org mode, as well as struggling to make Tiddlywiki as powerful with Markdown as it is with Wikitext, I realized the truth: Markdown as originally published was incomplete for many of these use cases, and never quite converged enough to enable the tooling that other formats enjoy. So I bit the bullet and got really comfortable with all three formats.
At its heart, Markdown is a lightweight markup that was intended to be read as easily in text form as in the rendered HTML form. As a result, it falls back to HTML for anything tough (tables come to mind, but there are lots of others), which makes Markdown prefer HTML as an output format over any other. Reading the markdown page over on Daring Fireball, it's clear how HTML-oriented markdown is:
Markdown is a text-to-HTML conversion tool for web writers. Markdown allows you to write using an easy-to-read, easy-to-write plain text format, then convert it to structurally valid XHTML (or HTML).Thus, “Markdown” is two things: (1) a plain text formatting syntax; and (2) a software tool, written in Perl, that converts the plain text formatting to HTML.
The second sentence of the intro is an extremely important distinction to consider when answering your question: both org and Markdown are both formats as well as tooling. I think this lies at the heart of your question, which I'll come back to in a minute. The point here is that in contrast to Markdown, org-mode was not overly coupled to HTML, instead written to be able to produce LaTeX, HTML, and plain text output.
Org tooling vastly outstripped Markdown tooling almost immediately. Because org was defined in Emacs, having robust tooling to work with it was very high ROI, as Emacs makes writing tools to work with text files really easy. Markdown, in contrast, has suffered fits and starts because the tooling was inadequate, and the specification was incomplete. As the Commonmark page outlines:
John Gruber’s canonical description of Markdown’s syntax does not specify the syntax unambiguously. In the absence of a spec, early implementers consulted the original Markdown.pl code to resolve these ambiguities. But Markdown.pl was quite buggy, and gave manifestly bad results in many cases, so it was not a satisfactory replacement for a spec. Markdown.pl was last updated December 17th, 2004.
Since Markdown relies heavily on HTML for basics, a host of "flavors" emerged (36 so far, according to Commonmark's Wiki). This diaspora of flavors made creating widely-adopted tooling difficult, beyond the basic syntax in the original markdown targeting HTML.
So while Markdown was figuring out what syntax to use across the whole community for stuff like tables and super/subscript, the org community was building tooling in Emacs to bring RSS feeds into org, develop webclippers to capture in org-mode, implement mind-mapping in org, and bring bookmarks and GTD to org mode, all through Emacs-based tooling. "One format" enabled this.
So at this point in the story, we have two ecosystems: a large, diverse group of companies, products, and users that leverage varying types of markdown, and a smaller, more coordinated group using a highly-consistent syntax and building extensive tooling around it. It would be a monumental effort to rewrite all that tooling against markdown. So the org-mode folks get ever-more entrenched because of all the benefits, and as others see those benefits, they wonder why they don't have them on the Markdown side...hence a desire for all products to "just use markdown". I think tools like Obsidian will eventually bring many of org-mode's benefits to the markdown community, and have already. I'm not an Obsidian user (not open source), but I've heard from many happy users.
Edit: since I wrote this last year, I've come to realize that the benefit of Obsidian most folks appreciate is that even if Obsidian disappears, all the notes are "just markdown". I think this is a mirage, since to achieve all the extra goodness Obsidian offers, markdown will need to be extended in ways other tools won't support. It's possible that Obsidian will have such a large following that its gravity causes these extensions to be standardized, but I'm not betting on it.
Coming back to the syntax-vs-tooling point I mentioned above. In several threads, there's this tension between content and directives (mentioned elsewhere in-thread as "meta-syntax"). Folks are rightly pointing out that org-mode has this, and markdown needed it, but never standardized. "YAML frontmatter" is a common fix for this, but is inelegant: it essentially embeds one syntax into another, and creates another divide when trying to answer "How do I render this markdown file?"
So I would call out org-mode's robust support for source-code blocks that can execute actual code and embed the result (as well as the up-front directives that specify directives and metadata) as reasons it is superior to Markdown mode. Of course you can arbitrarily extend Markdown syntax and then write tooling to interpret that to do what you want. In fact, it looks like Jupyter has done just this with MyST. This is fine, but it just makes the problem worse, doubling down on the yet-another-markdown-flavor-will-solve-it strategy.
Until someone comes along a genuinely unifies the syntax and tooling for markdown, I suspect many folks here will wonder "Why did you invent a new markup syntax instead of just using org-mode?"