docs: propose a pluggable search backend for Studio content search

[blarghmatey]

Description

This is an ADR proposing a direction, for discussion. No implementation accompanies it — the point is to agree (or not) on the shape before anyone writes the code.

It proposes a narrow, internal backend interface in content/search with implementations for both Meilisearch and Typesense, Meilisearch remaining the default so no existing deployment changes behaviour on upgrade.

Why now. edx-search ships a Typesense backend (search/typesense.py, contributed as FC-0091) serving Course Discovery and Learning-MFE courseware search. content/search is gated on MEILISEARCH_ENABLED and has no engine choice. An operator who wants both therefore runs two search engines for one platform, and that cost grows as more of the platform moves onto Typesense. The two subsystems cannot share an index. courseware_index.py builds each document from location_info, the block's index_dictionary(), id, start_date, content_groups and supplemental_fields — the strings breadcrumbs, block_type, publish_status, last_published, context_key, access_id and tags appear nowhere in that module — and it indexes under a published_only branch setting, while Studio authors drafts. edx-search's own module docstring says as much: "Other use cases are not supported."

Why this isn't a reversal. ADR 0001 is still in Draft, adopted Meilisearch "as an experiment and to evaluate it more thoroughly," and committed to keeping the engine code isolated "so it's relatively easy to swap out later if this experiment doesn't pan out." This proposes exercising that exit. That isolation largely held, by reference count across the app: api.py 105, and essentially all real coupling; handlers.py 21, of which 16 are the meilisearch_enabled settings gate; documents.py 17, all prose or the meili_id_… key-slugging helpers; tasks.py 16, of which 14 are MeilisearchError in retry handling — the one piece of genuine coupling outside api.py, needing the backend to supply its own retryable exception type.

High availability, and where its licence landed. ADR 0001's first stated concern was that Meilisearch "doesn't (yet) support High Availability via replication, although this is planned and under development." That has been resolved — but only in the Enterprise Edition.

Meilisearch now splits into two editions: Community Edition, MIT-licensed and free, and Enterprise Edition, licensed BUSL-1.1 and shipped as separate binaries and Docker images (getmeili/meilisearch-enterprise). Replicated sharding requires Enterprise Edition v1.37+. A self-hosted Community Edition deployment still has no replication, and so still has no HA.

That matters here more than it would elsewhere, because it is the same problem ADR 0001 was written to escape. Its opening argument against Elasticsearch is that "in 2021, the license of Elasticsearch changed from Apache 2.0 to a more restrictive license," and that this "is problematic for many Open edX operators." Meilisearch's HA answer has since arrived behind a BUSL-1.1 licence. An operator who cannot or will not take a commercial licence is where they started: Studio search on a single node with no replication path. Typesense's clustering isn't gated that way — GPL-3.0, Raft-backed replication in the ordinary self-hosted product, same typesense/typesense image with a shared nodes file.

One thing from ADR 0001 that should not be reused: its second concern, boolean operators in keyword search, is not a Typesense advantage. Typesense supports them in filters, as Meilisearch already does, and I established no difference in the keyword query itself.

None of this is a knock on Meilisearch's search quality. For a single-node deployment that never needs replication it remains a reasonable choice.

The strongest objection is in ADR 0001 itself, and the ADR addresses it head-on rather than hoping nobody notices: edx-search previously used django-haystack as a cross-engine abstraction, and that "became an obstacle to upgrades and efficiently utilizing Elasticsearch (the abstraction layer imposed significant limits)." That is a warning about a general abstraction spanning engines with different models. The interface proposed here is deliberately not that — it is private to content/search, shaped by the operations that app actually performs, offered to no other app, and changeable at will because both implementations live in-tree and nothing outside depends on it. If reviewers think that distinction doesn't hold, that is the thing to argue about, and it is worth arguing about before any code exists.

Which user roles this affects: Operators (may drop one of two search engines; no action required if they don't want to), and Developers working in content/search. No change for Course Authors or Learners.

Supporting information

Feasibility was checked rather than assumed. Every mapping below was executed against a running Typesense 30.2, with documents shaped like the three library document builders and the request shapes the Authoring MFE's search-manager actually issues — 30 of 30 checks pass:

• distinctAttribute → group_by + group_limit; ordered searchableAttributes → query_by + explicit query_by_weights; sortableAttributes → per-field sort; the rankingRules "sort"-first entry needs no counterpart, since an explicit sort_by already outranks text matching.
• create_index/swap_indexes/delete_index → a collection alias repoint, which also reclaims disk immediately.
• delete_documents(filter=…) → a filter_by delete.
• The tenant-token property ADR 0001 specifically credited Meilisearch for — minting a restricted, permission-scoped key locally so the browser queries the engine directly instead of routing through Django — holds. Typesense scoped search keys are an HMAC of a parent key over a rule carrying filter_by and expires_at, and need no API call to create. I minted one and confirmed the embedded filter is enforced server-side.
• _wait_for_meili_task and the polling around every write disappear on the Typesense side, since its writes are synchronous.

Three findings are recorded in the ADR because they will otherwise be rediscovered the hard way:

1. A content\..* string wildcard field — the pattern edx-search uses — rejects array sub-fields, so every container document fails to index with "field inside an array of objects must be an array type as well". Explicit string[] overrides are needed ahead of the wildcard. edx-search's own comment flags this as a risk; for library documents it is a certainty.
2. max_facet_values defaults to 10 and caps the reported total, so a truncated facet list is not detectable from the response. Measured: 25 distinct tags.level0 values return 10 with total_values: 10; with max_facet_values=100, 27 and 27. Left unset this would silently cut the tag filter tree off at ten branches. Meilisearch's equivalent defaults to 100.
3. documents.py serialises datetimes with .timestamp(), which yields a float, so those fields must be declared float. Copying edx-search's int64 coercion would reject every document.

Testing instructions

Documentation only; nothing to exercise. docutils parses the file with no warnings. Reviewers may want to check the two claims this rests on, both of which are cheap to verify: that ADR 0001's status is Draft, and that edx-search's Typesense backend cannot serve Studio search.

Deadline

None.

Other information

Sizing, so nobody is surprised later: the backend interface is the smaller half. The Authoring MFE's search-manager talks to the raw meilisearch JavaScript client rather than an InstantSearch adapter — data/api.ts alone is ~590 lines of engine-shaped calls — so a substantial share of it, concentrated in the data layer, needs to sit behind a client seam. Two specifics: Typesense returns grouped_hits where distinctAttribute returns flat hits, so the seam must normalise result shape; and attributesToCrop has no clean counterpart, since Typesense decides per field whether to snippet and how much context to keep, which needs a visual pass over the result cards rather than a rename.

Happy to take this to a Discourse thread or a working group instead if that is the better venue for a change at this level — I opened it as an ADR PR because that is where ADR 0001 lives and what it would supersede.

[openedx-webhooks]

Thanks for the pull request, @blarghmatey!

This repository is currently maintained by @openedx/wg-maintenance-openedx-platform-oncall.

Once you've gone through the following steps feel free to tag them in a comment and let them know that your changes are ready for engineering review.

🔘 Get product approval

If you haven't already, check this list to see if your contribution needs to go through the product review process.

• If it does, you'll need to submit a product proposal for your contribution, and have it reviewed by the Product Working Group.
  • This process (including the steps you'll need to take) is documented here.
• If it doesn't, simply proceed with the next step.

🔘 Provide context

To help your reviewers and other members of the community understand the purpose and larger context of your changes, feel free to add as much of the following information to the PR description as you can:

• Dependencies

  This PR must be merged before / after / at the same time as ...

• Blockers

  This PR is waiting for OEP-1234 to be accepted.

• Timeline information

  This PR must be merged by XX date because ...

• Partner information

  This is for a course on edx.org.

• Supporting documentation
• Relevant Open edX discussion forum threads

🔘 Get a green build

If one or more checks are failing, continue working on your changes until this is no longer the case and your build turns green.

Details

---

Where can I find more information?

If you'd like to get more details on all aspects of the review process for open source pull requests (OSPRs), check out the following resources:

• Overview of Review Process for Community Contributions
• Pull Request Status Guide
• Making changes to your pull request

When can I expect my changes to be merged?

Our goal is to get community contributions seen and reviewed as efficiently as possible.

However, the amount of time that it takes to review and merge a PR can vary significantly based on factors such as:

• The size and impact of the changes that it introduces
• The need for product review
• Maintenance status of the parent repository

💡 As a result it may take up to several weeks or months to complete a review and merge your PR.

[blarghmatey]

Correcting the original version of this description, in case anyone read it before now.

I had written that Meilisearch "has since shipped sharding and replication for self-hosted deployments" and that ADR 0001's high-availability concern therefore no longer applied. That is wrong. Replicated sharding requires Meilisearch Enterprise Edition v1.37+, which is BUSL-1.1 licensed and distributed as separate binaries and Docker images. Self-hosted Community Edition — MIT, and what an operator running the OSS stack would use — still has no replication.

The description has been updated. The correction strengthens the case rather than weakening it, which is exactly why I want it on the record explicitly: ADR 0001's opening argument is about Elasticsearch leaving Apache 2.0 being "problematic for many Open edX operators", and Meilisearch's high-availability answer has now landed behind a commercial licence of the same kind. That is a fair thing for reviewers to weigh, and I would rather it be weighed accurately than have it quietly do work in the background off a claim I got backwards.

[bradenmacdonald]

I think the bigger point here, or at least a very relevant point, is that Meilisearch and Typesense are both Algolia-style search engines and have very similar models/APIs, so abstracting across them is fairly easy. (As would be adding Algolia.)

[bradenmacdonald]

It would be better to get rid of the task-polling layer completely, and move to an asynchronous API as is being discussed on #38993 . The main motivation being that the task-polling is a performance bottleneck, but also that blocking the UI is not as good as a proper async UI update would be.

Fine to leave that out of scope for any initial implementation of this ADR, but I think it's worth referencing here.

[bradenmacdonald]

The search manager was originally written to use InstantSearch, however InstantSearch does not support a hierarchical menu that allows multiple selections (it only has this). I tried modifying InstantSearch to support it, but I found the code very difficult to work with (and others have concluded that multiple selections isn't really possible with InstantSearch), so we made our own implementation which was much more efficient at the cost of being tied to Meilisearch.

As far as I know, it is still an open question as to whether or not it's possible to implement the part of the Studio UI shown below (refining search with multi-select hierarchical facets) with either Typesense or Algolia. Even doing it with Meilisearch required some creative use of their advanced APIs.

---

must be reworked to sit behind a client seam

I think that given this is the biggest piece of work by far, it needs a more detailed plan.

[bradenmacdonald]

Note: one potential path forward could be dropping support for multiple selections from the UI, and switching back to InstantSearch.js. That would require product approval and also a cost-benefit analysis of its effect on bundle size.

Another path forward would be to implement our own minimal frontend abstraction layer.

Another path forward would be to implement the search logic on the backend, in the proposed python abstraction layer. Although this is clean, I prefer not do it as it can impact performance by tying up LMS worker threads merely to rewrite and pass on requests to the search engine. If we were to start using asyncio django, it could be worth a serious look though.

[bradenmacdonald]

I don't think this particular justification makes sense. It's much easier for the few operators using Typesense (I know only of MIT) to switch to Meilisearch than it is to refactor the Studio backend and frontend to support Typesense. The "path off the second engine" exists now, and it's to stop running Typesense.

However, I'm not disagreeing with your ADR in general; I just think the only reasonable justification here would be "operators with HA requirements cannot accept Meilisearch Enterprise Edition for licensing or cost reasons", and not "someone somewhere started with Typesense so they need to be able to continue with it conveniently".

[bradenmacdonald]

I would also add:

• Given the history of license changes in the past (Elasticsearch, Meilisearch, etc.), it is pragmatic to remain flexible, even if we try to encourage most people to use the same, simple solution; and
• 2U or others may wish to implement Algolia support in the future, given that they are already using it for many other parts of the platform, and since Typesense and Meilisearch are both heavily Algolia-inspired, that would be easy to do.
  • In fact, Algolia is referenced in 100 different Open edX source files (see link ^), and Typesense only 24.

[bradenmacdonald]

This is where I'd like to push back the most.

If we're going to implement a new abstraction layer, I'd prefer we do it properly with something that can replace both edx_search and content/search. Not by extending edx_search but by offering a cleaner alternative which can be used for Studio search as well as any other use cases we have (mostly: forum search, course directory search), with consistent API and tooling.

[blarghmatey]

Thanks — this is the review I was hoping for, and it has moved me on the scope question.

On "the path off the second engine is to stop running Typesense"

Two reasons that isn't available to us, and they are the two you identified as the reasonable justification.

No high availability. Replicated sharding requires Meilisearch Enterprise Edition v1.37+, which is BUSL-1.1 and ships as separate binaries and images (getmeili/meilisearch-enterprise). Community Edition has no replication at all. Typesense's Raft-backed clustering is part of the ordinary GPL-3.0 product — same image, a shared nodes file — and we run three-node clusters on it today against a single-replica Meilisearch. So "stop running Typesense" means running Studio search with no replication path, or buying a commercial licence.

The licence shift is the same problem that prompted this engine change in the first place. ADR 0001 opens with Elasticsearch moving off Apache 2.0 to "a more restrictive license," and that being "problematic for many Open edX operators." Meilisearch has since done the same thing to the capability we need. And it is not only new features going to Enterprise Edition: S3-streaming snapshots were available in Community Edition and were reclassified as an Enterprise feature, with CE 1.25–1.30 grandfathered. A capability that was open was withdrawn. That is precisely the pattern the original decision was made to get away from, and it is a reason to keep more than one engine viable rather than to consolidate harder on this one.

On the framing of Typesense as one operator's idiosyncratic choice — I would push back on that. The dichotomy is the platform's, not ours. FC-0091 added Typesense backends to edx-search, to forum, and to the docs, but never to content/search. So Typesense is today a supported engine for Course Discovery, Learning-MFE courseware search and forum search, and an unsupported one for Studio. We are not asking for a bespoke engine; we are asking for the part of a funded contribution that was never finished.

Some of that work also shipped incomplete rather than merely absent: the forum backend was non-functional on any search — per_page above Typesense's hard cap of 250, and a filter naming a field the collection schema does not declare. Fix in openedx/forum#289, with end-to-end tests against a real Typesense, since the existing tests mocked the client and so could not see a rejected request.

On scope — you've convinced me

I'd rather build the thing you actually want than land a narrow layer you'd have to unpick later. Doing this as the openedx/edx-search#245 re-architecture — one surface with registered use cases (studio search, library search, learner courseware search, forum search), consistent index management and tooling — is a better shape than an interface private to content/search, and I'd support taking it there.

One condition I'd want written into that work rather than discovered during it: Typesense needs to be a first-class target alongside Meilisearch from the start, not a backend added afterwards. That also serves your Algolia point better than a Meilisearch-shaped surface would — an abstraction designed against two engines from day one will take a third far more easily than one designed against a single engine and generalised later.

This does mean my ADR is aimed at the wrong layer. I'm happy to rework it against #245's framing, or to fold it in there and close this, whichever you prefer as the venue.

On keeping Django out of the search path

Agreed, and for your reason — tying up worker threads to rewrite and forward requests would be a real cost. Worth confirming that browser-direct survives the change: Typesense scoped search keys are the analogue of Meilisearch tenant tokens and share the property the original ADR valued. They're an HMAC of a parent key over a rule carrying filter_by and expires_at, derived locally with no API call. I've verified the embedded filter is enforced server-side, including that a key scoped to a different org returns nothing rather than erroring.

On the frontend

You were right that I under-scoped this, and right that the line count wasn't the interesting part. Since it's the question that decides whether any of this is worth doing, I went and answered it rather than leaving it open: multi-select hierarchical faceting works on Typesense. I replicated fetchAvailableTagOptions() against a real Typesense 30.2 using the document shape searchable_doc_tags() produces — 29 checks, all passing.

The mapping is one-for-one, not a workaround:

search-manager does	Typesense
searchForFacetValues({ facetName, facetQuery })	facet_by + facet_query with per_page=0
facet values and counts, narrowed by q and the active filters	the same request carries q and filter_by
tags.taxonomy / tags.levelN as facetable hierarchies	nested string[] facet fields, all four levels
multi-select as AND'd tag paths	&& of tags.levelN:= terms

Selections AND correctly across taxonomies and across levels, parent-and-child of the same branch behaves, and contradictory selections return nothing.

The part I expected to be hard turned out not to arise. I went in assuming disjunctive faceting would be the obstacle, and it isn't, because fetchAvailableTagOptions never passes the tags filter into the tree query — only extraFilter, the block-type filter and the parent filter. The tree is already computed without the current tag selections applied, so there is nothing special to reproduce.

Two respects where Typesense is actually the better fit for this UI:

• The facet ceiling. meilisearchFacetLimit = 100 is hard-coded in fetchAvailableTagOptions, and drives the mayBeMissingResults / "not all tags could be displayed" warning. Typesense accepts max_facet_values well past that — I returned 300 sibling tags in a single request. That warning could mostly go away.
• Round trips. searchForFacetValues is a separate endpoint that can't be folded into a multiSearch, so each tree node costs two requests. In Typesense these are ordinary searches, so the level query and the hasChildren look-ahead batch into one multi_search.

Two things that need care, neither of them blocking:

• Shared-prefix siblings still need the existing post-processing. Location > North America > Canada also prefix-matches Canada Extra > Nowhere, so the exact-parent check in the current code is still required. It's a property of prefix matching, not of fuzziness. Relatedly, facet_query_num_typos=0 gives an exact prefix match where Meilisearch's fuzziness isn't tunable, so one source of noise does disappear.
• Use :=, not :. The exact operator correctly excludes Canada Extra when filtering on Canada; the non-exact one matches both. Tag values containing colons, &&/||, quotes and non-ASCII all filter correctly.

To be clear about what this does and doesn't establish: I've proven the query mechanics, not a working UI. What remains untested is that TagOptions renders correctly against these responses and that a client seam can serve both engines without branching through the component tree. That's still real work — it just isn't unknown work any more, and the feasibility risk you flagged is retired. Happy to share the harness if it's useful to whoever picks the frontend up.

On async indexing

Agreed that relocating the task-polling layer is not as good as removing it, and I'll reference #38993 in whatever the ADR becomes.

One data point for that thread, since the reporter there is on Meilisearch 1.8 and your first suggestion was to upgrade: we see the same class of problem on v1.53.1, with a studio_content index at 1.32M documents and 11.91 GiB. So at least in our case it isn't explained by an old indexer. I'll follow up there separately rather than derail this PR.

[bradenmacdonald]

I went and answered it rather than leaving it open: multi-select hierarchical faceting works on Typesense. I replicated fetchAvailableTagOptions() against a real Typesense 30.2 using the document shape searchable_doc_tags() produces — 29 checks, all passing.

Nice, thanks a lot for checking on that. Together with the results you shared in the other thread, I'm now kinda leaning toward making Typesense the default engine and Meilisearch the other fully-supported option, in the proposed rework. In any case, fully agree that they should both have first-class support.

This does mean my ADR is aimed at the wrong layer. I'm happy to rework it against #245 framing, or to fold it in there and close this, whichever you prefer as the venue.

@ormsbee what do you think?

Personally my inclination is to create a new openedx_search repo, and let the revised version of this proposal become ADR 0001 for that repo. But we can also put it in edx-search as the answer to 245's "we need to refactor".

[blarghmatey]

Closing in favour of openedx/openedx-search#1, which is the reworked version of this proposal as ADR 0001 of the new repo.

@bradenmacdonald's review was right that this was aimed one layer too low. The rework follows from that:

• Scope is the platform rather than content/search. Use cases register (studio content search, library search, learner courseware search, course discovery, forum search) instead of each surface carrying its own engine integration — today there are four, across content/search, edx-search, forum and edx-notes-api.
• Meilisearch and Typesense are both first-class from day one, rather than one being the default and the other an opt-in backend. Which one is the default is left as an explicit open question, since that is separable from which are supported.
• The django-haystack objection gets a boundary instead of a promise. This version draws the line at the engine family — Algolia-shaped document stores in, Elasticsearch and OpenSearch out — which is the same line Prototype Typesense search support modular-learning#245 drew, and for the same reason.

Two things asked for in review are answered there rather than deferred: multi-select hierarchical faceting is demonstrated working on Typesense (29 checks against a real server, including that the disjunctive-faceting problem does not arise because fetchAvailableTagOptions never passes the tags filter into the tree query), and same-host indexing measurements are included as context for #38993.

Thanks for the review. It changed the proposal substantially and for the better.