MDEV-38918 Make large pages an explicit per-caller opt-in

[vaintroub]

Summary

my_large_malloc() attempted large pages whenever --large-pages was enabled, silently rounding the size up and reporting it back via an in/out parameter. ut_malloc_dontdump() never passed that adjusted size on to its own callers (the InnoDB redo log buffer and recv_sys_t::tmp_buf), so freeing later used the original, smaller size, causing the reported "faux memory leak".

Only the buffer pool and the MyISAM/Aria key caches are documented to benefit from large pages. Everything else that ended up calling my_large_malloc() only wanted its "do not dump to core" property and picked up large pages as an undocumented side effect; those buffers are also small and sequentially accessed, so they would have gained little from large pages anyway.

• Add MY_TRY_LARGE_PAGES: my_large_malloc() and my_large_virtual_alloc() now only attempt large pages when a caller passes this flag, instead of always trying whenever the global option is set. Only the buffer pool and the key caches pass it.
• The redo log buffer, tmp_buf, and row0log.cc's crypt buffers no longer request large pages at all, removing the size-rounding bug for them without touching that code.
• Fix a broken mtr suppression regex in main.large_pages that would fail the test on Windows.

Test plan

• main.large_pages passes
• Manual repro: mariadbd.exe --large-pages with an oversized buffer pool, confirmed graceful fallback and shutdown reports no leak

[dr-m]

The test main.large_pages is crashing across the board, apparently on the very first access to buf_pool.memory in buf_pool_t::create().
I would like to note that by default, large pages are unavailable on Linux. They have to be reserved separately:

echo 96|sudo tee /sys/kernel/mm/hugepages/hugepages-1048576kB/nr_hugepages
echo 1024|sudo tee /sys/kernel/mm/hugepages/hugepages-2048kB/nr_hugepages

Because such pages would be unavailable for normal allocation, I assume that we have no Linux environment in our CI where the large_pages option would actually work.

[vaintroub]

Thank you for catching this, and for the testing tip — reserving huge pages via /sys/kernel/mm/hugepages/.../nr_hugepages let me verify both the fallback and the real hugetlb-success paths locally. Addressed in the follow-up commit.

re-requested already

[Copilot]

Pull request overview

This PR makes large-page allocation an explicit per-caller opt-in by introducing MY_TRY_LARGE_PAGES (and helper my_large_pages_flag()), so only documented beneficiaries (buffer pool and key caches) attempt large pages, avoiding unintended size rounding and mismatched free sizes elsewhere.

Changes:

• Introduce MY_TRY_LARGE_PAGES and my_large_pages_flag() to explicitly request large pages per allocation/caller.
• Plumb myf my_flags through my_virtual_mem_{reserve,commit,decommit,release}() and my_large_virtual_alloc() and update key call sites (InnoDB buffer pool, Maria/MyISAM key caches/page cache).
• Fix the main.large_pages suppression regex to cover Windows MEM_LARGE_PAGES warnings.

Reviewed changes

Copilot reviewed 10 out of 10 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
storage/maria/ma_pagecache.c	Page cache allocation now explicitly opts into large pages only when enabled.
storage/innobase/buf/buf0buf.cc	Buffer pool virtual-memory reserve/commit/decommit/release now consistently passes the large-page opt-in flag.
mysys/my_virtual_mem.c	Virtual memory API now takes my_flags and uses MY_TRY_LARGE_PAGES to control large-page behavior.
mysys/my_largepage.c	Large-page allocators now attempt large pages only when MY_TRY_LARGE_PAGES is provided; reserve fallback protection behavior clarified.
mysys/my_alloc.c	MEM_ROOT vmem allocations updated to new my_virtual_mem_* signatures.
mysys/mf_keycache.c	Key cache allocations now explicitly opt into large pages when enabled.
mysql-test/main/large_pages.test	Suppression regex updated for Windows MEM_LARGE_PAGES wording.
mysql-test/main/large_pages.result	Expected output updated to match the new suppression regex.
include/my_virtual_mem.h	Public header updated for new my_virtual_mem_* signatures and myf type.
include/my_sys.h	Adds MY_TRY_LARGE_PAGES and my_large_pages_flag() helper; updates my_large_virtual_alloc() signature.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[dr-m]

Please squash this to a single commit, or make sure that each individual commit will pass the regression test suite.

[dr-m]

I would suggest the following change to reduce the amount of run-time conditions in this function (including the final mmap() call):

#ifdef _AIX
  int flags= MAP_PRIVATE | OS_MAP_ANON;
  int prot= PROT_READ | PROT_WRITE;
#else
  /* Illumos important to have MAP_NORESERVE otherwise reserves all swap
  on innodb_buffer_pool_size_max overallocation.
  Linux is controlled on sysctl vm.overcommit_memory. */
  int flags= MAP_PRIVATE | OS_MAP_ANON | MAP_NORESERVE;
  int prot= PROT_NONE;
#endif
  if (my_flags & MY_TRY_LARGE_PAGES)
  {
    size_t large_page_size;
    int page_i= 0;
#ifndef _AIX
    prot= PROT_READ | PROT_WRITE;
    flags= MAP_PRIVATE | OS_MAP_ANON;
#endif

[vaintroub]

you seem to really like AIX :) I did that, without second ifndef, I do not see how it hurts AIX. BTW, I still do not understand the AIX handling. Why did you use PROT_READWRITE, if everything is using PROT_NONE? Why is AIX present in buf0buf.cc, and why avoids a commit and decommit, I think those are functional, even if reserving virtual address space does not work(or does it)? Could you elaborate?

[dr-m]

my_use_large_pages is missing a comment that its value is not supposed to be initialized by my_init_large_pages() and not thereafter.

As far as I understand, with the removal of the assignment from my_large_virtual_alloc(), the variable my_use_large_pages would always be 1 outside Microsoft Windows. Therefore, it should only be declared there. I would suggest the following:

#ifdef _WIN32
/** MY_TRY_LARGE_PAGES or 0; not changed after my_init_large_pages() */
extern myf my_large_pages_flag;
#else
# define my_large_pages_flag MY_TRY_LARGE_PAGES
#endif

The initialisation would be as follows:

int my_init_large_pages(void)
{
#ifdef _WIN32
  my_large_pages_flag= my_obtain_privilege(SE_LOCK_MEMORY_NAME)
    ? MY_TRY_LARGE_PAGES : 0;

[vaintroub]

I added a comment. The suggestion is adding too much _WIN32, and I do not really understand it. We need to know if --large-pages was passed, even if you're on Linux. We do not try to allocate large pages if --large-pages was not requested. Somewhere you'd need to record it, whether large-pages was desired or not. Besides, I'm using the variable now, on Linux as well, set it to 0, if it turns out it is not possible to determine large page sizes on Linux, so that any attempt on large pages is futile.