[MXS-4155] MaxScale needs to detect it is running in a container Created: 2022-06-03  Updated: 2022-10-25  Resolved: 2022-10-03

Status: Closed
Project: MariaDB MaxScale
Component/s: Core
Affects Version/s: None
Fix Version/s: 22.08.0

Type: Bug Priority: Major
Reporter: Johan Wikman Assignee: markus makela
Resolution: Done Votes: 0
Labels: None

Issue Links:
Duplicate
is duplicated by MXS-4176 thread=auto does work well on docker ... Closed
is duplicated by MXS-4368 Detect if maxscale is running inside ... Closed
Relates
relates to MXS-3269 Make it possible to change at runtime... Closed
relates to MXS-4179 Verify that cgroup detection works Closed

 Description   

At startup, MaxScale detects how many CPU cores and memory the system has and acts accordingly; e.g. starts as many routing threads as there are cores and uses at most 15% of the memory for the query classifier cache.

However, if MaxScale runs in a Docker container whose number of CPU cores and amount of memory has been restricted and is less than what the host system has, the detection does not work correctly as the numbers reported by the OS in the container are those of the host system and not what is available in the container.

Thus, in a restricted container MaxScale may attempt to use more resources than what is available, which may lead to performance and other issues.

MaxScale should detect when it is running in a container and adapt to the resources available in the container.



 Comments   
Comment by markus makela [ 2022-06-17 ]

The thread count and memory usage are now based on the cgroups CPU and memory limits and/or the number of CPUs the process is bound to. The lowest of all of the limits is chosen.

Comment by Hongyuan Li [ 2022-06-28 ]

the newest code in github uses hardware_concurrency() to get the cpu usage, which is not actual processor. I created a jira about the thread also with a patch, in which i parse the cgroup file, which is also closed by your jira manager. So, maxscale does not have the plan to support this?
docker update should be another case, cause it has two step to do.1、get the actual process periodicity 2、make maxscale reload the configuration.

Comment by markus makela [ 2022-06-28 ]

The code was already added to the develop branch:

Comment by markus makela [ 2022-06-28 ]

The case of docker update will be covered by MXS-3269.

Comment by Hongyuan Li [ 2022-06-28 ]

Hi,markus makela , the code use /proc/sys/cgroup as the default parent directory, which should be parsed from /proc/self/mountinfo and /proc/self/cgroup logicaly. and also cgroupv2 can be found from the cgroup hierarchy.i think you can have a look at my patch and my jira, which does the whole thing. This is just a suggestion.

Comment by markus makela [ 2022-06-28 ]

Hongyuan in which cases are the cgroup files mounted somewhere else than /sys/fs/cgroup/?

Comment by Hongyuan Li [ 2022-06-28 ]

from "https://www.kernel.org/doc/Documentation/cgroup-v2.txt" and "https://www.kernel.org/doc/Documentation/cgroup-v1.txt", mount point can be decided by user, i think user can change the mount point by "mount -t cgroup .... mount_point". from containerd code, the cgroup file also parsed from /proc/self/mountinfo, golang code list below

func v1MountPoint() (string, error) {
	f, err := os.Open("/proc/self/mountinfo")
	if err != nil {
		return "", err
	}
	defer f.Close()
	scanner := bufio.NewScanner(f)
	for scanner.Scan() {
		var (
			text      = scanner.Text()
			fields    = strings.Split(text, " ")
			numFields = len(fields)
		)
		if numFields < 10 {
			return "", fmt.Errorf("mountinfo: bad entry %q", text)
		}
		if fields[numFields-3] == "cgroup" {
			return filepath.Dir(fields[4]), nil
		}
	}
	if err := scanner.Err(); err != nil {
		return "", err
	}
	return "", ErrMountPointNotExist
}

Comment by markus makela [ 2022-06-28 ]

How often is sysfs mounted somewhere else than /sys? If it's common, then I guess relying on procfs being mounted at /proc is also wrong. All in all, I don't think it's a big problem to require both /sys and /proc to be mounted in their standard locations, especially if this covers a significant majority of all use-cases.

Comment by Hongyuan Li [ 2022-06-29 ]

/proc/self/mountinfo is not wrong, which is written in kernel docs. How often is sysfs mounted somewhere else than /sys? i guess that mountpoint is usually /sys/fs/cgroup, so, as i say, that is just a suggestion.

Comment by markus makela [ 2022-06-29 ]

Yeah, we could add support for non-standard sysfs locations when someone presents a use-case where they do it. For the 22.08 release, MaxScale will only support the normal sysfs location of /sys.

Thank you for both your patch and the feedback. We really do appreciate it even if in this case we already added the feature on our own.

Comment by Hongyuan Li [ 2022-06-29 ]

that is non-standard, in my opinion, you can say in the release note,you can only support default cgroup mountpoint, which is mounted on /sys/fs/cgroup.

Comment by markus makela [ 2022-10-03 ]

The log message that tells how many cores and how much memory is available uses the container's values if they are limited using cgroups. This is intended to be informational and if the values greatly exceed what it configured in MaxScale, a warning is logged.

Generated at Thu Feb 08 04:26:36 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.