NUMA nodes and cpu layout

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

NUMA nodes and cpu layout

Stefan Kadow
Hello,

I have installed Debian 9.4 Stretch (stable) with Xen Hypervisor 4.8
(package from repository) on a NUMA machine with two AMD Epyc processors.

When booting the default Debian kernel, the command "numactl --hardware"
returns the following:
available: 8 nodes (0-7)
node 0 cpus: 0 1 2 3 32 33 34 35
node 0 size: 0 MB
node 0 free: 0 MB
node 1 cpus: 4 5 6 7 36 37 38 39
node 1 size: 32170 MB
node 1 free: 31758 MB
node 2 cpus: 8 9 10 11 40 41 42 43
node 2 size: 0 MB
node 2 free: 0 MB
...

But when booting the Xen Kernel, the command "xl cpupool-numa-split"
returns another layout for cpus and nodes:
Name               CPU list
Pool-node0         0,1,2,3,4,5,6,7
Pool-node1         8,9,10,11,12,13,14,15
Pool-node2         16,17,18,19,20,21,22,23
...

Shouldn't the layout be the same?

I want pools of 8 cpus each, but which cpus should be combined for
optimal performance?

Solution 1) from numactl
cpus="0-3,32-35"

Solution 2) from xl cpupool-numa-split
cpus="0-7"

--
Stefan

_______________________________________________
Xen-users mailing list
[hidden email]
https://lists.xenproject.org/mailman/listinfo/xen-users

numactl-hardware.txt (1K) Download Attachment
cpupool-numa-split.txt (496 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: NUMA nodes and cpu layout

George Dunlap
On Thu, Jun 14, 2018 at 11:19 PM, Stefan Kadow <[hidden email]> wrote:

> Hello,
>
> I have installed Debian 9.4 Stretch (stable) with Xen Hypervisor 4.8
> (package from repository) on a NUMA machine with two AMD Epyc processors.
>
> When booting the default Debian kernel, the command "numactl --hardware"
> returns the following:
> available: 8 nodes (0-7)
> node 0 cpus: 0 1 2 3 32 33 34 35
> node 0 size: 0 MB
> node 0 free: 0 MB
> node 1 cpus: 4 5 6 7 36 37 38 39
> node 1 size: 32170 MB
> node 1 free: 31758 MB
> node 2 cpus: 8 9 10 11 40 41 42 43
> node 2 size: 0 MB
> node 2 free: 0 MB
> ...
>
> But when booting the Xen Kernel, the command "xl cpupool-numa-split"
> returns another layout for cpus and nodes:
> Name               CPU list
> Pool-node0         0,1,2,3,4,5,6,7
> Pool-node1         8,9,10,11,12,13,14,15
> Pool-node2         16,17,18,19,20,21,22,23
> ...
>
> Shouldn't the layout be the same?

No, because the numbers don't mean the same thing.  numactl seems to
be reporting the ACPI IDs of the various logical processors, but ACPI
often gives you nonsensical numbers like you see there -- nodes as
high as 43 even though there are actually only 24 logical cpus,
disjoint sets of numbers for the same node, &c.

Xen uses its own numbering scheme, which is designed to be a bit more
rational: hyperthreads are always next to each other, cores are
contiguous, no holes in the number sequence.

The long and the short of it is: When using Xen tools, use the numbers
reported by `xl info`.  When using numactl, use numbers reported by
numactl.

 -George

_______________________________________________
Xen-users mailing list
[hidden email]
https://lists.xenproject.org/mailman/listinfo/xen-users
Reply | Threaded
Open this post in threaded view
|

Re: NUMA nodes and cpu layout

Stefan Kadow
Am 15.06.2018 um 09:00 schrieb George Dunlap:

> On Thu, Jun 14, 2018 at 11:19 PM, Stefan Kadow <[hidden email]> wrote:
>> Hello,
>>
>> I have installed Debian 9.4 Stretch (stable) with Xen Hypervisor 4.8
>> (package from repository) on a NUMA machine with two AMD Epyc processors.
>>
>> When booting the default Debian kernel, the command "numactl --hardware"
>> returns the following:
>> available: 8 nodes (0-7)
>> node 0 cpus: 0 1 2 3 32 33 34 35
>> node 0 size: 0 MB
>> node 0 free: 0 MB
>> node 1 cpus: 4 5 6 7 36 37 38 39
>> node 1 size: 32170 MB
>> node 1 free: 31758 MB
>> node 2 cpus: 8 9 10 11 40 41 42 43
>> node 2 size: 0 MB
>> node 2 free: 0 MB
>> ...
>>
>> But when booting the Xen Kernel, the command "xl cpupool-numa-split"
>> returns another layout for cpus and nodes:
>> Name               CPU list
>> Pool-node0         0,1,2,3,4,5,6,7
>> Pool-node1         8,9,10,11,12,13,14,15
>> Pool-node2         16,17,18,19,20,21,22,23
>> ...
>>
>> Shouldn't the layout be the same?
>
> No, because the numbers don't mean the same thing.  numactl seems to
> be reporting the ACPI IDs of the various logical processors, but ACPI
> often gives you nonsensical numbers like you see there -- nodes as
> high as 43 even though there are actually only 24 logical cpus,
> disjoint sets of numbers for the same node, &c.

This was just an excerpt, I did not want to post the whole list of all
64 CPUs, sorry. So there are no holes in the numer sequence.


> Xen uses its own numbering scheme, which is designed to be a bit more
> rational: hyperthreads are always next to each other, cores are
> contiguous, no holes in the number sequence.
>
> The long and the short of it is: When using Xen tools, use the numbers
> reported by `xl info`.  When using numactl, use numbers reported by
> numactl.

Fine, then I will trust the output of Xen tools.

>  -George
>
> _______________________________________________
> Xen-users mailing list
> [hidden email]
> https://lists.xenproject.org/mailman/listinfo/xen-users

Thanks for your quick answer!

--
Stefan

_______________________________________________
Xen-users mailing list
[hidden email]
https://lists.xenproject.org/mailman/listinfo/xen-users
Reply | Threaded
Open this post in threaded view
|

Re: NUMA nodes and cpu layout

Dario Faggioli-4
In reply to this post by Stefan Kadow
On Fri, 2018-06-15 at 00:19 +0200, Stefan Kadow wrote:

> When booting the default Debian kernel, the command "numactl --
> hardware"
> returns the following:
> available: 8 nodes (0-7)
> node 0 cpus: 0 1 2 3 32 33 34 35
> node 0 size: 0 MB
> node 0 free: 0 MB
> node 1 cpus: 4 5 6 7 36 37 38 39
> node 1 size: 32170 MB
> node 1 free: 31758 MB
> node 2 cpus: 8 9 10 11 40 41 42 43
> node 2 size: 0 MB
> node 2 free: 0 MB
> ...

> I want pools of 8 cpus each, but which cpus should be combined for
> optimal performance?
>
> Solution 1) from numactl
> cpus="0-3,32-35"
>
> Solution 2) from xl cpupool-numa-split
> cpus="0-7"
>
What George said. Check also the output of `xl info -n', hopefully that
will clarify things a bit (on how Xen sees things).

On a not so related note, you seem, at least according to numactl, to
have nodes with no RAM, is that correct/done on-purpose?

Regards,
Dario
--
<<This happens because I choose it to happen!>> (Raistlin Majere)
-----------------------------------------------------------------
Dario Faggioli, Ph.D, http://about.me/dario.faggioli
Software Engineer @ SUSE https://www.suse.com/
_______________________________________________
Xen-users mailing list
[hidden email]
https://lists.xenproject.org/mailman/listinfo/xen-users

signature.asc (849 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: NUMA nodes and cpu layout

Stefan Kadow
Am 15.06.2018 um 15:15 schrieb Dario Faggioli:

> On Fri, 2018-06-15 at 00:19 +0200, Stefan Kadow wrote:
>> When booting the default Debian kernel, the command "numactl --
>> hardware"
>> returns the following:
>> available: 8 nodes (0-7)
>> node 0 cpus: 0 1 2 3 32 33 34 35
>> node 0 size: 0 MB
>> node 0 free: 0 MB
>> node 1 cpus: 4 5 6 7 36 37 38 39
>> node 1 size: 32170 MB
>> node 1 free: 31758 MB
>> node 2 cpus: 8 9 10 11 40 41 42 43
>> node 2 size: 0 MB
>> node 2 free: 0 MB
>> ...
>
>> I want pools of 8 cpus each, but which cpus should be combined for
>> optimal performance?
>>
>> Solution 1) from numactl
>> cpus="0-3,32-35"
>>
>> Solution 2) from xl cpupool-numa-split
>> cpus="0-7"
>>
> What George said. Check also the output of `xl info -n', hopefully that
> will clarify things a bit (on how Xen sees things).
Yes, the numbering of xen tools is different from the numbers numactl
outputs, I understand that.
I attached the complete output of "xl info -n" and the output of "xl
dmesg". You will find the output of "numactl --hardware" in my first post.

> On a not so related note, you seem, at least according to numactl, to
> have nodes with no RAM, is that correct/done on-purpose?

I saw that, too. But I didn't do that. It is only the default output
after installing "numactl" on this machine.

> Regards,
> Dario

Thanks,
Stefan

_______________________________________________
Xen-users mailing list
[hidden email]
https://lists.xenproject.org/mailman/listinfo/xen-users

xl-info-numa.txt (10K) Download Attachment
xl-dmesg.txt (11K) Download Attachment