ARM: Xen on Vexpress

classic Classic list List threaded Threaded
61 messages Options
1234
Reply | Threaded
Open this post in threaded view
|

ARM: Xen on Vexpress

Jeenu Viswambharan
Hi,

I've built necessary files and setup uboot and board to boot Xen on
Vexpress following the Wiki page. But what I see is u-boot going silent,
presumably after it jumps to Xen. My logs are attached.

I've a couple of questions:

  - I've noticed from the Vexpress Wiki that the u-boot variable
    xen_addr_r is set to 0x80400000, but the xen-uImage is built with
    both load and entry addresses as 0x80200000. I also checked the
    xen/xen.axf file (from which xen/xen is built), and its entry too is
    0x80200000. I'm afraid to ask if this a typo/bit rot in the Wiki?

    Anyway, I retried boot procedure with xen_addr_r set to 0x80200000
    instead, but this time I got an UNDEF at instruction at 0x80200008

  - Are there any logs that I can enable in Xen so that I'd know what's
    going on?

Thanks.

--
Jeenu

-- IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium.  Thank you.

ARM Limited, Registered office 110 Fulbourn Road, Cambridge CB1 9NJ, Registered in England & Wales, Company No:  2557590
ARM Holdings plc, Registered office 110 Fulbourn Road, Cambridge CB1 9NJ, Registered in England & Wales, Company No:  2548782
_______________________________________________
Xen-users mailing list
[hidden email]
http://lists.xen.org/xen-users

xen.log.gz (2K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: ARM: Xen on Vexpress

Ian Campbell-10
CCing Julien and Stefano who I think have both run Xen on vexpress

On Tue, 2014-06-03 at 18:11 +0100, Jeenu Viswambharan wrote:
> Hi,
>
> I've built necessary files and setup uboot and board to boot Xen on
> Vexpress following the Wiki page. But what I see is u-boot going silent,
> presumably after it jumps to Xen. My logs are attached.

Jeenu, Which version of Xen are you building?

Skimming that wiki page it seems to be describing how to use an older
version of Xen (like 4.3 or early 4.4 development era) and not a modern
4.4 or 4.5 development hypervisor.

In particular I'm not sure it is referencing a u-boot which boots the
kernel in HYP mode (Xen used to carry a hacky workaround for thus, but
doesn't now) and the stuff about KERNEL_FLASH_ADDRESS (another
workaround for lack of a boot protocol which is no longer present) isn't
true any more either.

Julien/Stefano can you confirm and/or deny? I think Linaro tests on
vexpress so I suppose there are more modern steps somewhere?

> I've a couple of questions:
>
>   - I've noticed from the Vexpress Wiki that the u-boot variable
>     xen_addr_r is set to 0x80400000, but the xen-uImage is built with
>     both load and entry addresses as 0x80200000. I also checked the
>     xen/xen.axf file (from which xen/xen is built), and its entry too is
>     0x80200000. I'm afraid to ask if this a typo/bit rot in the Wiki?

I'm not sure, but I think in general it is OK for these to differ,
u-boot will relocate from the load address to the address requested by
the header as part of booting. So long as it is in RAM and 2MB aligned I
don't think the actual address matters too much.

>     Anyway, I retried boot procedure with xen_addr_r set to 0x80200000
>     instead, but this time I got an UNDEF at instruction at 0x80200008
>
>   - Are there any logs that I can enable in Xen so that I'd know what's
>     going on?

Are you building with "debug=y CONFIG_EARLY_PRINTK=vexpress" on your
make invocation? You will need to "make clean" if you add these now.
That should get you some output earlier.

Ian.


_______________________________________________
Xen-users mailing list
[hidden email]
http://lists.xen.org/xen-users
Reply | Threaded
Open this post in threaded view
|

Re: ARM: Xen on Vexpress

Jeenu Viswambharan
On Wed, Jun 04, 2014 at 10:43:38, Ian Campbell wrote:
> On Tue, 2014-06-03 at 18:11 +0100, Jeenu Viswambharan wrote:
> > Hi,
> >
> > I've built necessary files and setup uboot and board to boot Xen on
> > Vexpress following the Wiki page. But what I see is u-boot going
> > silent, presumably after it jumps to Xen. My logs are attached.
>
> Jeenu, Which version of Xen are you building?

I'm on Xen RELEASE-4.4.0 tag. I'm using u-boot v2013.10-rc2 tag, but
I've a feeling that it might be a bit old for Hyp-related stuff.

> >   - Are there any logs that I can enable in Xen so that I'd know
> >     what's going on?
>
> Are you building with "debug=y CONFIG_EARLY_PRINTK=vexpress" on your
> make invocation? You will need to "make clean" if you add these now.
> That should get you some output earlier.

OK, will try with that.

Thanks.

--
Jeenu

-- IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium.  Thank you.

ARM Limited, Registered office 110 Fulbourn Road, Cambridge CB1 9NJ, Registered in England & Wales, Company No:  2557590
ARM Holdings plc, Registered office 110 Fulbourn Road, Cambridge CB1 9NJ, Registered in England & Wales, Company No:  2548782
_______________________________________________
Xen-users mailing list
[hidden email]
http://lists.xen.org/xen-users
Reply | Threaded
Open this post in threaded view
|

Re: ARM: Xen on Vexpress

Julien Grall-2
In reply to this post by Ian Campbell-10
On 06/04/2014 10:43 AM, Ian Campbell wrote:
> CCing Julien and Stefano who I think have both run Xen on vexpress

Woah, I haven't boot Xen on the vexpress since a while :).

> On Tue, 2014-06-03 at 18:11 +0100, Jeenu Viswambharan wrote:
>> Hi,
>>
>> I've built necessary files and setup uboot and board to boot Xen on
>> Vexpress following the Wiki page. But what I see is u-boot going silent,
>> presumably after it jumps to Xen. My logs are attached.
>
> Jeenu, Which version of Xen are you building?
>
> Skimming that wiki page it seems to be describing how to use an older
> version of Xen (like 4.3 or early 4.4 development era) and not a modern
> 4.4 or 4.5 development hypervisor.

The wiki page is based on a Xen 4.4 with a proper multiboot support. I
don't think there was many change in Xen since the page were written
(back in september 2013).

> In particular I'm not sure it is referencing a u-boot which boots the
> kernel in HYP mode (Xen used to carry a hacky workaround for thus, but
> doesn't now) and the stuff about KERNEL_FLASH_ADDRESS (another
> workaround for lack of a boot protocol which is no longer present) isn't
> true any more either.

The wiki page for the Vexpress is referencing a valid u-boot:
http://wiki.xen.org/wiki/Xen_ARM_with_Virtualization_Extensions/Vexpress#Booting_with_U-Boot

I was using it with Xen 4.4 a while ago.

> Julien/Stefano can you confirm and/or deny? I think Linaro tests on
> vexpress so I suppose there are more modern steps somewhere?

We don't have any Linaro tests on the vexpress for Xen. IIRC, it was too
difficult to use U-boot with the validation loop so we chose to only
test Xen on the Arndale & Midway.

>> I've a couple of questions:
>>
>>   - I've noticed from the Vexpress Wiki that the u-boot variable
>>     xen_addr_r is set to 0x80400000, but the xen-uImage is built with
>>     both load and entry addresses as 0x80200000. I also checked the
>>     xen/xen.axf file (from which xen/xen is built), and its entry too is
>>     0x80200000. I'm afraid to ask if this a typo/bit rot in the Wiki?
>
> I'm not sure, but I think in general it is OK for these to differ,
> u-boot will relocate from the load address to the address requested by
> the header as part of booting. So long as it is in RAM and 2MB aligned I
> don't think the actual address matters too much.

the xen-uImage will be loaded at the xen_addr_r, then U-boot will
extract the image and load at 0x802000000.

We don't care about the ELF entry. It's a hack for the model to be able
to directly run Xen. Now we require to boot with either bootwrapper or
U-boot, so I think we can get a rid of it. See:

"# XXX: VE model loads by VMA so instead of
 # making a proper ELF we link with LMA == VMA and adjust crudely
"

Regards,

--
Julien Grall

_______________________________________________
Xen-users mailing list
[hidden email]
http://lists.xen.org/xen-users
Reply | Threaded
Open this post in threaded view
|

Re: ARM: Xen on Vexpress

Julien Grall-2
In reply to this post by Jeenu Viswambharan
On 06/04/2014 11:23 AM, Jeenu Viswambharan wrote:

> On Wed, Jun 04, 2014 at 10:43:38, Ian Campbell wrote:
>> On Tue, 2014-06-03 at 18:11 +0100, Jeenu Viswambharan wrote:
>>> Hi,
>>>
>>> I've built necessary files and setup uboot and board to boot Xen on
>>> Vexpress following the Wiki page. But what I see is u-boot going
>>> silent, presumably after it jumps to Xen. My logs are attached.
>>
>> Jeenu, Which version of Xen are you building?
>
> I'm on Xen RELEASE-4.4.0 tag. I'm using u-boot v2013.10-rc2 tag, but
> I've a feeling that it might be a bit old for Hyp-related stuff.

Looking again to the U-boot code. You are right, HYP support has been
added in v2013.10.

--
Julien Grall

_______________________________________________
Xen-users mailing list
[hidden email]
http://lists.xen.org/xen-users
Reply | Threaded
Open this post in threaded view
|

Re: ARM: Xen on Vexpress

Jeenu Viswambharan
In reply to this post by Julien Grall-2
On Wed, Jun 04, 2014 at 12:41:24, Julien Grall wrote:

> On 06/04/2014 10:43 AM, Ian Campbell wrote:
> > On Tue, 2014-06-03 at 18:11 +0100, Jeenu Viswambharan wrote:
> > > Hi,
> > >
> > > I've built necessary files and setup uboot and board to boot Xen
> > > on Vexpress following the Wiki page. But what I see is u-boot
> > > going silent, presumably after it jumps to Xen. My logs are
> > > attached.
> >
> > Jeenu, Which version of Xen are you building?
> >
> > Skimming that wiki page it seems to be describing how to use an
> > older version of Xen (like 4.3 or early 4.4 development era) and not
> > a modern 4.4 or 4.5 development hypervisor.
>
> The wiki page is based on a Xen 4.4 with a proper multiboot support. I
> don't think there was many change in Xen since the page were written
> (back in september 2013).

So I take it that the Wiki (release tags, build instructions) are all
up-to-date to be used with Xen 4.4.0? I'd have to connect my debugger to
the board and see what's going on... :/

--
Jeenu

-- IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium.  Thank you.

ARM Limited, Registered office 110 Fulbourn Road, Cambridge CB1 9NJ, Registered in England & Wales, Company No:  2557590
ARM Holdings plc, Registered office 110 Fulbourn Road, Cambridge CB1 9NJ, Registered in England & Wales, Company No:  2548782
_______________________________________________
Xen-users mailing list
[hidden email]
http://lists.xen.org/xen-users
Reply | Threaded
Open this post in threaded view
|

Re: ARM: Xen on Vexpress

Julien Grall-2
On 06/04/2014 02:14 PM, Jeenu Viswambharan wrote:

> On Wed, Jun 04, 2014 at 12:41:24, Julien Grall wrote:
>> On 06/04/2014 10:43 AM, Ian Campbell wrote:
>>> On Tue, 2014-06-03 at 18:11 +0100, Jeenu Viswambharan wrote:
>>>> Hi,
>>>>
>>>> I've built necessary files and setup uboot and board to boot Xen
>>>> on Vexpress following the Wiki page. But what I see is u-boot
>>>> going silent, presumably after it jumps to Xen. My logs are
>>>> attached.
>>>
>>> Jeenu, Which version of Xen are you building?
>>>
>>> Skimming that wiki page it seems to be describing how to use an
>>> older version of Xen (like 4.3 or early 4.4 development era) and not
>>> a modern 4.4 or 4.5 development hypervisor.
>>
>> The wiki page is based on a Xen 4.4 with a proper multiboot support. I
>> don't think there was many change in Xen since the page were written
>> (back in september 2013).
>
> So I take it that the Wiki (release tags, build instructions) are all
> up-to-date to be used with Xen 4.4.0? I'd have to connect my debugger to
> the board and see what's going on... :/

Honestly I didn't try to boot Xen on the vexpress for 6 months now. Xen
may have been broken since this date.

Did you try to enable early printk?

Regards,

--
Julien Grall

_______________________________________________
Xen-users mailing list
[hidden email]
http://lists.xen.org/xen-users
Reply | Threaded
Open this post in threaded view
|

Re: ARM: Xen on Vexpress

Jeenu Viswambharan
On Wed, Jun 04, 2014 at 14:16:57, Julien Grall wrote:
> On 06/04/2014 02:14 PM, Jeenu Viswambharan wrote:
> > So I take it that the Wiki (release tags, build instructions) are
> > all up-to-date to be used with Xen 4.4.0? I'd have to connect my
> > debugger to the board and see what's going on... :/
>
> Honestly I didn't try to boot Xen on the vexpress for 6 months now.
> Xen may have been broken since this date.
>
> Did you try to enable early printk?

Yes it helped and I got a bit further, but had to make the following
changes:

  - Move the u-boot tag to v2013.10 (rather than the RC), as Xen
    complained that it must enter the CPU in NS Hyp mode. Clearly RC
    wasn't doing that.

  - Xen was attempting to bring up CPU 2 and stalling there. But I
    believe the idea was to leave A7s in reset and use only 2 A15s. So I
    had to remove the Cortex A7 CPUs from the Linux DTS, and use the DTB
    from there

With those changes, I see Xen going further but hits a data abort. Logs
attached; I'll see if I can dig further.

Meanwhile, if you think the changes I made as above are valid, could you
update the Wiki to reflect them?

--
Jeenu

-- IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium.  Thank you.

ARM Limited, Registered office 110 Fulbourn Road, Cambridge CB1 9NJ, Registered in England & Wales, Company No:  2557590
ARM Holdings plc, Registered office 110 Fulbourn Road, Cambridge CB1 9NJ, Registered in England & Wales, Company No:  2548782

_______________________________________________
Xen-users mailing list
[hidden email]
http://lists.xen.org/xen-users

xen.log.gz (2K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: ARM: Xen on Vexpress

Julien Grall-2
Hi Jeenu,

On 06/04/2014 03:39 PM, Jeenu Viswambharan wrote:

> On Wed, Jun 04, 2014 at 14:16:57, Julien Grall wrote:
>> On 06/04/2014 02:14 PM, Jeenu Viswambharan wrote:
>>> So I take it that the Wiki (release tags, build instructions) are
>>> all up-to-date to be used with Xen 4.4.0? I'd have to connect my
>>> debugger to the board and see what's going on... :/
>>
>> Honestly I didn't try to boot Xen on the vexpress for 6 months now.
>> Xen may have been broken since this date.
>>
>> Did you try to enable early printk?
>
> Yes it helped and I got a bit further, but had to make the following
> changes:
>
>   - Move the u-boot tag to v2013.10 (rather than the RC), as Xen
>     complained that it must enter the CPU in NS Hyp mode. Clearly RC
>     wasn't doing that.

I've updated the wiki page to point to v2013.10.

>   - Xen was attempting to bring up CPU 2 and stalling there. But I
>     believe the idea was to leave A7s in reset and use only 2 A15s. So I
>     had to remove the Cortex A7 CPUs from the Linux DTS, and use the DTB
>     from there

We don't support big.LITTLE on Xen, so we have to drop either the A15s
or A7s.

It will surely need a paragraph in the wiki page about it.

> With those changes, I see Xen going further but hits a data abort. Logs
> attached; I'll see if I can dig further.
>
> Meanwhile, if you think the changes I made as above are valid, could you
> update the Wiki to reflect them?

I think it will be easier if you have an account on the wiki. So you
will be able to modify the page yourself.

Regards,

--
Julien Grall

_______________________________________________
Xen-users mailing list
[hidden email]
http://lists.xen.org/xen-users
Reply | Threaded
Open this post in threaded view
|

Re: ARM: Xen on Vexpress

Julien Grall-2
In reply to this post by Jeenu Viswambharan
On 06/04/2014 03:39 PM, Jeenu Viswambharan wrote:
> With those changes, I see Xen going further but hits a data abort. Logs
> attached; I'll see if I can dig further.

Sorry I forgot to answer to this part... I've copied a part of your log below.

(XEN) CPU0: Unexpected Trap: Data Abort
(XEN) ----[ Xen-4.4.0  arm32  debug=n  Tainted:    C ]----
(XEN) CPU:    0
(XEN) PC:     002470e0 gic_route_irq_to_guest+0x100/0x128

Could you use addr2line -e xen-syms youaddress to get the offending line in the code?

I suspect it's the irq_to_pending... if so I'm not sure why it happens.
Also could you print the IRQ number that Xen is trying to route to dom0?

Regards,

--
Julien Grall

_______________________________________________
Xen-users mailing list
[hidden email]
http://lists.xen.org/xen-users
Reply | Threaded
Open this post in threaded view
|

Re: ARM: Xen on Vexpress

Jeenu Viswambharan
On Thu, Jun 05, 2014 at 15:50:46, Julien Grall wrote:

> On 06/04/2014 03:39 PM, Jeenu Viswambharan wrote:
> > With those changes, I see Xen going further but hits a data abort.
> > Logs attached; I'll see if I can dig further.
>
> Sorry I forgot to answer to this part... I've copied a part of your
> log below.
>
> (XEN) CPU0: Unexpected Trap: Data Abort
> (XEN) ----[ Xen-4.4.0  arm32  debug=n  Tainted:    C ]----
> (XEN) CPU:    0
> (XEN) PC:     002470e0 gic_route_irq_to_guest+0x100/0x128
>
> Could you use addr2line -e xen-syms youaddress to get the offending
> line in the code?
>
> I suspect it's the irq_to_pending... if so I'm not sure why it
> happens.  Also could you print the IRQ number that Xen is trying to
> route to dom0?

It turned out to be a GCC 4.6 issue where the offset of vcpu inside
struct domain was wrongly generated as 0xe4 while it's 0xf8. I almost
forgot the recommendation to use GCC 4.8 in the Wiki. It goes past that
with 2014.04 tool chain from Linaro. My bad!

Xen is now launching DOM0. But next issue I'm now facing is that Linux
refusing to run when loaded at 0xa0008000 - it panics and complains
about failing to allocate memory in mm/memblock.c:memblock_alloc_base().
FWIW, I've built the kernel v3.14 with vexpress_defconfig and enabled
Xen and highmem support. The kernel however is happy to boot when loaded
at 0x80008000 (without Xen). The same with Xen though, the kernel
reports 'uncompression error'.

It's little to do with Xen, but are there any other options required to
be enabled in the DOM0 Kernel to run from 0xa0008000?

Also could you tell me at what point Xen jumps to guest entry? I tried
break at the eret instruction following the label return_to_hypervisor,
but the break point doesn't seem to hit.

Thanks.

--
Jeenu

-- IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium.  Thank you.

ARM Limited, Registered office 110 Fulbourn Road, Cambridge CB1 9NJ, Registered in England & Wales, Company No:  2557590
ARM Holdings plc, Registered office 110 Fulbourn Road, Cambridge CB1 9NJ, Registered in England & Wales, Company No:  2548782
_______________________________________________
Xen-users mailing list
[hidden email]
http://lists.xen.org/xen-users
Reply | Threaded
Open this post in threaded view
|

Re: ARM: Xen on Vexpress

Ian Campbell-10
On Wed, 2014-06-11 at 14:54 +0100, Jeenu Viswambharan wrote:
> It turned out to be a GCC 4.6 issue where the offset of vcpu inside
> struct domain was wrongly generated as 0xe4 while it's 0xf8. I almost
> forgot the recommendation to use GCC 4.8 in the Wiki. It goes past that
> with 2014.04 tool chain from Linaro. My bad!

You mean the contents of asm-offset.h was wrong or something else?

I routinely use Debian's gcc 4.6.3 to build Xen and I've not seen this.
Perhaps it's an issue with your specific build of gcc?

> Xen is now launching DOM0. But next issue I'm now facing is that Linux
> refusing to run when loaded at 0xa0008000 - it panics and complains
> about failing to allocate memory in mm/memblock.c:memblock_alloc_base().
> FWIW, I've built the kernel v3.14 with vexpress_defconfig and enabled
> Xen and highmem support. The kernel however is happy to boot when loaded
> at 0x80008000 (without Xen). The same with Xen though, the kernel
> reports 'uncompression error'.
>
> It's little to do with Xen, but are there any other options required to
> be enabled in the DOM0 Kernel to run from 0xa0008000?

Your kernel needs to be relocatable. I'm not 100% which option that is
(perhaps CONFIG_ARM_PATCH_PHYS_VIRT) but if you build a multiplatform
kernel (CONFIG_ARCH_MULTIPLATFORM) it certainly will be.

> Also could you tell me at what point Xen jumps to guest entry? I tried
> break at the eret instruction following the label return_to_hypervisor,
> but the break point doesn't seem to hit.

It should, that's the right place.

Ian.


_______________________________________________
Xen-users mailing list
[hidden email]
http://lists.xen.org/xen-users
Reply | Threaded
Open this post in threaded view
|

Re: ARM: Xen on Vexpress

Jeenu Viswambharan
On Wed, Jun 11, 2014 at 15:16:12, Ian Campbell wrote:

> On Wed, 2014-06-11 at 14:54 +0100, Jeenu Viswambharan wrote:
> > It turned out to be a GCC 4.6 issue where the offset of vcpu inside
> > struct domain was wrongly generated as 0xe4 while it's 0xf8. I
> > almost forgot the recommendation to use GCC 4.8 in the Wiki. It goes
> > past that with 2014.04 tool chain from Linaro. My bad!
>
> You mean the contents of asm-offset.h was wrong or something else?
>
> I routinely use Debian's gcc 4.6.3 to build Xen and I've not seen this.
> Perhaps it's an issue with your specific build of gcc?

To the call to irq_to_pending in gic_route_irq_to_guest, using GCC
4.8.3 (Linaro 2014.04), I get (*-objdump -S):

    /* TODO: do not assume delivery to vcpu0 */
    p = irq_to_pending(d->vcpu[0], irq->irq);
  24d420:       e59830f8        ldr     r3, [r8, #248]  ; 0xf8
  24d424:       e5930000        ldr     r0, [r3]
  24d428:       e5961000        ldr     r1, [r6]
  24d42c:       eb0020c4        bl      255744 <irq_to_pending>

But the same code with 4.6.3:

    /* TODO: do not assume delivery to vcpu0 */
    p = irq_to_pending(d->vcpu[0], irq->irq);
  237bb4:       e59830e4        ldr     r3, [r8, #228]  ; 0xe4  <- Wrong offset
        return -EBUSY;

    desc->action  = new;
    dsb();

    return 0;
  237bb8:       e1a05002        mov     r5, r2
        xfree(action);
        goto out;
    }

    /* TODO: do not assume delivery to vcpu0 */
    p = irq_to_pending(d->vcpu[0], irq->irq);
  237bbc:       e5961000        ldr     r1, [r6]
  237bc0:       e5930000        ldr     r0, [r3]                <- Faults here
  237bc4:       eb001926        bl      23e064 <irq_to_pending>

4.6.3 is what got installed via. apt-get on my Ubuntu 12.04 PC.
Admittedly I didn't dig further as the new tool chain saved me there.

> > Xen is now launching DOM0. But next issue I'm now facing is that
> > Linux refusing to run when loaded at 0xa0008000 - it panics and
> > complains about failing to allocate memory in
> > mm/memblock.c:memblock_alloc_base().  FWIW, I've built the kernel
> > v3.14 with vexpress_defconfig and enabled Xen and highmem support.
> > The kernel however is happy to boot when loaded at 0x80008000
> > (without Xen). The same with Xen though, the kernel reports
> > 'uncompression error'.
> >
> > It's little to do with Xen, but are there any other options required
> > to be enabled in the DOM0 Kernel to run from 0xa0008000?
>
> Your kernel needs to be relocatable. I'm not 100% which option that is
> (perhaps CONFIG_ARM_PATCH_PHYS_VIRT) but if you build a multiplatform
> kernel (CONFIG_ARCH_MULTIPLATFORM) it certainly will be.

Just found out that both of the configs are enabled in my build, and
I've no idea what's going on.

Like I said, when loaded at 0xa0008000, the kernel didn't boot neither
on bare metal nor Xen. But at 0x80008000, the kernel was happy to boot
bare metal but not with Xen. Does Xen care where the kernel is, as long
as the kernel is happy to execute from wherever it's loaded? If not,
since the latter looks more promising, I was hoping to debug that
instead of the former.

Maybe someone who had seen Xen on Vexpress comment on any special kernel
configs. Julien, perhaps?

> > Also could you tell me at what point Xen jumps to guest entry? I
> > tried break at the eret instruction following the label
> > return_to_hypervisor, but the break point doesn't seem to hit.
>
> It should, that's the right place.

Hm, OK. Good to have that confirmed.

--
Jeenu

-- IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium.  Thank you.

ARM Limited, Registered office 110 Fulbourn Road, Cambridge CB1 9NJ, Registered in England & Wales, Company No:  2557590
ARM Holdings plc, Registered office 110 Fulbourn Road, Cambridge CB1 9NJ, Registered in England & Wales, Company No:  2548782
_______________________________________________
Xen-users mailing list
[hidden email]
http://lists.xen.org/xen-users
Reply | Threaded
Open this post in threaded view
|

Re: ARM: Xen on Vexpress

Ian Campbell-10
On Wed, 2014-06-11 at 16:02 +0100, Jeenu Viswambharan wrote:

> > > Xen is now launching DOM0. But next issue I'm now facing is that
> > > Linux refusing to run when loaded at 0xa0008000 - it panics and
> > > complains about failing to allocate memory in
> > > mm/memblock.c:memblock_alloc_base().  FWIW, I've built the kernel
> > > v3.14 with vexpress_defconfig and enabled Xen and highmem support.
> > > The kernel however is happy to boot when loaded at 0x80008000
> > > (without Xen). The same with Xen though, the kernel reports
> > > 'uncompression error'.
> > >
> > > It's little to do with Xen, but are there any other options required
> > > to be enabled in the DOM0 Kernel to run from 0xa0008000?
> >
> > Your kernel needs to be relocatable. I'm not 100% which option that is
> > (perhaps CONFIG_ARM_PATCH_PHYS_VIRT) but if you build a multiplatform
> > kernel (CONFIG_ARCH_MULTIPLATFORM) it certainly will be.
>
> Just found out that both of the configs are enabled in my build, and
> I've no idea what's going on.
>
> Like I said, when loaded at 0xa0008000, the kernel didn't boot neither
> on bare metal nor Xen. But at 0x80008000, the kernel was happy to boot
> bare metal but not with Xen. Does Xen care where the kernel is, as long
> as the kernel is happy to execute from wherever it's loaded?

The kernel wants to be loaded within 128M of the start of RAM. Does
0xa0008000 meet that? What is loading the kernel there -- I don't think
Xen will (it knows about the 128MB thing). How much RAM does your
vexpress have? Is 0xa0008000 even in RAM on a vexpress? I thought it had
up to 1GB at 0x60000000-0xa0000000.

And just to be clear we are talking about the load address of the kernel
within the dom0 IPA space, aren't we? Not the host address where the
kernel is loaded to be passed to Xen (it doesn't matter what that is
really).

Can you post your boot logs, perhaps they will give me a clue.

Ian.


_______________________________________________
Xen-users mailing list
[hidden email]
http://lists.xen.org/xen-users
Reply | Threaded
Open this post in threaded view
|

Re: ARM: Xen on Vexpress

Julien Grall-2
On 06/11/2014 04:10 PM, Ian Campbell wrote:

> On Wed, 2014-06-11 at 16:02 +0100, Jeenu Viswambharan wrote:
>>>> Xen is now launching DOM0. But next issue I'm now facing is that
>>>> Linux refusing to run when loaded at 0xa0008000 - it panics and
>>>> complains about failing to allocate memory in
>>>> mm/memblock.c:memblock_alloc_base().  FWIW, I've built the kernel
>>>> v3.14 with vexpress_defconfig and enabled Xen and highmem support.
>>>> The kernel however is happy to boot when loaded at 0x80008000
>>>> (without Xen). The same with Xen though, the kernel reports
>>>> 'uncompression error'.
>>>>
>>>> It's little to do with Xen, but are there any other options required
>>>> to be enabled in the DOM0 Kernel to run from 0xa0008000?
>>>
>>> Your kernel needs to be relocatable. I'm not 100% which option that is
>>> (perhaps CONFIG_ARM_PATCH_PHYS_VIRT) but if you build a multiplatform
>>> kernel (CONFIG_ARCH_MULTIPLATFORM) it certainly will be.
>>
>> Just found out that both of the configs are enabled in my build, and
>> I've no idea what's going on.
>>
>> Like I said, when loaded at 0xa0008000, the kernel didn't boot neither
>> on bare metal nor Xen. But at 0x80008000, the kernel was happy to boot
>> bare metal but not with Xen. Does Xen care where the kernel is, as long
>> as the kernel is happy to execute from wherever it's loaded?
>
> The kernel wants to be loaded within 128M of the start of RAM. Does
> 0xa0008000 meet that? What is loading the kernel there -- I don't think
> Xen will (it knows about the 128MB thing). How much RAM does your
> vexpress have? Is 0xa0008000 even in RAM on a vexpress? I thought it had
> up to 1GB at 0x60000000-0xa0000000.

The versatile express memory range starts at 0x8000000. So the address
gave by Jeenu looks ok to me.

> And just to be clear we are talking about the load address of the kernel
> within the dom0 IPA space, aren't we? Not the host address where the
> kernel is loaded to be passed to Xen (it doesn't matter what that is
> really).

I remembered to had a similar issue on the Arndale a while a go. Linux
is unable to allocate memory because either:
        1) Linux is considering your memory is part of highmem
        2) virt-phys is again bogus?

I think we can safely ignore the latter one as Xen 4.4 has a workaround
to avoid it (see commit 6c21cb3 which as been revered on upstream).

For the former one, Linux is checking the memory in sanity_check_meminfo
(arch/arm/mm/mmu.c). You might want to check if you have in your log
message such as "Ignoring RAM"/"Truncating RAM".

Regards,

--
Julien Grall

_______________________________________________
Xen-users mailing list
[hidden email]
http://lists.xen.org/xen-users
Reply | Threaded
Open this post in threaded view
|

Re: ARM: Xen on Vexpress

Jeenu Viswambharan
On Wed, Jun 11, 2014 at 16:39:51, Julien Grall wrote:

> On 06/11/2014 04:10 PM, Ian Campbell wrote:
> > And just to be clear we are talking about the load address of the
> > kernel within the dom0 IPA space, aren't we? Not the host address
> > where the kernel is loaded to be passed to Xen (it doesn't matter
> > what that is really).
>
> > I remembered to had a similar issue on the Arndale a while a go.
> > Linux is unable to allocate memory because either:
> >         1) Linux is considering your memory is part of highmem
> >         2) virt-phys is again bogus?
>
> I think we can safely ignore the latter one as Xen 4.4 has a
> workaround to avoid it (see commit 6c21cb3 which as been revered on
> upstream).
>
> For the former one, Linux is checking the memory in
> sanity_check_meminfo (arch/arm/mm/mmu.c). You might want to check if
> you have in your log message such as "Ignoring RAM"/"Truncating RAM".

Yes, I did have them when compiled without HIGHMEM, but went away when I
enabled it. What is more suited - highmem enabled or disabled?

Did you manage to work this problem around?

--
Jeenu

-- IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium.  Thank you.

ARM Limited, Registered office 110 Fulbourn Road, Cambridge CB1 9NJ, Registered in England & Wales, Company No:  2557590
ARM Holdings plc, Registered office 110 Fulbourn Road, Cambridge CB1 9NJ, Registered in England & Wales, Company No:  2548782
_______________________________________________
Xen-users mailing list
[hidden email]
http://lists.xen.org/xen-users
Reply | Threaded
Open this post in threaded view
|

Re: ARM: Xen on Vexpress

Julien Grall-2
On 06/11/2014 04:49 PM, Jeenu Viswambharan wrote:

> On Wed, Jun 11, 2014 at 16:39:51, Julien Grall wrote:
>> On 06/11/2014 04:10 PM, Ian Campbell wrote:
>>> And just to be clear we are talking about the load address of the
>>> kernel within the dom0 IPA space, aren't we? Not the host address
>>> where the kernel is loaded to be passed to Xen (it doesn't matter
>>> what that is really).
>>
>>> I remembered to had a similar issue on the Arndale a while a go.
>>> Linux is unable to allocate memory because either:
>>>         1) Linux is considering your memory is part of highmem
>>>         2) virt-phys is again bogus?
>>
>> I think we can safely ignore the latter one as Xen 4.4 has a
>> workaround to avoid it (see commit 6c21cb3 which as been revered on
>> upstream).
>>
>> For the former one, Linux is checking the memory in
>> sanity_check_meminfo (arch/arm/mm/mmu.c). You might want to check if
>> you have in your log message such as "Ignoring RAM"/"Truncating RAM".
>
> Yes, I did have them when compiled without HIGHMEM, but went away when I
> enabled it. What is more suited - highmem enabled or disabled?

Does your dom0 boot with highmem disabled?

> Did you manage to work this problem around?

It might be another issue. I don't have any modification in the kernel
for the Arndale.

Regards,

--
Julien Grall

_______________________________________________
Xen-users mailing list
[hidden email]
http://lists.xen.org/xen-users
Reply | Threaded
Open this post in threaded view
|

Re: ARM: Xen on Vexpress

Jeenu Viswambharan
In reply to this post by Ian Campbell-10
On Wed, Jun 11, 2014 at 16:10:46, Ian Campbell wrote:

> On Wed, 2014-06-11 at 16:02 +0100, Jeenu Viswambharan wrote:
> > > > Xen is now launching DOM0. But next issue I'm now facing is that
> > > > Linux refusing to run when loaded at 0xa0008000 - it panics and
> > > > complains about failing to allocate memory in
> > > > mm/memblock.c:memblock_alloc_base().  FWIW, I've built the
> > > > kernel v3.14 with vexpress_defconfig and enabled Xen and highmem
> > > > support.  The kernel however is happy to boot when loaded at
> > > > 0x80008000 (without Xen). The same with Xen though, the kernel
> > > > reports 'uncompression error'.
> > > >
> > > > It's little to do with Xen, but are there any other options
> > > > required to be enabled in the DOM0 Kernel to run from
> > > > 0xa0008000?
> > >
> > > Your kernel needs to be relocatable. I'm not 100% which option
> > > that is (perhaps CONFIG_ARM_PATCH_PHYS_VIRT) but if you build a
> > > multiplatform kernel (CONFIG_ARCH_MULTIPLATFORM) it certainly will
> > > be.
> >
> > Just found out that both of the configs are enabled in my build, and
> > I've no idea what's going on.
> >
> > Like I said, when loaded at 0xa0008000, the kernel didn't boot
> > neither on bare metal nor Xen. But at 0x80008000, the kernel was
> > happy to boot bare metal but not with Xen. Does Xen care where the
> > kernel is, as long as the kernel is happy to execute from wherever
> > it's loaded?
>
> The kernel wants to be loaded within 128M of the start of RAM. Does
> 0xa0008000 meet that? What is loading the kernel there -- I don't
> think Xen will (it knows about the 128MB thing). How much RAM does
> your vexpress have? Is 0xa0008000 even in RAM on a vexpress? I thought
> it had up to 1GB at 0x60000000-0xa0000000.
The TC2 has 1GB RAM from 0x80000000 to 0xc0000000. The instructions on
the Wiki tells u-boot to load it at 0xa0008000 and modifies the DTB
accordingly.

> And just to be clear we are talking about the load address of the
> kernel within the dom0 IPA space, aren't we? Not the host address
> where the kernel is loaded to be passed to Xen (it doesn't matter what
> that is really).

I think, upon guest entry, the PC will be at 0xa0008000, making both IPA
and PA same [1]. Is that what you were asking?

> Can you post your boot logs, perhaps they will give me a clue.

I only have

  (XEN) DOM0: Uncompressing Linux... done, booting the kernel.

at the console. The rest is in the log buffer which I can view through
the debugger. I've attached the best I could muster.

--
Jeenu

[1] http://wiki.xenproject.org/wiki/Xen_ARM_with_Virtualization_Extensions/Vexpress#Set_up_U-Boot_to_boot_via_PXE




-- IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium.  Thank you.

ARM Limited, Registered office 110 Fulbourn Road, Cambridge CB1 9NJ, Registered in England & Wales, Company No:  2557590
ARM Holdings plc, Registered office 110 Fulbourn Road, Cambridge CB1 9NJ, Registered in England & Wales, Company No:  2548782

_______________________________________________
Xen-users mailing list
[hidden email]
http://lists.xen.org/xen-users

memory_dump.xxd.txt (10K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: ARM: Xen on Vexpress

Julien Grall-2
On 06/11/2014 05:12 PM, Jeenu Viswambharan wrote:

> On Wed, Jun 11, 2014 at 16:10:46, Ian Campbell wrote:
>> On Wed, 2014-06-11 at 16:02 +0100, Jeenu Viswambharan wrote:
>>>>> Xen is now launching DOM0. But next issue I'm now facing is that
>>>>> Linux refusing to run when loaded at 0xa0008000 - it panics and
>>>>> complains about failing to allocate memory in
>>>>> mm/memblock.c:memblock_alloc_base().  FWIW, I've built the
>>>>> kernel v3.14 with vexpress_defconfig and enabled Xen and highmem
>>>>> support.  The kernel however is happy to boot when loaded at
>>>>> 0x80008000 (without Xen). The same with Xen though, the kernel
>>>>> reports 'uncompression error'.
>>>>>
>>>>> It's little to do with Xen, but are there any other options
>>>>> required to be enabled in the DOM0 Kernel to run from
>>>>> 0xa0008000?
>>>>
>>>> Your kernel needs to be relocatable. I'm not 100% which option
>>>> that is (perhaps CONFIG_ARM_PATCH_PHYS_VIRT) but if you build a
>>>> multiplatform kernel (CONFIG_ARCH_MULTIPLATFORM) it certainly will
>>>> be.
>>>
>>> Just found out that both of the configs are enabled in my build, and
>>> I've no idea what's going on.
>>>
>>> Like I said, when loaded at 0xa0008000, the kernel didn't boot
>>> neither on bare metal nor Xen. But at 0x80008000, the kernel was
>>> happy to boot bare metal but not with Xen. Does Xen care where the
>>> kernel is, as long as the kernel is happy to execute from wherever
>>> it's loaded?
>>
>> The kernel wants to be loaded within 128M of the start of RAM. Does
>> 0xa0008000 meet that? What is loading the kernel there -- I don't
>> think Xen will (it knows about the 128MB thing). How much RAM does
>> your vexpress have? Is 0xa0008000 even in RAM on a vexpress? I thought
>> it had up to 1GB at 0x60000000-0xa0000000.
>
> The TC2 has 1GB RAM from 0x80000000 to 0xc0000000. The instructions on
> the Wiki tells u-boot to load it at 0xa0008000 and modifies the DTB
> accordingly.

The 0xa0008000 in the u-boot runes is not the address where the kernel
will be loaded in DOM0. You should see this address in Xen log smth like
"Loading zImage at...".

Also, why did you had to modify the DTB? Are you talking about the A7s cpus?

>
>   (XEN) DOM0: Uncompressing Linux... done, booting the kernel.
>
> at the console. The rest is in the log buffer which I can view through
> the debugger. I've attached the best I could muster.

It's a bit hard to read. Can you apply the below patch to your Linux
tree? It will print everything to the Xen console log before the HVC
console has been initialised.

diff --git a/drivers/tty/hvc/hvc_xen.c b/drivers/tty/hvc/hvc_xen.c
index 2dc2831..931c72a 100644
--- a/drivers/tty/hvc/hvc_xen.c
+++ b/drivers/tty/hvc/hvc_xen.c
@@ -630,7 +630,6 @@ void xen_raw_console_write(const char *str)
        ssize_t len = strlen(str);
        int rc = 0;

-       if (xen_domain()) {
                rc = dom0_write_console(0, str, len);
 #ifdef CONFIG_X86
                if (rc == -ENOSYS && xen_hvm_domain())
@@ -642,7 +641,6 @@ outb_print:
                for (i = 0; i < len; i++)
                        outb(str[i], 0xe9);
 #endif
-       }
 }

 void xen_raw_printk(const char *fmt, ...)
diff --git a/kernel/printk/printk.c b/kernel/printk/printk.c
index a45b509..907922b 100644
--- a/kernel/printk/printk.c
+++ b/kernel/printk/printk.c
@@ -45,6 +45,7 @@
 #include <linux/poll.h>
 #include <linux/irq_work.h>
 #include <linux/utsname.h>
+#include <xen/hvc-console.h>

 #include <asm/uaccess.h>

@@ -1571,6 +1572,8 @@ asmlinkage int vprintk_emit(int facility, int level,
                }
        }

+       xen_raw_console_write(text);
+
        if (level == -1)
                level = default_message_loglevel;

Regards,

--
Julien Grall

_______________________________________________
Xen-users mailing list
[hidden email]
http://lists.xen.org/xen-users
Reply | Threaded
Open this post in threaded view
|

Re: ARM: Xen on Vexpress

Ian Campbell-10
In reply to this post by Jeenu Viswambharan
On Wed, 2014-06-11 at 17:12 +0100, Jeenu Viswambharan wrote:

> > The kernel wants to be loaded within 128M of the start of RAM. Does
> > 0xa0008000 meet that? What is loading the kernel there -- I don't
> > think Xen will (it knows about the 128MB thing). How much RAM does
> > your vexpress have? Is 0xa0008000 even in RAM on a vexpress? I thought
> > it had up to 1GB at 0x60000000-0xa0000000.
>
> The TC2 has 1GB RAM from 0x80000000 to 0xc0000000.

Ah, I think I was looking at the older one.

>  The instructions on
> the Wiki tells u-boot to load it at 0xa0008000 and modifies the DTB
> accordingly.

Right, so this is the host load address which I was talking about
before. This doesn't have any relationship to the address where Xen will
place the kernel in the dom0 address space.

> > And just to be clear we are talking about the load address of the
> > kernel within the dom0 IPA space, aren't we? Not the host address
> > where the kernel is loaded to be passed to Xen (it doesn't matter what
> > that is really).
>
> I think, upon guest entry, the PC will be at 0xa0008000, making both IPA
> and PA same [1]. Is that what you were asking?

On guest entry the PC will be at the start of the kernel image, at
whatever address Xen has loaded it. Xen will load the dom0 kernel near
the start of dom0's RAM. Xen allocates dom0 RAM using a 1:1 mapping.
Perhaps Xen has allocated addresses starting at 0xa0000000 to dom0?
(that seems unlikely if you have loaded the kenrel module to pass to Xen
to that address though, since they would then clash)

> > Can you post your boot logs, perhaps they will give me a clue.
>
> I only have
>
>   (XEN) DOM0: Uncompressing Linux... done, booting the kernel.

It's all the Xen output from before this which I am interested in. Since
that answers the questions I have about where Xen has placed guest RAM
etc.

Ian.


_______________________________________________
Xen-users mailing list
[hidden email]
http://lists.xen.org/xen-users
1234