Discussion:
Bug#868082: linux-image-4.11.0-1-686: fails to boot on i386 (Soekris net5501)
Add Reply
Francesco Poli (wintermute)
2017-07-11 21:40:03 UTC
Reply
Permalink
Raw Message
Package: src:linux
Version: 4.11.6-1
Severity: important

Hello,
after upgrading an i386 box (Soekris net5501) running testing (buster),
I found out that the newly migrated Linux kernel fails to boot.

Looking at the boot through the serial console reveals the following
few messages:

Booting 'Debian GNU/Linux, with Linux 4.11.0-1-686'

Loading Linux 4.11.0-1-686 ...
Loading initial ramdisk ...

and then nothing else.

The previous kernel image (4.9.0-3-686) works fine.
I have currently altered GRUB_DEFAULT in /etc/default/grub, in order to
force the machine to automatically boot that kernel version.

I hope that this bug may be fixed soon.
Thanks to the Debian Kernel Team for any help they may provide.




-- Package-specific info:

** Model information

** PCI devices:
00:01.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] CS5536 [Geode companion] Host Bridge [1022:2080] (rev 33)
Subsystem: Advanced Micro Devices, Inc. [AMD] CS5536 [Geode companion] Host Bridge [1022:2080]
Control: I/O+ Mem- BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap- 66MHz+ UDF- FastB2B- ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 248, Cache Line Size: 32 bytes

00:01.2 Entertainment encryption device [1010]: Advanced Micro Devices, Inc. [AMD] Geode LX AES Security Block [1022:2082]
Subsystem: Advanced Micro Devices, Inc. [AMD] Geode LX AES Security Block [1022:2082]
Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap- 66MHz+ UDF- FastB2B- ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0, Cache Line Size: 32 bytes
Interrupt: pin A routed to IRQ 10
Region 0: Memory at a0000000 (32-bit, non-prefetchable) [size=16K]
Kernel driver in use: Geode LX AES
Kernel modules: geode_rng, geode_aes

00:06.0 Ethernet controller [0200]: VIA Technologies, Inc. VT6105M [Rhine-III] [1106:3053] (rev 96)
Subsystem: VIA Technologies, Inc. VT6105M [Rhine-III] [1106:0106]
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV+ VGASnoop- ParErr- Stepping- SERR+ FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 64 (750ns min, 2000ns max), Cache Line Size: 32 bytes
Interrupt: pin A routed to IRQ 11
Region 0: I/O ports at e100 [size=256]
Region 1: Memory at a0004000 (32-bit, non-prefetchable) [size=256]
Capabilities: <access denied>
Kernel driver in use: via-rhine
Kernel modules: via_rhine

00:07.0 Ethernet controller [0200]: VIA Technologies, Inc. VT6105M [Rhine-III] [1106:3053] (rev 96)
Subsystem: VIA Technologies, Inc. VT6105M [Rhine-III] [1106:0106]
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV+ VGASnoop- ParErr- Stepping- SERR+ FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 64 (750ns min, 2000ns max), Cache Line Size: 32 bytes
Interrupt: pin A routed to IRQ 5
Region 0: I/O ports at e200 [size=256]
Region 1: Memory at a0004100 (32-bit, non-prefetchable) [size=256]
Capabilities: <access denied>
Kernel driver in use: via-rhine
Kernel modules: via_rhine

00:08.0 Ethernet controller [0200]: VIA Technologies, Inc. VT6105M [Rhine-III] [1106:3053] (rev 96)
Subsystem: VIA Technologies, Inc. VT6105M [Rhine-III] [1106:0106]
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV+ VGASnoop- ParErr- Stepping- SERR+ FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 64 (750ns min, 2000ns max), Cache Line Size: 32 bytes
Interrupt: pin A routed to IRQ 9
Region 0: I/O ports at e300 [size=256]
Region 1: Memory at a0004200 (32-bit, non-prefetchable) [size=256]
Capabilities: <access denied>
Kernel driver in use: via-rhine
Kernel modules: via_rhine

00:09.0 Ethernet controller [0200]: VIA Technologies, Inc. VT6105M [Rhine-III] [1106:3053] (rev 96)
Subsystem: VIA Technologies, Inc. VT6105M [Rhine-III] [1106:0106]
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV+ VGASnoop- ParErr- Stepping- SERR+ FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 64 (750ns min, 2000ns max), Cache Line Size: 32 bytes
Interrupt: pin A routed to IRQ 12
Region 0: I/O ports at e400 [size=256]
Region 1: Memory at a0004300 (32-bit, non-prefetchable) [size=256]
Capabilities: <access denied>
Kernel driver in use: via-rhine
Kernel modules: via_rhine

00:14.0 ISA bridge [0601]: Advanced Micro Devices, Inc. [AMD] CS5536 [Geode companion] ISA [1022:2090] (rev 03)
Subsystem: Advanced Micro Devices, Inc. [AMD] CS5536 [Geode companion] ISA [1022:2090]
Control: I/O+ Mem- BusMaster- SpecCycle+ MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap- 66MHz+ UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Region 0: I/O ports at 6000 [size=8]
Region 1: I/O ports at 6100 [size=256]
Region 2: I/O ports at 6200 [size=64]
Kernel driver in use: cs5535-mfd
Kernel modules: cs5535_mfd

00:14.2 IDE interface [0101]: Advanced Micro Devices, Inc. [AMD] CS5536 [Geode companion] IDE [1022:209a] (rev 01) (prog-if 80 [Master])
Subsystem: Advanced Micro Devices, Inc. [AMD] CS5536 [Geode companion] IDE [1022:209a]
Control: I/O+ Mem- BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap- 66MHz+ UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0, Cache Line Size: 32 bytes
Region 0: [virtual] Memory at 000001f0 (32-bit, non-prefetchable) [size=8]
Region 1: [virtual] Memory at 000003f0 (type 3, non-prefetchable)
Region 2: [virtual] Memory at 00000170 (32-bit, non-prefetchable) [size=8]
Region 3: [virtual] Memory at 00000370 (type 3, non-prefetchable)
Region 4: I/O ports at e000 [size=16]
Kernel driver in use: pata_amd
Kernel modules: pata_amd, pata_cs5536, ata_generic

00:15.0 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] CS5536 [Geode companion] OHC [1022:2094] (rev 02) (prog-if 10 [OHCI])
Subsystem: Advanced Micro Devices, Inc. [AMD] CS5536 [Geode companion] OHC [1022:2094]
Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap+ 66MHz+ UDF- FastB2B- ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0, Cache Line Size: 32 bytes
Interrupt: pin A routed to IRQ 15
Region 0: Memory at a0005000 (32-bit, non-prefetchable) [size=4K]
Kernel driver in use: ohci-pci
Kernel modules: ohci_pci

00:15.1 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] CS5536 [Geode companion] EHC [1022:2095] (rev 02) (prog-if 20 [EHCI])
Subsystem: Advanced Micro Devices, Inc. [AMD] CS5536 [Geode companion] EHC [1022:2095]
Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap+ 66MHz+ UDF- FastB2B- ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0, Cache Line Size: 32 bytes
Interrupt: pin A routed to IRQ 15
Region 0: Memory at a0006000 (32-bit, non-prefetchable) [size=4K]
Kernel driver in use: ehci-pci
Kernel modules: ehci_pci


** USB devices:
Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Bus 002 Device 001: ID 1d6b:0001 Linux Foundation 1.1 root hub


-- System Information:
Debian Release: buster/sid
APT prefers testing
APT policy: (800, 'testing'), (500, 'unstable')
Architecture: i386 (i586)

Kernel: Linux 4.9.0-3-686 (SMP w/1 CPU core)
Locale: LANG=en_US.UTF-8, LC_CTYPE=en_US.UTF-8 (charmap=UTF-8), LANGUAGE=en_US:en (charmap=UTF-8)
Shell: /bin/sh linked to /bin/dash
Init: sysvinit (via /sbin/init)

Versions of packages linux-image-4.11.0-1-686 depends on:
ii initramfs-tools [linux-initramfs-tool] 0.130
ii kmod 24-1
ii linux-base 4.5

Versions of packages linux-image-4.11.0-1-686 recommends:
ii firmware-linux-free 3.4
ii irqbalance 1.1.0-2.3

Versions of packages linux-image-4.11.0-1-686 suggests:
pn debian-kernel-handbook <none>
ii grub-pc 1.99-17
pn linux-doc-4.11 <none>

Versions of packages linux-image-4.11.0-1-686 is related to:
pn firmware-amd-graphics <none>
pn firmware-atheros <none>
pn firmware-bnx2 <none>
pn firmware-bnx2x <none>
pn firmware-brcm80211 <none>
pn firmware-cavium <none>
pn firmware-intel-sound <none>
pn firmware-intelwimax <none>
pn firmware-ipw2x00 <none>
pn firmware-ivtv <none>
pn firmware-iwlwifi <none>
pn firmware-libertas <none>
pn firmware-linux-nonfree <none>
pn firmware-misc-nonfree <none>
pn firmware-myricom <none>
pn firmware-netxen <none>
pn firmware-qlogic <none>
pn firmware-realtek <none>
pn firmware-samsung <none>
pn firmware-siano <none>
pn firmware-ti-connectivity <none>
pn xen-hypervisor <none>

-- no debconf information
Ben Hutchings
2017-07-16 21:40:02 UTC
Reply
Permalink
Raw Message
Control: tag -1 moreinfo
Post by Francesco Poli (wintermute)
Package: src:linux
Version: 4.11.6-1
Severity: important
Hello,
after upgrading an i386 box (Soekris net5501) running testing (buster),
I found out that the newly migrated Linux kernel fails to boot.
Looking at the boot through the serial console reveals the following
  Booting 'Debian GNU/Linux, with Linux 4.11.0-1-686'
  Loading Linux 4.11.0-1-686 ...
  Loading initial ramdisk ...
and then nothing else.
[...]

If you add 'earlyprintk=ttyS0' to the kernel command line and delete
'quiet', does it log anything?

Ben.
--
Ben Hutchings
If the facts do not conform to your theory, they must be disposed of.
Francesco Poli
2017-07-17 08:10:02 UTC
Reply
Permalink
Raw Message
On Sun, 16 Jul 2017 22:27:10 +0100 Ben Hutchings wrote:

[...]
Post by Ben Hutchings
If you add 'earlyprintk=ttyS0' to the kernel command line and delete
'quiet', does it log anything?
Hello Ben,
thanks for your follow up.

I have just tried with 'earlyprintk=ttyS0' and without 'quiet', but the
result seems to be the same:

Booting a command list

Loading Linux 4.11.0-1-686 ...
Loading initial ramdisk ...

and then nothing else.

I am not removing the "moreinfo" tag from the bug report, since you
could need some further information... Please let me know.

Thanks for your time!
--
http://www.inventati.org/frx/
There's not a second to spare! To the laboratory!
..................................................... Francesco Poli .
GnuPG key fpr == CA01 1147 9CD2 EFDF FB82 3925 3E1C 27E1 1F69 BFFE
Francesco Poli
2017-08-26 11:40:01 UTC
Reply
Permalink
Raw Message
Control: found -1 linux/4.12.6-1
Post by Francesco Poli
[...]
Post by Ben Hutchings
If you add 'earlyprintk=ttyS0' to the kernel command line and delete
'quiet', does it log anything?
Hello Ben,
thanks for your follow up.
I have just tried with 'earlyprintk=ttyS0' and without 'quiet', but the
Booting a command list
Loading Linux 4.11.0-1-686 ...
Loading initial ramdisk ...
and then nothing else.
Hello again,
I tested the new kernel version that has recently migrated to Debian
testing.

Unfortunately, I get the same exact boot failure.

And no useful output on the serial console, even with
'earlyprintk=ttyS0' and without 'quiet':

Booting a command list

Loading Linux 4.12.0-1-686 ...
Loading initial ramdisk ...

and then nothing else.


Please note that, after the test, I again booted the machine with the
old Linux 4.9.0-3-686 and noticed "recovering journal" for the root
partition among the boot messages.
Hence, I suppose the root filesystem was mounted in read/write mode
during the failed boot of Linux 4.12.0-1-686.
I don't know whether this piece of information may be useful in order
to pinpoint the issue...

Please let me know, in case you need any further information and/or in
case there's some progress on this bug.

Thanks for your time!

Bye.
--
http://www.inventati.org/frx/
There's not a second to spare! To the laboratory!
..................................................... Francesco Poli .
GnuPG key fpr == CA01 1147 9CD2 EFDF FB82 3925 3E1C 27E1 1F69 BFFE
Francesco Poli
2017-09-24 16:00:02 UTC
Reply
Permalink
Raw Message
On Sat, 26 Aug 2017 13:02:18 +0200 Francesco Poli wrote:

Control: found -1 linux/4.12.13-1
[...]
Post by Francesco Poli
Hello again,
I tested the new kernel version that has recently migrated to Debian
testing.
Unfortunately, I get the same exact boot failure.
And no useful output on the serial console, even with
Booting a command list
Loading Linux 4.12.0-1-686 ...
Loading initial ramdisk ...
and then nothing else.
Another kernel version has just migrated to Debian testing.
Another boot failure with no useful output on the serial console, even
with 'earlyprintk=ttyS0' and without 'quiet':

Booting a command list

Loading Linux 4.12.0-2-686 ...
Loading initial ramdisk ...

and nothing else.

Is there anything that can be done in order to pinpoint the issue and
solve it once and for all?!?

Please let me know, thanks for your time.
--
http://www.inventati.org/frx/
There's not a second to spare! To the laboratory!
..................................................... Francesco Poli .
GnuPG key fpr == CA01 1147 9CD2 EFDF FB82 3925 3E1C 27E1 1F69 BFFE
Francesco Poli
2017-09-24 17:00:01 UTC
Reply
Permalink
Raw Message
Control: found -1 linux/4.12.13-1

[I should have put the above control command at the beginning of my
previous message... I am doing so now.]
--
http://www.inventati.org/frx/
There's not a second to spare! To the laboratory!
..................................................... Francesco Poli .
GnuPG key fpr == CA01 1147 9CD2 EFDF FB82 3925 3E1C 27E1 1F69 BFFE
Ben Hutchings
2017-09-24 19:30:01 UTC
Reply
Permalink
Raw Message
Post by Francesco Poli
Control: found -1 linux/4.12.13-1
[...]
Post by Francesco Poli
Hello again,
I tested the new kernel version that has recently migrated to Debian
testing.
Unfortunately, I get the same exact boot failure.
And no useful output on the serial console, even with
Booting a command list
Loading Linux 4.12.0-1-686 ...
Loading initial ramdisk ...
and then nothing else.
Another kernel version has just migrated to Debian testing.
Another boot failure with no useful output on the serial console, even
Booting a command list
Loading Linux 4.12.0-2-686 ...
Loading initial ramdisk ...
and nothing else.
Is there anything that can be done in order to pinpoint the issue and
solve it once and for all?!?
You could test the 4.10 kernel version from snapshot.debian.org, which
would narrow down the regresion a bit.

Ben.
Post by Francesco Poli
Please let me know, thanks for your time.
--
Ben Hutchings
Make three consecutive correct guesses and you will be considered an
expert.
Francesco Poli
2017-10-02 22:10:01 UTC
Reply
Permalink
Raw Message
Control: found -1 linux/4.11-1~exp1
[...]
Post by Ben Hutchings
Post by Francesco Poli
Is there anything that can be done in order to pinpoint the issue and
solve it once and for all?!?
You could test the 4.10 kernel version from snapshot.debian.org, which
would narrow down the regresion a bit.
OK, here's the result of the test, after some bisecting:

• linux-image-4.10.0-trunk-686-unsigned_4.10.7-1~exp1_i386.deb

boots correctly

$ uname -a
Linux $HOSTNAME 4.10.0-trunk-686 #1 SMP Debian 4.10.7-1~exp1 (2017-03-30) i586 GNU/Linux


• linux-image-4.11.0-trunk-686_4.11-1~exp1_i386.deb

fails to boot

Booting a command list

Loading Linux 4.11.0-trunk-686 ...
Loading initial ramdisk ...



I hope this helps in pinpointing the cause of the issue.
--
http://www.inventati.org/frx/
There's not a second to spare! To the laboratory!
..................................................... Francesco Poli .
GnuPG key fpr == CA01 1147 9CD2 EFDF FB82 3925 3E1C 27E1 1F69 BFFE
Francesco Poli
2017-10-12 22:50:11 UTC
Reply
Permalink
Raw Message
Control: found -1 linux/4.13.4-1


On Mon, 2 Oct 2017 23:26:40 +0200 Francesco Poli wrote:

[...]
Post by Francesco Poli
• linux-image-4.10.0-trunk-686-unsigned_4.10.7-1~exp1_i386.deb
boots correctly
$ uname -a
Linux $HOSTNAME 4.10.0-trunk-686 #1 SMP Debian 4.10.7-1~exp1 (2017-03-30) i586 GNU/Linux
• linux-image-4.11.0-trunk-686_4.11-1~exp1_i386.deb
fails to boot
Booting a command list
Loading Linux 4.11.0-trunk-686 ...
Loading initial ramdisk ...
I hope this helps in pinpointing the cause of the issue.
For the record, I have just tried with the latest Linux kernel in
testing:

Loading Linux 4.13.0-1-686 ...
Loading initial ramdisk ...


Another boot failure...

Is there any progress in the bug investigation?
--
http://www.inventati.org/frx/
There's not a second to spare! To the laboratory!
..................................................... Francesco Poli .
GnuPG key fpr == CA01 1147 9CD2 EFDF FB82 3925 3E1C 27E1 1F69 BFFE
R.
2017-09-19 18:20:01 UTC
Reply
Permalink
Raw Message
Hello,

I'm experiencing the same problem on Soekris net5501, also with the latest
testing 4.12.0.1-686 kernel : the boot process gets stuck on "Loading
initial ramdisk".

The 4.9.0.3-686 boots correctly.

Regards
Post by Francesco Poli
Control: found -1 linux/4.12.6-1
Post by Francesco Poli
[...]
Post by Ben Hutchings
If you add 'earlyprintk=ttyS0' to the kernel command line and delete
'quiet', does it log anything?
Hello Ben,
thanks for your follow up.
I have just tried with 'earlyprintk=ttyS0' and without 'quiet', but the
Booting a command list
Loading Linux 4.11.0-1-686 ...
Loading initial ramdisk ...
and then nothing else.
Hello again,
I tested the new kernel version that has recently migrated to Debian
testing.
Unfortunately, I get the same exact boot failure.
And no useful output on the serial console, even with
Booting a command list
Loading Linux 4.12.0-1-686 ...
Loading initial ramdisk ...
and then nothing else.
Please note that, after the test, I again booted the machine with the
old Linux 4.9.0-3-686 and noticed "recovering journal" for the root
partition among the boot messages.
Hence, I suppose the root filesystem was mounted in read/write mode
during the failed boot of Linux 4.12.0-1-686.
I don't know whether this piece of information may be useful in order
to pinpoint the issue...
Please let me know, in case you need any further information and/or in
case there's some progress on this bug.
Thanks for your time!
Bye.
--
http://www.inventati.org/frx/
There's not a second to spare! To the laboratory!
Ben Hutchings
2017-10-12 22:30:16 UTC
Reply
Permalink
Raw Message
On Wed, 2017-10-11 at 23:30 +0200, Francesco Poli wrote:
[...]
For the record, I have just tried with the latest Linux kernel in
Loading Linux 4.13.0-1-686 ...
Loading initial ramdisk ...
Another boot failure...
Is there any progress in the bug investigation?
I don't think anyone's investigating this, I'm afraid.

Going back to the lack of output with 'earlyprintk=ttyS0', I had a
check and the default bit rate for that is 9600(!) whereas I (and maybe
you) would have expected 115200. So perhaps you could try
'earlyprintk=ttyS0,115200' (or whichever bit rate you want).

Ben.
--
Ben Hutchings
Man invented language to satisfy his deep need to complain. - Lily
Tomlin
Francesco Poli
2017-10-14 18:00:01 UTC
Reply
Permalink
Raw Message
Post by Ben Hutchings
[...]
For the record, I have just tried with the latest Linux kernel in
Loading Linux 4.13.0-1-686 ...
Loading initial ramdisk ...
Another boot failure...
Is there any progress in the bug investigation?
I don't think anyone's investigating this, I'm afraid.
Huh? What was point of having me bisecting among versions
uploaded to Debian experimental, then?
Post by Ben Hutchings
Going back to the lack of output with 'earlyprintk=ttyS0', I had a
check and the default bit rate for that is 9600(!) whereas I (and maybe
you) would have expected 115200. So perhaps you could try
'earlyprintk=ttyS0,115200' (or whichever bit rate you want).
I have just tried with 'earlyprintk=ttyS0,19200,n8' (and without
'quiet'), since I have 'console=ttyS0,19200,n8' among the normal
parameters.

Unfortunately, I still get no useful debug output at all:

Booting a command list

Loading Linux 4.13.0-1-686 ...
Loading initial ramdisk ...
--
http://www.inventati.org/frx/
There's not a second to spare! To the laboratory!
..................................................... Francesco Poli .
GnuPG key fpr == CA01 1147 9CD2 EFDF FB82 3925 3E1C 27E1 1F69 BFFE
Francesco Poli
2017-10-22 16:40:02 UTC
Reply
Permalink
Raw Message
Control: found -1 linux/4.13.4-2


On Sat, 14 Oct 2017 19:51:57 +0200 Francesco Poli wrote:

[...]
Post by Francesco Poli
I have just tried with 'earlyprintk=ttyS0,19200,n8' (and without
'quiet'), since I have 'console=ttyS0,19200,n8' among the normal
parameters.
Booting a command list
Loading Linux 4.13.0-1-686 ...
Loading initial ramdisk ...
I have just tried with the new kernel currently in testing, but I got
the same exact result (no boot and no debug output)...
--
http://www.inventati.org/frx/
There's not a second to spare! To the laboratory!
..................................................... Francesco Poli .
GnuPG key fpr == CA01 1147 9CD2 EFDF FB82 3925 3E1C 27E1 1F69 BFFE
Loading...