Commit graph

5 commits

Author SHA1 Message Date
Thierry Reding
b11d5c025d scripts/spdxcheck.py: always open files in binary mode
commit 3a6ab5c7dc114057fd67750e308e1745dafc0e6a upstream.

The spdxcheck script currently falls over when confronted with a binary
file (such as Documentation/logo.gif).  To avoid that, always open files
in binary mode and decode line-by-line, ignoring encoding errors.

One tricky case is when piping data into the script and reading it from
standard input.  By default, standard input will be opened in text mode,
so we need to reopen it in binary mode.

The breakage only happens with python3 and results in a
UnicodeDecodeError (according to Uwe).

Link: http://lkml.kernel.org/r/20181212131210.28024-1-thierry.reding@gmail.com
Fixes: 6f4d29df66ac ("scripts/spdxcheck.py: make python3 compliant")
Signed-off-by: Thierry Reding <treding@nvidia.com>
Reviewed-by: Jeremy Cline <jcline@redhat.com>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: Jonathan Corbet <corbet@lwn.net>
Cc: Joe Perches <joe@perches.com>
Cc: Uwe Kleine-König <u.kleine-koenig@pengutronix.de>
Cc: <stable@vger.kernel.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
2018-12-19 19:19:50 +01:00
Uwe Kleine-König
ccbc11b384 scripts/spdxcheck.py: make python3 compliant
commit 6f4d29df66acd49303a99025046b85cabe7aa17a upstream.

Without this change the following happens when using Python3 (3.6.6):

	$ echo "GPL-2.0" | python3 scripts/spdxcheck.py -
	FAIL: 'str' object has no attribute 'decode'
	Traceback (most recent call last):
	  File "scripts/spdxcheck.py", line 253, in <module>
	    parser.parse_lines(sys.stdin, args.maxlines, '-')
	  File "scripts/spdxcheck.py", line 171, in parse_lines
	    line = line.decode(locale.getpreferredencoding(False), errors='ignore')
	AttributeError: 'str' object has no attribute 'decode'

So as the line is already a string, there is no need to decode it and
the line can be dropped.

/usr/bin/python on Arch is Python 3.  So this would indeed be worth
going into 4.19.

Link: http://lkml.kernel.org/r/20181023070802.22558-1-u.kleine-koenig@pengutronix.de
Signed-off-by: Uwe Kleine-König <u.kleine-koenig@pengutronix.de>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: Joe Perches <joe@perches.com>
Cc: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
Cc: <stable@vger.kernel.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
2018-11-21 09:19:25 +01:00
Jeremy Cline
bed95c43c1 scripts: add Python 3 compatibility to spdxcheck.py
"dict.has_key(key)" on dictionaries has been replaced with "key in
dict".  Additionally, when run under Python 3 some files don't decode
with the default encoding (tested with UTF-8).  To handle that, don't
open the file in text mode and decode text line-by-line, ignoring
encoding errors.

This remains compatible with Python 2 and should have no functional
change.

Link: http://lkml.kernel.org/r/20180717190635.29467-1-jcline@redhat.com
Signed-off-by: Jeremy Cline <jcline@redhat.com>
Acked-by: Thomas Gleixner <tglx@linutronix.de>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2018-08-17 16:20:27 -07:00
Joe Perches
fde5e903fb scripts/spdxcheck.py: work with current HEAD LICENSES/ directory
Depending on how old your -next tree is, it may not have a master that
has the LICENSES directory.

Change the lookup to HEAD and find whatever LICENSE directory files are
used in that branch.

Miscellanea:

 - Remove the checkpatch test as it will have its own SPDX license
   identifier.

Link: http://lkml.kernel.org/r/7eeefc862194930c773e662cb2152e178441d3b8.camel@perches.com
Signed-off-by: Joe Perches <joe@perches.com>
Reviewed-by: Thomas Gleixner <tglx@linutronix.de>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2018-08-17 16:20:27 -07:00
Thomas Gleixner
5385a295ec scripts: Add SPDX checker script
The SPDX-License-Identifiers are growing in the kernel and so grow
expression failures and license IDs are used which have no corresponding
license text file in the LICENSES directory.

Add a script which gathers information from the LICENSES directory,
i.e. the various tags in the licenses and exception files and then scans
either input from stdin, which it treats as a single file or if started
without arguments it scans the full kernel tree.

It checks whether the license expression syntax is correct and also
validates whether the license identifiers used in the expressions are
available in the LICENSES files.

scripts/spdxcheck.py -h
usage: spdxcheck.py [-h] [-m MAXLINES] [-v] [path [path ...]]

SPDX expression checker

positional arguments:
  path                  Check path or file. If not given full git tree scan.
                        For stdin use "-"

optional arguments:
  -h, --help            show this help message and exit
  -m MAXLINES, --maxlines MAXLINES
                        Maximum number of lines to scan in a file. Default 15
  -v, --verbose         Verbose statistics output

include/dt-bindings/reset/amlogic,meson-axg-reset.h: 9:41 Invalid License ID: BSD

drivers/pinctrl/sh-pfc/pfc-r8a77965.c: 1:28 Invalid License ID: GPL-2.
include/dt-bindings/reset/amlogic,meson-axg-reset.h: 9:41 Invalid License ID: BSD

arch/x86/kernel/jailhouse.c: 1:28 Invalid License ID: GPL2.0
include/dt-bindings/reset/amlogic,meson-axg-reset.h: 9:41 Invalid License ID: BSD

arch/arm/mach-s3c24xx/h1940-bluetooth.c: 1:28 Invalid License ID: GPL-1.0
arch/x86/kernel/jailhouse.c: 1:28 Invalid License ID: GPL2.0
drivers/pinctrl/sh-pfc/pfc-r8a77965.c: 1:28 Invalid License ID: GPL-2.
include/dt-bindings/reset/amlogic,meson-axg-reset.h: 9:41 Invalid License ID: BSD
arch/x86/include/asm/jailhouse_para.h: 1:28 Invalid License ID: GPL2.0

arch/arm/mach-s3c24xx/h1940-bluetooth.c: 1:28 Invalid License ID: GPL-1.0
arch/x86/kernel/jailhouse.c: 1:28 Invalid License ID: GPL2.0
drivers/pinctrl/sh-pfc/pfc-r8a77965.c: 1:28 Invalid License ID: GPL-2.
include/dt-bindings/reset/amlogic,meson-axg-reset.h: 9:41 Invalid License ID: BSD
arch/x86/include/asm/jailhouse_para.h: 1:28 Invalid License ID: GPL2.0

License files:               14
Exception files:              1
License IDs                  19
Exception IDs                 1

Files checked:            61332
Lines checked:           669181
Files with SPDX:          16169
Files with errors:            5

real	0m2.642s
user	0m2.231s
sys	0m0.467s

That's a full tree sweep on my laptop. Note, this runs single threaded.

It scans by default the first 15 lines for a SPDX identifier where the
current max inside a top comment is at line 10. But that's going to be
faster once the identifiers are all in the first two lines as documented.

The python wizards will surely know how to do that smarter and faster, but
its at least better than no tool at all.

Signed-off-by: Thomas Gleixner <tglx@linutronix.de>
Reviewed-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
[jc: Fixed ironically erroneous SPDX tag and did chmod +x ]
Signed-off-by: Jonathan Corbet <corbet@lwn.net>
2018-04-27 16:45:49 -06:00