[patch] Improve Wikipedia tag validators

[nyurik]

Wikipedia tags need a few more validators with fixers:

• update the list of valid Wikipedia languages
• auto-replace language be-x-old with be-tarask
• auto-replace language cz with cs
• remove space before page title ("en: Foo" -> "en:Foo")
• replace "_" with spaces ("en:Foo_bar" -> "en:Foo bar")
• capitalize first letter if it's not "jbo" language ("en:foo" -> "en:Foo")
• double language link ("en:en:Foo" -> "en:Foo")

This patch also adds a few additional validator functions: trim, upper and lower

[nyurik]

patch to fix this issue

[nyurik]

overrides previous patch

[Don-vip]

Please don't write this:

 	29	  throwWarning: tr("wikipedia ''be-x-old'' language is obsolete, use ''be-tarask'' instead");
 	36	  throwWarning: tr("wikipedia ''cz'' language is invalid, use ''cs'' instead");

This makes an entry to translate for each value we will list in an individual check. Instead, this only creates a generic string to translate:

 	29	  throwWarning: tr("wikipedia ''{0}'' language is obsolete, use ''{1}'' instead", "be-x-old", "be-tarask");
 	36	  throwWarning: tr("wikipedia ''{0}'' language is invalid, use ''{1}'' instead", "cz", "cs");

I didn't check the other strings, you see the point :)

[nyurik]

@Don-vip, please recheck. I so wish we used git for this. Attaching and revising patches like this is very arcane.

[nyurik]

@don-vip, is something still missing for this ticket to be merged?

[Don-vip]

Sorry I forgot this one I'll review it tonight :)

[Don-vip]

In 11756/josm:

fix #14425 - Improve Wikipedia tag validators (patch by nyurik, modified)

[Don-vip]

Thanks for the patch! :)
However for the next time, please test it with JOSM first, I had to fix many syntax issues (wrong quote characters, missing semicolons, etc.)

[Klumbumbus]

"wikipedia tag is set, but there is no wikidata tag. Wikipedia plugin might help with wikidata id lookups" does this really fit into the category warning? wikidata tag seems to me an optional tag.

[Don-vip]

In 11771/josm:

fix #14425 - reduce wikidata warning to info level

[Klumbumbus]

@nyurik: please add documentation for the new functions at wiki:Help/Styles/MapCSSImplementation

[Don-vip]

I didn't notice that the assertions are not met:

GRAVE: Expecting test 'wikipedia tag has no language given, use 'wikipedia'='language:page title'' (i.e., [*[wikipedia][wikipedia'NREGEX'^[-a-z]{2,12}:]]) to not match node wikipedia=en-GB:Foobar (i.e., TagMap[wikipedia=en-GB:Foobar])
GRAVE: Expecting test 'wikipedia tag has an unknown language prefix' (i.e., [*[wikipedia'REGEX'^[-a-z]{2,12}:][wikipedia'NREGEX'^https?:\/\/][wikipedia'NREGEX'^(aa|ab|ace|ady|ady-cyrl|aeb|aeb-arab|aeb-latn|af|ak|aln|als|am|an|ang|anp|ar|arc|arn|arq|ary|arz|as|ase|ast|av|avk|awa|ay|az|azb|ba|ban|bar|bat-smg|bbc|bbc-latn|bcc|bcl|be|be-tarask|be-x-old|bg|bgn|bh|bho|bi|bjn|bm|bn|bo|bpy|bqi|br|brh|bs|bto|bug|bxr|ca|cbk-zam|cdo|ce|ceb|ch|cho|chr|chy|ckb|co|cps|cr|crh|crh-cyrl|crh-latn|cs|csb|cu|cv|cy|cz|da|de|de-at|de-ch|de-formal|din|diq|dsb|dtp|dty|dv|dz|ee|egl|el|eml|en|en-ca|en-gb|eo|es|et|eu|ext|fa|ff|fi|fit|fiu-vro|fj|fo|fr|frc|frp|frr|fur|fy|ga|gag|gan|gan-hans|gan-hant|gd|gl|glk|gn|gom|gom-deva|gom-latn|gor|got|grc|gsw|gu|gv|ha|hak|haw|he|hi|hif|hif-latn|hil|ho|hr|hrx|hsb|ht|hu|hy|hz|ia|id|ie|ig|ii|ik|ike-cans|ike-latn|ilo|inh|io|is|it|iu|ja|jam|jbo|jut|jv|ka|kaa|kab|kbd|kbd-cyrl|kea|kg|khw|ki|kiu|kj|kk|kk-arab|kk-cn|kk-cyrl|kk-kz|kk-latn|kk-tr|kl|km|kn|ko|ko-kp|koi|kr|krc|kri|krj|krl|ks|ks-arab|ks-deva|ksh|ku|ku-arab|ku-latn|kv|kw|ky|la|lad|lb|lbe|lez|lfn|lg|li|lij|liv|lki|lmo|ln|lo|loz|lrc|lt|ltg|lus|luz|lv|lzh|lzz|mai|map-bms|mdf|mg|mh|mhr|mi|min|mk|ml|mn|mo|mr|mrj|ms|mt|mus|mwl|my|myv|mzn|na|nah|nan|nap|nb|nds|nds-nl|ne|new|ng|niu|nl|nl-informal|nn|no|nod|nov|nrm|nso|nv|ny|nys|oc|olo|om|or|os|ota|pa|pag|pam|pap|pcd|pdc|pdt|pfl|pi|pih|pl|pms|pnb|pnt|prg|ps|pt|pt-br|qu|qug|rgn|rif|rm|rmy|rn|ro|roa-rup|roa-tara|ru|rue|rup|ruq|ruq-cyrl|ruq-latn|rw|rwr|sa|sah|sat|sc|scn|sco|sd|sdc|sdh|se|sei|ses|sg|sgs|sh|shi|shi-latn|shi-tfng|shn|si|simple|sje|sk|sl|sli|sm|sma|smj|sn|so|sq|sr|sr-ec|sr-el|srn|srq|ss|st|stq|su|sv|sw|szl|ta|tcy|te|tet|tg|tg-cyrl|tg-latn|th|ti|tk|tl|tly|tn|to|tokipona|tpi|tr|tru|ts|tt|tt-cyrl|tt-latn|tum|tw|ty|tyv|tzm|udm|ug|ug-arab|ug-latn|uk|ur|uz|uz-cyrl|uz-latn|ve|vec|vep|vi|vls|vmf|vo|vot|vro|wa|war|wo|wuu|xal|xh|xmf|yi|yo|yue|za|zea|zh|zh-classical|zh-cn|zh-hans|zh-hant|zh-hk|zh-min-nan|zh-mo|zh-my|zh-sg|zh-tw|zh-yue|zu):]]) to match node wikipedia=X-Y-Z:Foobar (i.e., TagMap[wikipedia=X-Y-Z:Foobar])
GRAVE: Expecting test 'wikipedia title should not have 'wiki/' prefix' (i.e., [*[wikipedia'REGEX'^[-a-z]{2,12}:wiki\/]]) to match node wikipedia=en: foo (i.e., TagMap[wikipedia=en: foo])

Can you please submit a patch that fix them? To check it you can simply run MapCSSTagCheckerTest.testInit() as JUnit test.

[bastiK]

In 11800/josm:

fixed #14425 - fix wikipedia tag validator regexes

[Klumbumbus]

In 12415/josm:

fix #14947, see #14425 - fix wikipedia warning message