Using the following script to compare output from asciidoc vs asciidoc3:

import difflib
import os

for entry in os.listdir('tests/data'):
    if not entry.endswith('html'):
        continue
    with open('tests/data/' + entry, 'r') as new, open('../asciidoc/tests/data/' + entry, 'r') as old:
        lines = []
        for line in difflib.unified_diff(old.readlines(), new.readlines(), n=0):
            lines.append(line)
        if len(lines) > 0:
            print(entry)
            with open('/tmp/' + entry, 'w') as open_file:
                for line in lines:
                    print(line, file=open_file)

Which gives:

lang-ja-book-test-html5.html
lang-ja-book-test-html4.html
lang-ja-book-test-xhtml11.html
lang-ja-article-test-xhtml11.html
lang-ja-article-test-html4.html
lang-ja-article-test-html5.html

due to the above regex for indexes.

So the reason the JA tests are failing is that the indexterm2 regex is (?<!\()[\\]?\(\((?P<attrlist>[^\s\(][^\(].*?)\)\)(?!\)) with the key bit being in the middle: ((?P[^\s\\(][^\\(].*?))) which implies that an index term must be greater than two characters (emphasized). It was working originally because the one JA character for monkey, 猿, was being internally represented as \xe7\x8c\xbf by Python2 and so the regex was matching it. However, now on Python3, it's properly internally represented as the one character and the regex fails. Fixing it requires removing the necessary second character to give just ((?P[^\s\\(].*?))). Putting it together gives (?<!\()[\\]?\(\((?P<attrlist>[^\s\(].*?)\)\)(?!\)).

I couldn't find any explicit mention in the asciidocs (either for asciidoc or asciidoctor, asciidoctor also seems to allow single character indexterms) that makes mention of minimum character length for an index term so I can't tell why this is, so I've committed the above change, which by all accounts brings asciidoc3 to match asciidoc output on all testcases.

I've modified my above test script to be stricter now to check and ensure output is byte indentical:

import os

for entry in os.listdir('tests/data'):
    if not entry.endswith('html'):
        continue
    with open('tests/data/' + entry, 'rb') as new, open('../asciidoc/tests/data/' + entry, 'rb') as old:
        if new.read() != old.read():
            print(entry)

which I'm happy to report it is and Travis-CI reports no failures for any test cases.

@elextr: I think this should now be ready for merging?

@MasterOdin thanks for continuing this work, hopefully will be able to progress it in the next few days.

Possibly the repo will be renamed as part of the first initial preparation for Asciidoc Python 2 EOL. IIUC github redirects links to the original repo to the new name, so your fork should still work.

I am pleasantly surprised you only found one regex that needed changing due to the difference between Python 2 ASCII and Python3 Unicode. I was expecting many more.

Once its committed users will be encouraged to test it for real.

Yeah, it was good to see that the only regexes that really needed modification wasn't even really for something that was failing for a "true" unicode issue. If anything, the regexes are now guaranteed to be more stable and easier to write (as evidenced below) due to not having to worry about characters being expanded to codepoints.

Turns out I made a mistake in my comparison (and forgot to compare docbooks).

It's left me with these failures:

lang-ja-book-test-docbook.xml
newtables-docbook.xml

The failures for the lang-ja-book-test-docbook.xml is:

---

+++

@@ -16 +16 @@

-<chapter id="_前書">

+<preface id="_前書">

@@ -19,2 +19,2 @@

-</chapter>

-<chapter id="_奥付">

+</preface>

+<colophon id="_奥付">

@@ -23 +23 @@

-</chapter>

+</colophon>

However, this looks like it's a bug in asciidoc2 (which is the - lines) as it wasn't properly matching:
https://github.com/asciidoc/asciidoc3/blob/0bea48486a7265ae0c4cc792ab041df95ca142f2/lang-ja.conf#L48-L50
against the Title attribute, due to the fact that the ? in the regexes was only being applied to the final character of the code point instead of the entire character in Python2.

Still investigating newtables-docbook.xml which has a bunch of columns with a colwidth of 42* instead of 43*. Not sure what's up there.

IIUC github redirects links to the original repo to the new name, so your fork should still work.

Yeah, Github redirects the original name to the new name so long as you don't make a repository at the original name. Not sure what you're referring to with 'my fork' or renaming though.

And fixed newtables-docbook.xml. Did not know that round() had changed from Python2 to Python3, so added a shim to get that functionality back for use for calculating column widths for docbook. With that, generated testcases are byte identical between asciidoc2 and asciidoc3 with the noted exception of lang-ja-book-test-docbook.xml, but that's a bug with asciidoc2 and its interaction of code points inside the JA regex string.

Note, PR title changed to "Continue ..." because I was getting confused about two PRs with the same title, nearly applied them in the wrong order :)