I tried a different process for the coloring again. This time also keeping the lineart in the final and after finishing it actually looks good.
