Issue:
When starting Clockspring you receive this error:
Exception in thread "main" java.lang.NoClassDefFoundError: com/fasterxml/jackson/core/JsonParseException at org.apache.nifi.bootstrap.RunNiFi.start(RunNiFi.java:955) at org.apache.nifi.bootstrap.RunNiFi.main(RunNiFi.java:242) Caused by: java.lang.ClassNotFoundException: com.fasterxml.jackson.core.JsonParseException at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:581) at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:178) at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:522) ... 2 more
or possibly this error:
2021-09-10 07:20:38,504 ERROR [main] org.apache.nifi.NiFi Failure to launch NiFi due to java.util.ServiceConfigurationError: org.apache.nifi.processor.Processor: Provider org.apache.nifi.processors.hive.PutHiveStreaming could not be instantiated java.util.ServiceConfigurationError: org.apache.nifi.processor.Processor: Provider org.apache.nifi.processors.hive.PutHiveStreaming could not be instantiated at java.base/java.util.ServiceLoader.fail(ServiceLoader.java:582) at java.base/java.util.ServiceLoader$ProviderImpl.newInstance(ServiceLoader.java:804) at java.base/java.util.ServiceLoader$ProviderImpl.get(ServiceLoader.java:722) at java.base/java.util.ServiceLoader$3.next(ServiceLoader.java:1395) at org.apache.nifi.nar.StandardExtensionDiscoveringManager.loadExtensions(StandardExtensionDiscoveringManager.java:156) at org.apache.nifi.nar.StandardExtensionDiscoveringManager.discoverExtensions(StandardExtensionDiscoveringManager.java:131) at org.apache.nifi.nar.StandardExtensionDiscoveringManager.discoverExtensions(StandardExtensionDiscoveringManager.java:117) at org.apache.nifi.web.server.JettyServer.start(JettyServer.java:1042) at org.apache.nifi.NiFi.<init>(NiFi.java:158) at org.apache.nifi.NiFi.<init>(NiFi.java:72) at org.apache.nifi.NiFi.main(NiFi.java:301) Caused by: org.xerial.snappy.SnappyError: [FAILED_TO_LOAD_NATIVE_LIBRARY] null at org.xerial.snappy.SnappyLoader.load(SnappyLoader.java:239) at org.xerial.snappy.Snappy.<clinit>(Snappy.java:48) at org.apache.nifi.processors.hive.PutHiveStreaming.<clinit>(PutHiveStreaming.java:156) at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:490) at java.base/java.util.ServiceLoader$ProviderImpl.newInstance(ServiceLoader.java:780) ... 9 common frames omitted 2021-09-10 07:20:38,513 INFO [Thread-0] org.apache.nifi.NiFi Initiating shutdown of Jetty web server... 2021-09-10 07:20:38,514 INFO [Thread-0] org.apache.nifi.NiFi Jetty web server shutdown completed (nicely or otherwise).
Cause:
In RedHat/CentOS 8 the DISA STIG Guide has the server configured with the fapolicyd package. fapolicyd enables application blacklisting and whitelisting across your Linux platform to ensure that rouge applications are not present on your system. This is a security best practice, but does require additional configuration options for non-standard applications. This is generally associated with SELinux running which can be checked by using 'getenforce' command.
Resolution:
Clockspring must be whitelisted within the fapolicyd application to ensure things are running smoothly.
TL;DR - Sometimes you just want the answer, so add the following rule to your /etc/fapolicyd/fapolicyd.rules file above rule #7 and restart the fapolicyd service. NOTE: The path to java listed after exe below may be different in your environment
# Allow Clockspring to use Java allow perm=open exe=/usr/lib/jvm/java-11-openjdk-11.0.12.0.7-0.el8_4.x86_64/bin/java : all # Only allow known ELF libs - this is ahead of executable because typical # executable is linked with a dozen or more libraries. allow perm=open all : ftype=application/x-sharedlib trust=1 deny_audit perm=open all : ftype=application/x-sharedlib
Ensure that this rule is placed before the x-sharedlib deny rule (as shown above) that is causing the deny action.
For those that want to better understand the impacts fapolicyd has on their Clockspring environment follow these steps:
- Stop the fapolicyd service from running in the background so that we can enable debug. This step is imporant as if you try to start the debug service while the current service is running you may freeze your system and have to perform a hard reboot.
$ sudo systemctl stop fapolicyd
- Open a second terminal to your Clockspring server as we'll need to run fapolicyd in the foreground. We'll refer to these sessions as Session A and Session B
- On Session Astart the fapolicyd service into failure debug mode to find what executable is being denied and the associated deny rule
$ sudo fapolicyd --debug
- On Session B attempt to start the Clockspring service. We'll use the servicecommand to start Clockspring so that we can see error messages on the terminal rather than having them suppressed by systemctl.
$ sudo service clockspring start Java home: Clockspring home: /opt/clockspring Bootstrap Config File: /opt/clockspring/conf/bootstrap.conf Exception in thread "main" java.lang.NoClassDefFoundError: com/fasterxml/jackson/core/JsonParseException at org.apache.nifi.bootstrap.RunNiFi.start(RunNiFi.java:955) at org.apache.nifi.bootstrap.RunNiFi.main(RunNiFi.java:242) Caused by: java.lang.ClassNotFoundException: com.fasterxml.jackson.core.JsonParseException at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:581) at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:178) at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:522) ... 2 more
- Session Ashould show all denies from fapolicyd similar to:
Starting to listen for events rule=12 dec=deny_audit perm=open auid=-1 pid=2301 exe=/usr/lib/jvm/java-11-openjdk-11.0.12.0.7-0.el8_4.x86_64/bin/java : path=/opt/clockspring/lib/bootstrap/bcprov-jdk15on-1.66.jar ftype=application/java-archive rule=12 dec=deny_audit perm=open auid=-1 pid=2301 exe=/usr/lib/jvm/java-11-openjdk-11.0.12.0.7-0.el8_4.x86_64/bin/java : path=/opt/clockspring/lib/bootstrap/bcpkix-jdk15on-1.66.jar ftype=application/java-archive rule=12 dec=deny_audit perm=open auid=-1 pid=2301 exe=/usr/lib/jvm/java-11-openjdk-11.0.12.0.7-0.el8_4.x86_64/bin/java : path=/opt/clockspring/lib/bootstrap/bcrypt-0.9.0.jar ftype=application/java-archive
This shows us that the /usr/lib/jvm/java-11-openjdk-11.0.12.0.7-0.el8_4.x86_64/bin/java package was denied by Rule #12
Hit ctrl-c on Session A to stop the fapolicyd service that is running in the foreground but do not close the terminal as we'll need data from the debug output to create our rule - We now know that Rule #12 is blocking Clockspring from starting, but we need to deterime what Rule #12 is. To do this we can use the fapolicyd-cli --listcommand
$ sudo fapolicyd-cli --list 1. allow perm=any uid=0 : dir=/var/tmp/ 2. allow perm=any uid=0 trust=1 : all 3. deny_audit perm=any pattern=ld_so : all 4. allow perm=open exe=/usr/bin/rpm : all 5. allow perm=open exe=/usr/libexec/platform-python3.6 comm=dnf : all 6. allow perm=open all : ftype=application/x-sharedlib trust=1 7. deny_audit perm=open all : ftype=application/x-sharedlib 8. allow perm=execute all : trust=1 9. allow perm=any uid=0 : dir=/tmp/ansible 10. allow perm=any uid=0 : dir=/root/.ansible/tmp/ 11. allow perm=open all : ftype=%languages trust=1 12. deny_audit perm=any all : ftype=%languages 13. allow perm=any all : ftype=text/x-shellscript 14. deny_audit perm=execute all : all 15. allow perm=open all : all 16. deny perm=any all : all
This output shows us that Rule #12 is deny_audit perm=any all : ftype=%languages so we must add our allow rule before that rule runs - Edit the /etc/fapolicyd/fapolicyd.rules file to add our custom allow rule. Start by finding the current deny rule and add a new rule above this rule:
$ sudo vi /etc/fapolicyd/fapolicyd.rules ... # Allow Clockspring to use Java allow perm=open exe=/usr/lib/jvm/java-11-openjdk-11.0.12.0.7-0.el8_4.x86_64/bin/java : all # Allow any program to open trusted language files allow perm=open all : ftype=%languages trust=1 deny_audit perm=any all : ftype=%languages ...
Save and quit from your editor and repeat Steps 3-5 above to ensure the rule is working. It is not uncommon for an application to be denied initially on one rule when performing specific actions, then later the application creates a new command which could still be blocked by yet another rule.
Performing this test again will show that Clockspring is denied later in the boot process by rule #7. Modify the /etc/fapolicyd/fapolicyd.rules file again and move the new rule above Rule #7.$ sudo vi /etc/fapolicyd/fapolicyd.rules ... # Allow Clockspring to use Java allow perm=open exe=/usr/lib/jvm/java-11-openjdk-11.0.12.0.7-0.el8_4.x86_64/bin/java : all # Only allow known ELF libs - this is ahead of executable because typical # executable is linked with a dozen or more libraries. allow perm=open all : ftype=application/x-sharedlib trust=1 deny_audit perm=open all : ftype=application/x-sharedlib ...
- Repeat Steps 3-5 again to ensure the Clockspring fully starts and runs with no additional denies.
- Once no more denies are found the fapolicyd service can be started and Clockspring will run as expected.
$ sudo systemctl start fapolicyd