Tag: zeppelin

a few notes before we lunch an emr cluster with zeppelin:

• Zeppelin is installed on the master node of the EMR cluster ( choose the right installation for you )

• Don’t forget to add the AWS glue connectors
• Don’t forget to add Spark …
• full documentations: https://zeppelin.apache.org/docs/0.7.3/
• ML notebook example with zeppelinhttps://raw.githubusercontent.com/hortonworks-gallery/zeppelin-notebooks/hdp-2.6/2CCBNZ5YY/note.json

If you want, you can automate the above process by using an EMR step. Please find attached a simple shell script that will download your zeppelin-site.xml file from S3 onto your EMR cluster and restart the Zeppelin service.

To run it, simply copy the script to an S3 bucket and then use the script-runner.jar process as outlined in [2] below with the script s3 location as its only argument.

To do this via the AWS EMR Console:

1  – Under the “Add steps (optional)” section, select “Custom JAR” for the “Step type” and click the “Configure” button.

2 – In the pop-up window, for us-east-1 the JAR location for script-runner.jar is:

s3://us-east-1.elasticmapreduce/libs/script-runner/script-runner.jar

3 – For the argument, you would pass in your S3 bucket and location of the “setupZeppelin.sh” file e.g.,:

s3://mybucket/mylocation/setupZeppelin.sh

Once done, click “Add” and continue on with your EMR cluster creation (this process is included when cloning an EMR cluster).

Zeppelin Config optimisation

• In the interpreter
  • zeppelin.interpreter.output.limit 1024 instead 102400
  • zeppelin.spark.maxResult 100 instead of 1000
• In zeppelin-env.sh
  sudo nano /etc/zeppelin/conf.dist/zeppelin-env.sh
  • export ZEPPELIN_MEM=”-Xms44024m -Xmx46024m -XX:MaxPermSize=512m”
  • export ZEPPELIN_INTP_MEM=”-Xms44024m -Xmx46024m -XX:MaxPermSize=512m”

Saving your Zeppelin notebooks on the amazon s3 storage for durability:

https://aws.amazon.com/blogs/big-data/running-an-external-zeppelin-instance-using-s3-backed-notebooks-with-spark-on-amazon-emr/

Example of config zeppelin-site.xml

<?xml version=”1.0″?>
<?xml-stylesheet type=”text/xsl” href=”configuration.xsl”?>
<!–
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
this work for additional information regarding copyright ownership.
The ASF licenses this file to You under the Apache License, Version 2.0
(the “License”); you may not use this file except in compliance with
the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an “AS IS” BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
–>

<configuration>

<property>
<name>zeppelin.server.addr</name>
<value>0.0.0.0</value>
<description>Server address</description>
</property>

<property>
<name>zeppelin.server.port</name>
<value>8080</value>
<description>Server port.</description>
</property>

<property>
<name>zeppelin.server.ssl.port</name>
<value>8890</value>
<description>Server ssl port. (used when ssl property is set to true)</description>
</property>

<property>
<name>zeppelin.server.context.path</name>
<value>/</value>
<description>Context Path of the Web Application</description>
</property>

<property>
<name>zeppelin.war.tempdir</name>
<value>webapps</value>
<description>Location of jetty temporary directory</description>
</property>

<property>
<name>zeppelin.notebook.dir</name>
<value>notebook</value>
<description>path or URI for notebook persist</description>
</property>

<property>
<name>zeppelin.notebook.homescreen</name>
<value></value>
<description>id of notebook to be displayed in homescreen. ex) 2A94M5J1Z Empty value displays default home screen</description>
</property>

<property>
<name>zeppelin.notebook.homescreen.hide</name>
<value>false</value>
<description>hide homescreen notebook from list when this value set to true</description>
</property>

 

<!– Amazon S3 notebook storage –>
<!– Creates the following directory structure: s3://{bucket}/{username}/{notebook-id}/note.json –>
<property>
<name>zeppelin.notebook.s3.user</name>
<value>user</value>
<description>user name for s3 folder structure</description>
</property>

<property>
<name>zeppelin.notebook.s3.bucket</name>
<value>my-zeppelin-bucket</value>
<description>bucket name for notebook storage</description>
</property>

<property>
<name>zeppelin.notebook.s3.endpoint</name>
<value>s3.amazonaws.com</value>
<description>endpoint for s3 bucket</description>
</property>

<property>
<name>zeppelin.notebook.storage</name>
<value>org.apache.zeppelin.notebook.repo.S3NotebookRepo</value>
<description>notebook persistence layer implementation</description>
</property>
–>

<!– Additionally, encryption is supported for notebook data stored in S3 –>
<!– Use the AWS KMS to encrypt data –>
<!– If used, the EC2 role assigned to the EMR cluster must have rights to use the given key –>
<!– See https://aws.amazon.com/kms/ and http://docs.aws.amazon.com/kms/latest/developerguide/concepts.html –>
<!–
<property>
<name>zeppelin.notebook.s3.kmsKeyID</name>
<value>AWS-KMS-Key-UUID</value>
<description>AWS KMS key ID used to encrypt notebook data in S3</description>
</property>
–>

<!– provide region of your KMS key –>
<!– See http://docs.aws.amazon.com/general/latest/gr/rande.html#kms_region for region codes names –>
<!–
<property>
<name>zeppelin.notebook.s3.kmsKeyRegion</name>
<value>us-east-1</value>
<description>AWS KMS key region in your AWS account</description>
</property>
–>

<!– Use a custom encryption materials provider to encrypt data –>
<!– No configuration is given to the provider, so you must use system properties or another means to configure –>
<!– See https://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/com/amazonaws/services/s3/model/EncryptionMaterialsProvider.html –>
<!–
<property>
<name>zeppelin.notebook.s3.encryptionMaterialsProvider</name>
<value>provider implementation class name</value>
<description>Custom encryption materials provider used to encrypt notebook data in S3</description>
</property>
–>

 

<!– If using Azure for storage use the following settings –>
<!–
<property>
<name>zeppelin.notebook.azure.connectionString</name>
<value>DefaultEndpointsProtocol=https;AccountName=<accountName>;AccountKey=<accountKey></value>
<description>Azure account credentials</description>
</property>

<property>
<name>zeppelin.notebook.azure.share</name>
<value>zeppelin</value>
<description>share name for notebook storage</description>
</property>

<property>
<name>zeppelin.notebook.azure.user</name>
<value>user</value>
<description>optional user name for Azure folder structure</description>
</property>

<property>
<name>zeppelin.notebook.storage</name>
<value>org.apache.zeppelin.notebook.repo.AzureNotebookRepo</value>
<description>notebook persistence layer implementation</description>
</property>
–>

<!– Notebook storage layer using local file system
<property>
<name>zeppelin.notebook.storage</name>
<value>org.apache.zeppelin.notebook.repo.VFSNotebookRepo</value>
<description>local notebook persistence layer implementation</description>
</property>
–>

<!– For connecting your Zeppelin with ZeppelinHub –>
<!–
<property>
<name>zeppelin.notebook.storage</name>
<value>org.apache.zeppelin.notebook.repo.GitNotebookRepo, org.apache.zeppelin.notebook.repo.zeppelinhub.ZeppelinHubRepo</value>
<description>two notebook persistence layers (versioned local + ZeppelinHub)</description>
</property>
–>

<property>
<name>zeppelin.notebook.storage</name>
<value>org.apache.zeppelin.notebook.repo.GitNotebookRepo</value>
<description>versioned notebook persistence layer implementation</description>
</property>

<property>
<name>zeppelin.notebook.one.way.sync</name>
<value>false</value>
<description>If there are multiple notebook storages, should we treat the first one as the only source of truth?</description>
</property>

<property>
<name>zeppelin.interpreter.dir</name>
<value>interpreter</value>
<description>Interpreter implementation base directory</description>
</property>

<property>
<name>zeppelin.interpreter.localRepo</name>
<value>local-repo</value>
<description>Local repository for interpreter’s additional dependency loading</description>
</property>

<property>
<name>zeppelin.interpreter.dep.mvnRepo</name>
<value>http://repo1.maven.org/maven2/</value&gt;
<description>Remote principal repository for interpreter’s additional dependency loading</description>
</property>

<property>
<name>zeppelin.dep.localrepo</name>
<value>local-repo</value>
<description>Local repository for dependency loader</description>
</property>

<property>
<name>zeppelin.helium.npm.registry</name>
<value>http://registry.npmjs.org/</value&gt;
<description>Remote Npm registry for Helium dependency loader</description>
</property>

<property>
<name>zeppelin.interpreters</name>
<value>org.apache.zeppelin.spark.SparkInterpreter,org.apache.zeppelin.spark.PySparkInterpreter,org.apache.zeppelin.rinterpreter.RRepl,org.apache.zeppelin.rinterpreter.KnitR,org.apache.zeppelin.spark.SparkRInterpreter,org.apache.zeppelin.spark.SparkSqlInterpreter,org.apache.zeppelin.spark.DepInterpreter,org.apache.zeppelin.markdown.Markdown,org.apache.zeppelin.angular.AngularInterpreter,org.apache.zeppelin.shell.ShellInterpreter,org.apache.zeppelin.file.HDFSFileInterpreter,org.apache.zeppelin.flink.FlinkInterpreter,,org.apache.zeppelin.python.PythonInterpreter,org.apache.zeppelin.python.PythonInterpreterPandasSql,org.apache.zeppelin.python.PythonCondaInterpreter,org.apache.zeppelin.python.PythonDockerInterpreter,org.apache.zeppelin.lens.LensInterpreter,org.apache.zeppelin.ignite.IgniteInterpreter,org.apache.zeppelin.ignite.IgniteSqlInterpreter,org.apache.zeppelin.cassandra.CassandraInterpreter,org.apache.zeppelin.geode.GeodeOqlInterpreter,org.apache.zeppelin.postgresql.PostgreSqlInterpreter,org.apache.zeppelin.jdbc.JDBCInterpreter,org.apache.zeppelin.kylin.KylinInterpreter,org.apache.zeppelin.elasticsearch.ElasticsearchInterpreter,org.apache.zeppelin.scalding.ScaldingInterpreter,org.apache.zeppelin.alluxio.AlluxioInterpreter,org.apache.zeppelin.hbase.HbaseInterpreter,org.apache.zeppelin.livy.LivySparkInterpreter,org.apache.zeppelin.livy.LivyPySparkInterpreter,org.apache.zeppelin.livy.LivyPySpark3Interpreter,org.apache.zeppelin.livy.LivySparkRInterpreter,org.apache.zeppelin.livy.LivySparkSQLInterpreter,org.apache.zeppelin.bigquery.BigQueryInterpreter,org.apache.zeppelin.beam.BeamInterpreter,org.apache.zeppelin.pig.PigInterpreter,org.apache.zeppelin.pig.PigQueryInterpreter,org.apache.zeppelin.scio.ScioInterpreter</value>
<description>Comma separated interpreter configurations. First interpreter become a default</description>
</property>

<property>
<name>zeppelin.interpreter.group.order</name>
<value>spark,md,angular,sh,livy,alluxio,file,psql,flink,python,ignite,lens,cassandra,geode,kylin,elasticsearch,scalding,jdbc,hbase,bigquery,beam</value>
<description></description>
</property>

<property>
<name>zeppelin.interpreter.connect.timeout</name>
<value>30000</value>
<description>Interpreter process connect timeout in msec.</description>
</property>

<property>
<name>zeppelin.interpreter.output.limit</name>
<value>10240</value>
<description>Output message from interpreter exceeding the limit will be truncated</description>
</property>

<property>
<name>zeppelin.ssl</name>
<value>true</value>
<description>Should SSL be used by the servers?</description>
</property>

<property>
<name>zeppelin.ssl.client.auth</name>
<value>false</value>
<description>Should client authentication be used for SSL connections?</description>
</property>

<property>
<name>zeppelin.ssl.keystore.path</name>
<value>/home/hadoop/certificate.p12</value>
<description>Path to keystore relative to Zeppelin configuration directory</description>
</property>

<property>
<name>zeppelin.ssl.keystore.type</name>
<value>PKCS12</value>
<description>The format of the given keystore (e.g. JKS or PKCS12)</description>
</property>

<property>
<name>zeppelin.ssl.keystore.password</name>
<value>MyPassword</value>
<description>Keystore password. Can be obfuscated by the Jetty Password tool</description>
</property>

<!–
<property>
<name>zeppelin.ssl.key.manager.password</name>
<value>change me</value>
<description>Key Manager password. Defaults to keystore password. Can be obfuscated.</description>
</property>
–>

<property>
<name>zeppelin.ssl.truststore.path</name>
<value>truststore</value>
<description>Path to truststore relative to Zeppelin configuration directory. Defaults to the keystore path</description>
</property>

<property>
<name>zeppelin.ssl.truststore.type</name>
<value>JKS</value>
<description>The format of the given truststore (e.g. JKS or PKCS12). Defaults to the same type as the keystore type</description>
</property>

<!–
<property>
<name>zeppelin.ssl.truststore.password</name>
<value>change me</value>
<description>Truststore password. Can be obfuscated by the Jetty Password tool. Defaults to the keystore password</description>
</property>
–>

<property>
<name>zeppelin.server.allowed.origins</name>
<value>*</value>
<description>Allowed sources for REST and WebSocket requests (i.e. http://onehost:8080,http://otherhost.com). If you leave * you are vulnerable to https://issues.apache.org/jira/browse/ZEPPELIN-173</description&gt;
</property>

<property>
<name>zeppelin.anonymous.allowed</name>
<value>false</value>
<description>Anonymous user allowed by default</description>
</property>

<property>
<name>zeppelin.notebook.public</name>
<value>false</value>
<description>Make notebook public by default when created, private otherwise</description>
</property>

<property>
<name>zeppelin.websocket.max.text.message.size</name>
<value>1024000</value>
<description>Size in characters of the maximum text message to be received by websocket. Defaults to 1024000</description>
</property>

<property>
<name>zeppelin.server.default.dir.allowed</name>
<value>false</value>
<description>Enable directory listings on server.</description>
</property>

<!–
<property>
<name>zeppelin.server.jetty.name</name>
<value>Jetty(7.6.0.v20120127)</value>
<description>Hardcoding Application Server name to Prevent Fingerprinting</description>
</property>
–>
<!–
<property>
<name>zeppelin.server.xframe.options</name>
<value>SAMEORIGIN</value>
<description>The X-Frame-Options HTTP response header can be used to indicate whether or not a browser should be allowed to render a page in a frame/iframe/object.</description>
</property>
–>

<!–
<property>
<name>zeppelin.server.strict.transport</name>
<value>max-age=631138519</value>
<description>The HTTP Strict-Transport-Security response header is a security feature that lets a web site tell browsers that it should only be communicated with using HTTPS, instead of using HTTP. Enable this when Zeppelin is running on HTTPS. Value is in Seconds, the default value is equivalent to 20 years.</description>
</property>
–>
<!–
<property>
<name>zeppelin.server.xxss.protection</name>
<value>1</value>
<description>The HTTP X-XSS-Protection response header is a feature of Internet Explorer, Chrome and Safari that stops pages from loading when they detect reflected cross-site scripting (XSS) attacks. When value is set to 1 and a cross-site scripting attack is detected, the browser will sanitize the page (remove the unsafe parts).</description>
</property>
–>
</configuration>

Shiri.ini example

# Licensed to the Apache Software Foundation (ASF) under one or more

# contributor license agreements. See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the “License”); you may not use this file except in compliance with
# the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an “AS IS” BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#

[users]
# List of users with their password allowed to access Zeppelin.
# To use a different strategy (LDAP / Database / …) check the shiro doc at http://shiro.apache.org/configuration.html#Configuration-INISections

Omid = BigDataNinjaForever, admin

# Sample LDAP configuration, for user Authentication, currently tested for single Realm
[main]
### A sample for configuring Active Directory Realm
#activeDirectoryRealm = org.apache.zeppelin.realm.ActiveDirectoryGroupRealm
#activeDirectoryRealm.systemUsername = userNameA

#use either systemPassword or hadoopSecurityCredentialPath, more details in http://zeppelin.apache.org/docs/latest/security/shiroauthentication.html
#activeDirectoryRealm.systemPassword = passwordA
#activeDirectoryRealm.hadoopSecurityCredentialPath = jceks://file/user/zeppelin/zeppelin.jceks
#activeDirectoryRealm.searchBase = CN=Users,DC=SOME_GROUP,DC=COMPANY,DC=COM
#activeDirectoryRealm.url = ldap://ldap.test.com:389
#activeDirectoryRealm.groupRolesMap = “CN=admin,OU=groups,DC=SOME_GROUP,DC=COMPANY,DC=COM”:”admin”,”CN=finance,OU=groups,DC=SOME_GROUP,DC=COMPANY,DC=COM”:”finance”,”CN=hr,OU=groups,DC=SOME_GROUP,DC=COMPANY,DC=COM”:”hr”
#activeDirectoryRealm.authorizationCachingEnabled = false

### A sample for configuring LDAP Directory Realm
#ldapRealm = org.apache.zeppelin.realm.LdapGroupRealm
## search base for ldap groups (only relevant for LdapGroupRealm):
#ldapRealm.contextFactory.environment[ldap.searchBase] = dc=COMPANY,dc=COM
#ldapRealm.contextFactory.url = ldap://ldap.test.com:389
#ldapRealm.userDnTemplate = uid={0},ou=Users,dc=COMPANY,dc=COM
#ldapRealm.contextFactory.authenticationMechanism = simple

### A sample PAM configuration
#pamRealm=org.apache.zeppelin.realm.PamRealm
#pamRealm.service=sshd

### A sample for configuring ZeppelinHub Realm
#zeppelinHubRealm = org.apache.zeppelin.realm.ZeppelinHubRealm
## Url of ZeppelinHub
#zeppelinHubRealm.zeppelinhubUrl = https://www.zeppelinhub.com
#securityManager.realms = $zeppelinHubRealm

sessionManager = org.apache.shiro.web.session.mgt.DefaultWebSessionManager

### If caching of user is required then uncomment below lines
#cacheManager = org.apache.shiro.cache.MemoryConstrainedCacheManager
#securityManager.cacheManager = $cacheManager

securityManager.sessionManager = $sessionManager
# 86,400,000 milliseconds = 24 hour
securityManager.sessionManager.globalSessionTimeout = 86400000
shiro.loginUrl = /api/login

[roles]
dataScience = *
admin = *

[urls]
# This section is used for url-based security.
# You can secure interpreter, configuration and credential information by urls. Comment or uncomment the below urls that you want to hide.
# anon means the access is anonymous.
# authc means Form based Auth Security
# To enfore security, comment the line below and uncomment the next one
/api/version = anon
#/api/interpreter/** = authc, roles[admin]
#/api/configurations/** = authc, roles[admin]
#/api/credential/** = authc, roles[admin]
#/** = anon
/api/notebook/** = anon
/** = authc

Want to get more content about big data?

• Contact me via linked in Omid Vahdaty
• website: https://amazon-aws-big-data-demystified.ninja/
• subscribe to our AWS Big Data Demystified youtube channel
• subscribe to our Big Data Demystified youtube channel

——————————————————————————————————————————

I put a lot of thoughts into these blogs, so I could share the information in a clear and useful way. If you have any comments, thoughts, questions, or you need someone to consult with, feel free to contact me:

https://www.linkedin.com/in/omid-vahdaty/

in case of having Shiro authentication for Zeppelin, first you need to login from the command line using the credentials. Please find the below procedure in order to run a job from command line.

The first step is to authentic through the API with a POST to /api/login. I have used the default username and password from shiro.ini file. Use the below command in order to login from command line. Modify the IP address and credentials. In your case it looks something like this. ‘userName=GalGadot&password=WonderWoman’.

curl -i –data ‘userName=admin&password=password1’ -X POST http://ec2-34-27-130-194.us-west-2.compute.amazonaws.com:8890/api/login

The command will return the below output if the login is successful.

HTTP/1.1 200 OK

Date: Monday, May 7, 2018 6:10:03 PM UTC

Access-Control-Allow-Credentials: true

Access-Control-Allow-Headers: authorization,Content-Type

Access-Control-Allow-Methods: POST, GET, OPTIONS, PUT, HEAD, DELETE

X-FRAME-OPTIONS: SAMEORIGIN

X-XSS-Protection: 1

Set-Cookie: rememberMe=deleteMe; Path=/; Max-Age=0; Expires=Sun, 06-May-2018 18:10:03 GMT

Set-Cookie: JSESSIONID=f50827a1-59d4-40f3-9c60-d1b20ef47a83; Path=/; HttpOnly

Set-Cookie: JSESSIONID=deleteMe; Path=/; Max-Age=0; Expires=Sun, 06-May-2018 18:10:03 GMT

Set-Cookie: JSESSIONID=7143ce79-17be-4900-bad1-5d865bffe000; Path=/; HttpOnly

Set-Cookie: rememberMe=deleteMe; Path=/; Max-Age=0; Expires=Sun, 06-May-2018 18:10:03 GMT

Content-Type: application/json

Date: Mon, 07 May 2018 18:10:03 GMT

Content-Length: 123

Server: Jetty(9.2.15.v20160210)

{“status”:”OK”,”message”:””,”body”:{“principal”:”admin”,”ticket”:”7fcecbea-3a1e-40ed-8e21-0514e43e68a9″,”roles”:”[admin]”}}

2. Next, Using this cookie we can make authorized request to the API by simply setting this as a cookie in the cURL request. Use the “Set-Cookie” from the above output. I got two “Set-Cookie” from my outout. When I gave the first one it failed. However when I tried giving the second one i.e Set-Cookie: JSESSIONID=7143ce79-17be-4900-bad1-5d865bffe000; Path=/; HttpOnly, it worked. So please try the below command with the appropriate cookie. and the job url.  

curl -i -b ‘JSESSIONID=7143ce79-17be-4900-bad1-5d865bffe000; Path=/; HttpOnly’ http://ec2-xx-xx-xxx.us-west-2.compute.amazonaws.com:8890/api/notebook/job/2A94M5J1Z

So if the authentication  is successful, it returns the ” 200 OK” http status.

HTTP/1.1 200 OK

Date: Monday, May 7, 2018 6:13:22 PM UTC

Access-Control-Allow-Credentials: true

Access-Control-Allow-Headers: authorization,Content-Type

Access-Control-Allow-Methods: POST, GET, OPTIONS, PUT, HEAD, DELETE

X-FRAME-OPTIONS: SAMEORIGIN

X-XSS-Protection: 1

Content-Type: application/json

Date: Mon, 07 May 2018 18:13:22 GMT

Content-Length: 1184

Server: Jetty(9.2.15.v20160210)

 

{“status”:”OK”,”body”:[{“progress”:”100″,”started”:”Mon May 07 17:59:43 UTC 2018″,”finished”:”Mon May 07 17:59:45 UTC 2018″,”id”:”20150213-231621_168813393″,”status”:”FINISHED”},{“progress”:”100″,”started”:”Mon May 07 17:59:43 UTC 2018″,”finished”:”Mon May 07 17:59:48 UTC 2018″,”id”:”20150210-015259_1403135953″,”status”:”FINISHED”},{“progress”:”100″,”started”:”Mon May 07 17:59:44 UTC 2018″,”finished”:”Mon May 07 18:00:01 UTC 2018″,”id”:”20150210-015302_1492795503″,”status”:”FINISHED”},{“progress”:”100″,”started”:”Mon May 07 17:59:49 UTC 2018″,”finished”:”Mon May 07 18:00:03 UTC 2018″,”id”:”20150212-145404_867439529″,”status”:”FINISHED”},{“progress”:”100″,”started”:”Mon May 07 18:00:01 UTC 2018″,”finished”:”Mon May 07 18:00:05 UTC 2018″,”id”:”20150213-230422_1600658137″,”status”:”FINISHED”},{“progress”:”100″,”started”:”Mon May 07 17:59:45 UTC 2018″,”finished”:”Mon May 07 17:59:45 UTC 2018″,”id”:”20150213-230428_1231780373″,”status”:”FINISHED”},{“progress”:”100″,”started”:”Mon May 07 17:59:45 UTC 2018″,”finished”:”Mon May 07 17:59:45 UTC 2018″,”id”:”20150326-214658_12335843″,”status”:”FINISHED”},{“progress”:”100″,”id”:”20150703-133047_853701097″,”status”:”FINISHED”}]}

By running the job using the above command, if it is successful, the status will be “FINISHED”. I hope the above information is useful.

Need to learn more about aws big data (demystified)?

• Contact me via linked in Omid Vahdaty
• website: https://amazon-aws-big-data-demystified.ninja/
• Join our meetup, FB group and youtube channel
  • Join our meetup : https://www.meetup.com/AWS-Big-Data-Demystified/
  • Join our facebook group https://www.facebook.com/groups/amazon.aws.big.data.demystified/
  • subscribe to our youtube channel https://www.youtube.com/channel/UCzeGqhZIWU-hIDczWa8GtgQ?view_as=subscriber

——————————————————————————————————————————

I put a lot of thoughts into these blogs, so I could share the information in a clear and useful way. If you have any comments, thoughts, questions, or you need someone to consult with, feel free to contact me:

https://www.linkedin.com/in/omid-vahdaty/

Apache Shiro is part of the installation of EMR with the options below:

• Basic authentication (via Apache SHIRO): user management (user,pass,groups), even LDAP

https://zeppelin.apache.org/docs/0.7.3/security/shiroauthentication.html

• notebook permissions management: read/write/share

https://zeppelin.apache.org/docs/0.7.3/security/notebook_authorization.html

• Data source authorization (e.g 3rd party DB):

https://zeppelin.apache.org/docs/0.7.3/security/datasource_authorization.html

• Zeppelin with kerberos
• https://zeppelin.apache.org/docs/latest/interpreter/spark.html#setting-up-zeppelin-with-kerberos

Adding HTTPS/SSL to the EMR Zeppelin GUI (3 options)

• You can use a tunnel as used in EMR GUI websites (secured by default)
• Authentication and SSL via nginx
  • https://zeppelin.apache.org/docs/0.7.3/security/authentication.html#http-basic-authentication-using-nginx
  • https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.6.3/bk_zeppelin-component-guide/content/config-ssl-zepp.html
• you can add ELB on top of EMR , in 443, out 8890  for the zeppelin gui via HTTPS

Walkthrough to add https to Zeppelin:

1) Generating PKCS1 keystore file : Log into the master instance of the EMR cluster and run the following commands:

openssl req -newkey rsa:2048 -nodes -keyout key.pem -x509 -days 365 -out certificate.pem

openssl x509 -text -noout -in certificate.pem

openssl pkcs12 -inkey key.pem -in certificate.pem -export -out certificate.p12

openssl pkcs12 -in certificate.p12 -noout -info

Please enter the public DNS name of the master node when asked for hostname. The above commands would create a file named: /home/hadoop/certificate.p12 This file is your certificate

2) Change the below properties in the zeppelin-site.xml file located at /etc/zeppelin/conf.dist/zeppelin-site.xml (If not present, copy the /etc/zeppelin/conf.dist/zeppelin-site.xml.template file and rename)

zeppelin.ssl

true

Should SSL be used by the servers?

zeppelin.ssl.keystore.path

/home/hadoop/certificate.p12

Path to keystore relative to Zeppelin configuration directory

zeppelin.ssl.keystore.type

PKCS12

The format of the given keystore (e.g. JKS or PKCS12)

zeppelin.ssl.keystore.password

password

Keystore password.

Can be obfuscated by the Jetty Password tool zeppelin.server.ssl.port 8445 Server ssl port. (used when ssl property is set to true) 3)

Restart Zeppelin :sudo stop zeppelin

sudo start zeppelin

4) You would be able to access Zeppelin over https on port 8445 : https://:8445/#/

User management Via Shiro

Now in order to manage groups/roles, you could create the groups/roles under the “[roles]” section in the “shiro.ini” file. For example, I could have a set of groups like:

    [roles]

    admin = *

    readonly = *

    poweruser = *

    scientist = *

    engineer = *

Then in the “[users]” sections, it could be looking like the below:

    [users]

    admin = password>, admin

    user1 = password>, scientist, poweruser

    user2 = password>, engineer, poweruser

    user3 = password>, readonly

For example, the above means that:

    – user “admin” is in “admin” group;

    – user “user1”  is in “poweruser” and “scientist” group

    – etc.

    Owners  admin

Writers scientist,engineer,poweruser

Readers readonly

Once the groups/roles are created, the authorization setting will be similar to what described in https://zeppelin.apache.org/docs/0.7.3/security/notebook_authorization.html . For instance, when in a notebook permission page, you can put the group name, instead of the individual users.

good read: recommendation from horton works:

https://community.hortonworks.com/articles/141589/zeppelin-best-practices.html

Need to learn more about aws big data (demystified)?

• Contact me via linked in Omid Vahdaty
• website: https://amazon-aws-big-data-demystified.ninja/
• Join our meetup, FB group and youtube channel
  • Join our meetup : https://www.meetup.com/AWS-Big-Data-Demystified/
  • Join our facebook group https://www.facebook.com/groups/amazon.aws.big.data.demystified/
  • subscribe to our youtube channel https://www.youtube.com/channel/UCzeGqhZIWU-hIDczWa8GtgQ?view_as=subscriber

——————————————————————————————————————————

I put a lot of thoughts into these blogs, so I could share the information in a clear and useful way. If you have any comments, thoughts, questions, or you need someone to consult with, feel free to contact me:

https://www.linkedin.com/in/omid-vahdaty/