Running all images on a single host

[aliabbasjp]

It is required to run the entire system with all the containers on a powerful single host system , what changes would be required and which mode would be the best to do this ? MESOS or standalone?

[kfoss]

Your best bet is probably going to be the Docker containers/scripts from AMPlab. There are other similar Docker environments designed for the "all-on-one-machine" setup, but start with AMPlab's since they created Spark.

You could try to make this repo fit that use case, but it is really tailored towards a multi-machine environment. For example, the mesos-mastered version sends jobs to a separate Mesos master that either pulls the Spark binary from HDFS or uses the Docker containerizer for task execution; similarly, the standalone version uses a dedicated notebook server, dedicated Spark master, and dedicated worker node(s) that each run the same container. Basically, we needed a way to have a portable and homogeneous environment across several dedicated nodes.

In order to make that multi-server environment work, this repo's Docker containers encapsulate all necessary libraries/packages, while the accompanying shell scripts handle the network routing. That network routing is the main part you'd have to update if working on one machine since it currently binds the host's hostname to each container as part of the container2->host1->host2->container2 routing. If setting this up within one machine, you'd have to configure Docker links if running multiple containers; alternatively, it might be possible to run all nodes within the same container.

All that work above probably isn't worth it since your end state would be what AMPlab and others have already put together. It should be a fairly straightforward exercise to start the single-machine cluster and add an IPython/Jupyter notebook bound to the external-facing host port (i.e. 8888) and be on your way.

[agude]

In addition to the AMPlab repo, you might want to check out jupyter/docker-stacks. I used their docker file to get a single machine instance of Spark with Jupyter running on my laptop on Ubuntu 15.10.

[aliabbasjp]

@agude Thanks , I am already using this https://github.com/jupyter/docker-stacks/tree/master/pyspark-notebook

but the only problem is that it does not have hdfs and cloudera/hue/hive installed for that I have installed another docker from https://github.com/intropro/single-node-cloudera-env.git

But now the new problem I am facing is that I cant access this hadoop installation in this single node docker from the pyspark notebook using hdfs://.

@kfoss Also based on my above description is this amplab thing needed and amplab was last committed 2 years back?

[kfoss]

Ah, now I understand what you're trying to do and where most single-system containers fall a bit short. To your specific question, you don't need to use the AMPlab container; it looks like @agude provided a better alternative for your case

We had that same issue with Hadoop integration; whatever you end up using, you'll have to add a few things:

1. Hadoop libraries and dependencies: We built ours off JDK7 and CDH5. You could easily copy/paste the contents of the oracle-jdk7 and cdh5-hadoop Dockerfiles into whatever you're using. Both of those are within our repo at dockerfiles/base/lab41
2. Hadoop config: our repo's runtime/cdh5/[hadoop|hive]/conf contain the shell files you can update for accessing your cluster within the container. In our setup, we bind those directories to the containers' /etc/[hadoop|hive]/conf directories so the CDH libraries "know" where to access the cluster. Check out the startup script 4-run-spark-client-ipython.sh to get an idea of how we did that binding with --volume=$__host_dir_hadoop_conf:/etc/hadoop/conf
3. Container linking: it sounds like you want pyspark within container1 to access hadoop in container2. You'll have to make sure those two containers can "talk" to each other, which for only two containers is easiest with a link. You could pass something like, --link=hadoop_container_name:dns_name_within_pyspark_container when you startup the pyspark container, which would let you then route network traffic with the alias dns_name_within_pyspark_container. I think the return routing (hadoop back to pyspark) should be handled via spark, but you might need to setup that return route if for some reason you see jobs hanging or other indications of network mis-routes.

Once the container has the Hadoop libraries and config files, you can then leverage the typical commands (i.e. hadoop fs -ls) to access your cluster.